Copy Jobs Error: Bad Data. Failed to call CryptDecrypt AesAlg failed to decrypt

[mrholm]

Hi
One of our copy jobs are failing with Error: Bad Data. Failed to call CryptDecrypt AesAlg failed to decrypt, keySet: ID: 5447c6b8fa73ee429c3c4bc8838b19cb (session), keys: 1, repair records: 1 (master keys: 7caaa4e9faf94c9da1edd4231bb8d10e) Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

We have 52 other jobs (400 servers) on this site that works fine so it's just this job on one server.
This job has never worked it seems to stop on the same disk every time, it works for several disk but one disk is cretaing this error

I have a case: 03779785 and have given logs to it. The answer as for now is that the SOBR is to heaily used but we run this when no othe rjob (backup or copy) is done on the SOBR.

[wishr]

Hi Mats,

I'd suggest keeping discussing it with the support team and pass those details to them directly. Feel free to escalate it if necessary.

Btw, how many disks the affected VM has?

Thanks

[mrholm]

Yes of course I continue on what support give us. Can't see how the number of cuncurrant tasks should be an issue when it's the only server in one copy job and no other jobs are running. Soo it will allways be just one task.
I have changed the throttling down to just 1 Gbit for this source and see ifthat changes anything since support says it's the netwokr that is prolematic

[wishr]

How many disks this VM has? I'm asking because of "one task=one VM disk".

[mrholm]

8 disks but as said we run the environment witj over 400 server being copied during night without any problem so it's not a performance issue, it always stop on same disk so I guess that's something on the disk that's causing this. Will contine to work on this with support.
Also changed down to just one task now to test the idea around to many tasks

[mrholm]

I say here as well as to support this on copy job not on backup. Backups are done to a SOBR (7 servers with one repo each) using veeam encryption and that works fine. When doing copy on this perticular job with just one server it stops with this error. We do bakcups of 400 server and do copy of 400 servers as well each night.

[wishr]

Thank you for the info!

As far as I see there is an ongoing conversation with our support engineer and it seems there is some progress. I'll keep an eye on it, but could you please post a final outcome here once the case is fully resolved so forum community members will benefit from it.

Regards,
Fedor

[mrholm]

An update, we have ran the Veeam.Backup.Validator and can see that one vmdk file is corrupt on the backup, will do a new Active Full and and also do Health Check on current chain and se if it's possible to auto heal

[mrholm]

Hi
Case: 04232786
Before upgrading to v10 we had some of these erros coming but not regularily. After upgradring to v10 we now had this error on at least 5 server during the last week. This comes in copy jobs and the copy job will continue to fail:
Error: Bad Data. Failed to call CryptDecrypt AesAlg failed to decrypt, keySet: ID: cd1f982c5367902801b844948f7d63f8 (session), keys: 1, repair records: 1 (master keys: 88066c97fb26dac58c9eb737313b7eb0)

Performing a Health Check on the backup job ends up with failure like:
Disk xxx_1-flat.vmdk on host xxx is corrupted. Possible reason: Storage I/O issue. Corrupted data is located in the following backup files
This ends up sometimes giving us the whole backup chain being corrupted.

We see this on four different repositories (all local disk on Windows servers) so it's not related to one repository. We encryot all of our backups to disk and I believe it's in the encryption (based on the error message) the error resides.

Have anyone seen this as well and escalating after upgrading to v10?
We can't have to create extra Active Fulls every now and then we trust in Veeam and Synthetic Full and Incr/reverse but if this contiues we need to look into other way of securing or data

//Mats

[Gostev]

Looks like it's time to replace your backup storage. I would be very concerned seeing more than one server impacted occasionally, but 5 servers impacted clearly indicates a persistent data corruption situation, for example due to a sticky bit issue in some chip.

[mrholm]

Hi
To clarify we see this on different Veeam sites and now on 6 different repositories so it's not one storage with problem this seems to be Veeam v10 problem since we only have had this 3 or 4 times in the whole environment before upgrading to v10.
Yesterday I did a health check on one of the backup jobs where the copy is failing. The backup is doing synthetic full followed by incremental (30 points) and guess what suddenly Veeam finds corruption in the VBK file meanin thayt the whole chain is suddenly corrupted. This is not ok to continue doing incrementals in a chain and suddenly find out that the full backup is corrupted (second job I have to do it for).
So I have to run a new active full backup on the job (since we still can't pinpoint one server in a job) on a job with 25 TB in size, thanks for that Veeam.
I will go ahead and escalate this since now Veeam starts destrouying whole backup chains in a way never seen before.

I have worked with Veeam for so long time but I really regret going to v10 at this point. We have this major issue and we're thinking about going away from encryption since all errors seems related to that in backup jobs also having support case with GUI being totally unresponsive without any luck of support yet.

[toma1269]

I agree...more issues w/ v10 than ever before. QA/QC is not what it once was at veeam I fear. I just reseeded a customer via portable media, copied the new VBK files over to our CC storage, and now the CC job won't run due to this error. I've never seen this before. Our storage is good (using a SOBR). Both us and customer are on v10 patch 2.

[veremin]

You didn't provide us with the support ticket (or even open one), single support ticket that we had from your fellow Service Provider in was closed due to no response. Unfortunately, we cannot assist you with the resolution process without debug log analysis and infrastructure assessment.

That's said, if you still experience the same issue, kindly, open a support case and we will make sure R&D team is following the investigation.

Thanks!

[mrholm]

We're investigating this with support (04232786) and for now we have started to go away from encryption on the backup jobs to see if we can get rid of these errors. Will have a webex with support today or tomorrow