HELP! All Backup Files Corrupted

[magnosantos]

One of my machines went kaput and i need to restore it.
I have 16 backups points, with 3 Syntetic Full Backups.
However, not matter which backup point i choose i always receive this error:

Code: Select all

Error    Restore job failed Error: Failed to decompress LZ4 block: incorrect block header.
                             Failed to download disk.
                             Agent failed to process method {DataTransfer.SyncDisk}.

Strangely (at least for me), if I try to export an Incremental Backup, Veeam is able to create the new VBK without any errors. But same behavior on restoring.
I already check the disk for errors and not were reported, and i'm able to move files to another system without any issue, but same behaviour when trying to extract the .vhdx image inside.

I'm using Veeam Backup and Replication 9.5 u4 with a NFR License.

[wishr]

Hello Magno,

Welcome to Veeam Community Forums and thanks for posting.

We require all technical issues to be submitted into Veeam Support first and foremost. Please raise a support case and post case ID here, in this thread, as requested when you click the "New topic" button and in Forum Rules.

Most likely, the blocks have been corrupted due to an underlying issue with the storage, but there is a chance it can be fixed with the assistance of our tech. support engineers.

Thanks

[Gostev]

Actually, considering that salvaging all "good" data from backups hit by storage-level corruption is a manual process that requires many man/hours, I doubt our support engineers will be willing to do that for anyone without a support contract. Besides, since using an NFR license in production environments is specifically prohibited by Veeam EULA, lost data should not be that important to start with?

[magnosantos]

Agree,

I was using the NFR to showcase my management the Veeam benefits for a future purchase (waiting 2020 budget).
Now that I would have that opportunity the "system" has failed me, not being able to recover a VM.

I really appreciate any help that can be done.

The Storage is a NAS, which helds all of the backups.
What i find very strange is the fact all of the 16 backups (Full and Incremental) give the error on this VM. If the files with have issues, i should have erros on copying it, no? Besides this, other VMs backups seems to be ok (i was able to extract vhdx) on the same storage.

[Gostev]

No, you will never see errors on just copying files in case of silent data corruption such as bit rot, which is the most common data corruption type. There's simply no way for the OS to know if the content of the file is corrupted or not. Veeam on the other hand can see a corruption because we check whether the content of a backup file block retrieved from the storage device matches its CRC value stored along with the block. In your case, NAS returns different content than what was written to it back when the backup file was created.

Also, I don't think it is completely fair to say it was "system" that failed you. Technically your NAS did fail - but because storage-level corruptions are quite common, especially with low-end NAS, you should not have completely relied on its reliability and availability anyway.

What failed you is: not following the 3-2-1 rule of backups and keeping a copy of your backups on another storage, and not using SureBackup jobs to check your backups for recoverability. Or at least scheduling storage-level corruption guard feature of Veeam to run more frequently to account for missing recoverability testing with SureBackup. Even the latter feature alone would have caught bit rot, and moreover automatically healed it on the next backup run by re-reading the affected data blocks from source storage (this is fully automated process, no action is required on your part).

[magnosantos]

Just so i understand... when Veeam performs a ful syntetic backup it does not check for these issues?

[Gostev]

No, as doing this would slow down synthetic full processing, significantly increasing backup window. Each block would have to be decompressed and decrypted to get to the raw data before we can compare one against the checksum, which is CPU-intensive process.

Besides, this still won't ensure that your backups are recoverable, because bit rot is just one of many issues that make recoveries fail. Seeing them all in our technical support was the reason why we introduced SureBackup in the first place.

[magnosantos]

understood. So it's always a good idea do have intermediate real full backups to break a possible bit rot chain.

In this case, what options to i have to recover the data? It was the ERP of my company.. and now i'm in a big trouble.

[Gostev]

Actually, storage-level corruption guard feature in Veeam gives you much better protection against bit rot than an Active Full backup, because it physically reads stored backup (as opposed to merely shooting new full backup at potentially malfunctioning backup storage).

Not to mention that Active Full puts much more stress on your production environment due to the sheer volume of data to be read and copied over, and makes VM snapshots grow very large due to the time required - with all the associated large VM snapshot commit issues. While corruption guard activity is completely isolated to the backup target.

Honestly, you would have been in much better position if you used Community Edition instead. Not only you are allowed to use one in production environments, but you also have a chance to get support based on staff availability. While data loss is a problem, you put your company into much bigger risks by directly violating Veeam EULA and using NFR license in production environment to protect your company's production ERP (!!!) system.

[magnosantos]

I will right away solve the NFR issue and move to a Community Edition.
This is not really a production environment, but an evaluation of the product, since the goal was to purchase a license at beginning of the year. But in this case for this specific VM (and a couple more test VMs) it was the only solution taking care of Backup

Also going to implement your suggestions regarding storage-level corruption guard

[magnosantos]

But is there any way to recover the VM?

For instance if the company decide to move right away to a commercial license?

[veremin]

If you can get a commercial license (which should be discussed with our sales representatives) with valid support contract, you can reach our support team and see whether they can apply some advanced techniques to salvage some data from the corrupted backup. However, there are no guarantees that that will be possible (depends on how many blocks have been hit by storage corruption). Thanks!

[magnosantos]

We are now purchasing a Veeam Enterprise for 5 instances. I will came back when it is done, to request Valid Suport

[wishr]

Hello Magno,

Please remember that this is not a support forum. You need to open support cases using Customer Portal.

Thanks

[Gostev]

This is not really a production environment, but an evaluation of the product

For evaluation purposes, Veeam provides trial licenses. Unlike NFR, trial licenses these can (and should) be used in production environments. Plus, you can always request an extension of your trial.

If you can get a commercial license (which should be discussed with our sales representatives) with valid support contract, you can reach our support team and see whether they can apply some advanced techniques to salvage some data from the corrupted backup.

Normally, our support will refuse to support issues originating from CE/NFR licensed deployments, as this promotes unhealthy behavior of using free offerings for years at no cost, and only acquiring a minimum possible license to get support if you actually need one. Since I like analogies, this is best compared to buying insurance against some event after the event has already happened. Of course, no insurance company will do that.

Now, support may do exceptions on case by case basis, just because they like to help everyone whenever possible - what I'm saying here is that this is definitely not guaranteed with your late support purchase.

[magnosantos]

I understand that, and it make sense.

I hope to have some support and I really appreciate any help that can be given. (hope is the last thing to be lost)