Unable to restore, errors, CRC

[mkatich]

Hi, I currently have a problem restoring a backup for a customer and would like to hear any insights about this issue that people here can offer.

I have a support ticket open, Case #05022429, and there is no response yet.

I am trying to import and restore a VBK full backup (with or without VBM file) and receive the following errors below. This first one happens immediately when I try importing just the VBK file by itself.

Dir failed, storageFileName 'myfilenamehere.vbk'
Failed to reload metadata bank. Declared and actual CRC are different for all bank snapshots.
--tr:Failed to load metastore
--tr:Failed to load metadata partition. Delayed loading: [1]
Failed to open storage for read access. Storage: [myfilenamehere.vbk].
--tr:Failed to run synchronized operation in MTA backup apartment.
--tr:Failed to list items in the storage [myfilenamehere.vbk].
--tr:Client failed to process the command. Command: [dir].
--tr:event:3:

Error in the application.

If I first import the VBM file, this happens-

Importing backup
Creating database records...
Backup is encrypted and will appear under Encrypted backups node until a valid password is supplied.

I find the Backups on the home screen and see the "Disk (Encrypted)", and I then specify the password. It says "Decrypting encryption key sets" and finishes, and then I have a "Disk (Imported)" showing. It shows a backup job name and platform is Hyper-V.

When I try to start a restore, I select the earliest restore point there, the full backup, and receive a similar error-

Failed to reload metadata bank. Declared and actual CRC are different for all bank snapshots.
Failed to open storage for read access. Storage: [myfilenamehere.vbk].
Failed to restore file from local backup. VFS link: [summary.xml]. Target file:
[MemFs://frontend::CDataTransferCommandSet::RestoreText_{some hex code here}]. CHMOD mask: [24].
Agent failed to process method {DataTransfer.RestoreText}.

I'm guessing the CRC error is the big issue here. Is that correct? Is there anything I can do at all about this? I have searched these forums but CRC errors are not mentioned much.

Is there any way to decrypt and restore anyway? Trying something is better than nothing in this situation, any data is better than no data. Maybe it will all be bad data. But if there's a chance some data will be good, I would pursue it.

Does Veeam have any method for forcing a decrypt attempt? If not, is there another known method of decrypting Veeam backups? I am open to trying another tool or even developing something myself. That may be possible if the encryption scheme used by Veeam is completely understood. I have read some encryption details in the online docs, but I would really need some information from Veeam for replicating that.

Ask any clarifying questions you may have. I'll answer what I can. My background on this customer's backup situation is limited. However I have been told they previously had a ransomware attack and recovered from that and decrypted all the data. Plenty of the other data has decrypted correctly. For example other Veeam backups there function correctly, and the VBM file for the problem backup is a complete XML file and looks fine.

I appreciate any input!

[wishr]

Hi Michael,

There are chances our technical support engineers will be able to help you with that.

As far as I see the case has been opened on behalf of a free product - is this correct? I'm asking because technical support for free Veeam products is provided on a best effort basis, based on the staff availability and it does not guarantee your case will be processed 100%.

By the way, are you a service provider? If you are, please apply for Veeam Cloud & Service Providers forum because your topic will be soon moved to it. You should be a registered VCSP to get the approval.

Thanks

[PetrM]

Hello,

It seems that data consistency check (CRC - Cycle Redundancy Check) for backup file metadata is failed because it is corrupted, most probably due to some hardware issue at the level of backup storage or during communication with the storage. I don't think that there is a way to pull out consistent data from such a backup file but let's wait for what our support engineers can find out. You can also take a look at this KB as it describes a similar issue with backup file metadata.

Thanks!

[mkatich]

Hi wishr and PetrM, thank you for your replies!

I did just use the free product as testing myself. I thought I had signed up for the free trial so that would give me more full access, but in the support area I had no options really besides the community edition. Possibly I am missing something. I am a data recovery service provider, so probably not the type of service provider you are thinking of. We are more of a last resort in this situation. We have used Veeam maybe a few times for clients but not enough to warrant having a license of our own. My company does not use Veeam ourselves. Software licensing is sometimes a big roadblock to us helping our clients, so I appreciate that Veeam has a free trial and a community edition.

I see the support ticket has been closed with no response. If I have the trial, do you think my support ticket would get looked at? Like I said, I might have missed something there.

Do you think I should apply for the Veeam Cloud & Service Providers forum?

[Mildur]

Software licensing is sometimes a big roadblock to us helping our clients, so I appreciate that Veeam has a free trial and a community edition.

By the way, That‘s forbidden by the Veeam License Agreement. You are not allowed to offer services (Implementation, Management, Backup Monitoring, helping with issues) for new and existing installations to your customer with the community edition

If your clients are using community edition, please have them buy a prober license. Or have a look at veeam rental license program, if you are company manages the backup server for the customer. With that, you have a license for your customers with guaranteed veeam support.

[Gostev]

Hi, Michael

No reason to apply for the forum as you won't be approved anyway, because the approval is automated and based on your contract with Veeam.

As for the backups, unfortunately in this case they are broken beyond any repair. We store metadata twice for redundancy, and in your case all copies are corrupted. This would indicate either massive backup storage corruption, or perhaps malicious activity. But without this information, it is impossible for us to make sense from the rest of the backup file.

Thanks!

P.S. And Mildur is right, importing customer backups into your Community Edition install directly violates Veeam EULA:
"5.0 Free Licenses and Community Licenses ... You may not use the Free and Community Licenses to ... process third party data"
So you may want to make your company's management aware of this on-going 3rd party software misuse.

[mkatich]

All right. Is it also forbidden for the trial? If not, I will go back and figure out why I don't have the trial installed. I did sign up for the trial. This is actually the first time I have used Veeam. If it is also forbidden for the trial, then I am likely finished working on this. I assume my client has a current Veeam license. I will inquire about that, but I guess we cannot help either way with any problems related to Veeam.

Will do. I will uninstall the community edition and check with management about anybody else using it.

[Mildur]

If the client has a active veeam license (Perpetual Maintenance or VUL) he can make you case admin for his license.
That way you can open a support case for his backup infrastructure. You are doing the restore for your client. Why not use his license to open the support case?
In the end, you are doing the restore for him and his license. Could be helpful for next time.

https://www.veeam.com/kb2211

[Gostev]

All right. Is it also forbidden for the trial?

Indeed, processing 3rd party data is not possible without a special license:
"3.0 Prohibited Use. You may not (a) process third party data ... without purchasing the specific Veeam license to do so"
CE or Trial licenses are not such special licenses.

What you can do:
1. Use your client's backup server to open your client's backup file. This is not against EULA because they remain the primary user of software in this case, and you're just helping them with one specific problem (as opposed to providing continuous/repeating service that involves Veeam software products).
2. Open support cases on your client's behalf, if they designate you as a case admin.

What you cannot do is:
1. Import your client's backups into backup servers licensed to you (this use case requires a special license).
2. Use your client's license to deploy a backup server in your environment (as licenses are not transferrable).

Normally, the only way you can legally interact with 3rd party data using Veeam software products is to register as Veeam Cloud & Service Provide and rent your clients a license. You will then be able to import backups created by Veeam installations with such licenses into your own backup servers. As far as I understand your business, this approach does not really fit because you have absolutely random Veeam customers coming to you for help (as opposed to the specific scope of your clients). Because of that, the only option for your case would be a custom licensing agreement between your company at Veeam.

[mkatich]

I appreciate the help on this from everyone.

You are correct about our business. Normally people contact us for the first time when they have already lost data. Often this is related to a hardware failure, but it could also be a problem in the data. So we do not have ongoing contracts such as a managed service provider or a backup service provider would have. We perform a one-time service for a one-time fee if we are successful. We do not actually charge a fee if not successful.

Some recovery work involves custom data processing. Rarely do we need to actually attempt any custom work within files. We will occasionally verify that files function depending on the customer's objective and the recovery conditions - like limited corruption or scattered unrecoverable sectors. Frequently our customers require data to be recovered that was created with a software we have not encountered before. Sometimes it is even from the 1980s. We can't test every type of data obviously because we don't have access to every possible software, so this includes Veeam backups.