Any info on recovering data from VBK files?

[crp0499]

Back in April of 2022, I ran two backups of some data from our content dev and marketing people. One backup file is 16TB, and the other is 7TB. The backups were pretty standard backups to some storage I have and from what I recall, the backups went off without any issues. Fast forward to today, and I need to look at those two backups and restore a couple of folders from there and get those folders back to the content dev and marketing department. the problem is both of the file seem corrupt. When I try to import them into Veeam B&R, they error out with "Failed to import backup Error: Unable to import backup: failed to parse the summary.xml due to a backup file corruption."

I tried the Veeam Backup Extractor utility but that's only for extracting VMs I guess. These are flat file backups of two large drive arrays.

The next thing I tried was mounting the files with OFSMount, which can mount the backups just fine, and shows the size, but I can't read the drives. I get an incorrect parameter error.

I used the Veeam-R tool, and I can mount either file and I can see the vmx, the vmxf, the nvram, the vmdks, but that's as far as I get.

I opened a ticket with Veeam (Case 07509935) - which honestly, ALWAYS seems to be a waste of my time - but I did it all the same and they of course, can't help, are no help, don't know, can't fix it, don't know why it doesn't work, etc, etc, etc. Basically, I asked them why do I have backup software if, when I need to restore, I can't even open the backup with the same software I used to do the original backup? I mean what's the point. My client literally asked me why we pay for Veeam and I told him I don't know. Facepalm...

So, here in the community, does anyone know of a way to access these two vbk files? Has anyone had any success recovering data from a Veeam backup file that Veeam itself can't even open? I have 16TB and 6TB of data and I REALLY do not want to go tell the content dev peeps and the marketing peeps that I lost their 22TB of data.

Thanks in advance...

[Gostev]

Well, judging purely on the error these backups have likely experienced storage-level corruption. I understand your client's first reaction to blame backup software, but you as a backup professional should be well aware of this possibility? As this is exactly why 3-2-1 rule of backups mandates always maintaining a copy of your backups and on a different media.

Not to mention the best practice of testing backups periodically for recoverability - either manually or using built-in Veeam capabilities such as SureBackup or basic backup health check. The first recoverability testing for backups in question should have never happened at the moment when you actually need to restore, and almost 3 years after the backup was created.

Now, if you don't have another copy of these backups, then I'm afraid your only option is to work with our Support in trying to salvage some data from these backups with special tools. I don't know if T1 folks have access to those tools or not, however if they can't help you then they would be forced to escalate your support case to higher tiers.

[crp0499]

Gostev - agree on all points. I made a backup to a single location (on spinners no less) and I have no other copies BECAUSE the client won't invest the money in a proper backup. Their content dev and marketing people produce massive amounts of video and graphics, and I'm forced to move all of this over gigabit connections with spinners. I made the two backups I'm trying to restore in an effort to reduce the size of what we DO backup, and now they need some stuff from those backups.

You mention T1 support and other tools? When I was working with support, they used Veeam-R tool, but that's as far as they went before telling it the data was lost. Are there other tools or other tiers of support who can look at this?

[Gostev]

One key requirement for these support tools to be any useful is to have at least one healthy instance of metadata. Because of their criticality, we actually store them twice in each backup file for redundancy (full copies). So losing a few bits due to a bit rot, or even loss of a fairly long consecutive chunk of backup file e.g. due to failed write by RAID controller will not prevent recovery.

However, in case of really bad storage-level corruption that hits BOTH copies of metadata, unfortunately it will not be impossible to make any sense of the content of backup file, even if some big chunks are still "good". Based on your support engineer comments, it might be your case. Nevertheless, I suggest you ask for a second opinion using Talk to a Manager functionality of the support portal and they will assign someone senior who specialized on data loss cases to review the work already performed by your engineer. And may be try some other approaches not known to regular support engineers.

[crp0499]

I can't speak to lost chunks, bad RAID, or whatever. I know the storage the backups live on only do one thing...it's hosts these two files and the storage array has been solid since the original backup. I'll open another ticket, or look at my existing one, and see if I can get another tech to look.

Assuming Veeam can't get anything from the files, what do you think my luck will be like if I send it out to a data recovery company?

[Gostev]

I would not do this unless they charge only in case of successful restores. Just because our support is pretty good at salvaging data from corrupted backups as storage-level corruption is actually quite common, and we see it quite often in Support (especially with low-end NAS devices, which are responsible for easily 9 out of 10 of all data losses). So assuming second opinion will be the same, I would not pursue this further - in almost 20 years I'm yet to hear someone sharing here that a 3rd party company has recovered data from a Veeam backup after our support team could not.

[crp0499]

So, your suggestion is reopen ticket, work with higher tier until it’s either recovered or they give up?

Do I have that right?

[Gostev]

Just use Talk to a Manager functionality, explain in your message that this backup is your only chance to recover the data, and ask for someone senior to review what was already tried in the existing support case and confirm that truly nothing else can be done. Note that the message from this form will got directly to the Support Management distribution list.

[crp0499]

Understood, and thank you!