FYI potential Data Corruption issue with Proxmox

[Dagr]

Hi,

Are R&D aware of the post in Proxmox forums about a potential data corruption issue with Proxmox backup?

https://forum.proxmox.com/threads/veeam ... on.155212/

[PTide]

Hi,

Yes, they are : ) We are looking into that.

Thanks for bringing this up

Cheers

[Sean S]

Currently just had a VCSP just open a case about this issue.

[nathano]

yep, this is worrying....

[nathano]

@PTide could you please check case 07448910 and see if that's connected to this.

as part of testing with the issue I'm having I've discovered that excluding a drive from a VM in a pve backup causes problems when you try to mount it (restore guest files - windows). VBR just hangs at this point. volume2 is the drive that's been excluded. update, after about 10-15min it finally presented the file browser, have updated the case

[PTide]

Hi,

So, FLR works, it just took too long for the file-browser to show up? Does the browser show the content properly?

Thanks!

[nathano]

Yes it did work after a while. But the case I logged was for what seems to be a corrupted backup and could not launch that browser.

I did a new active fill and at least I can browse now. Concerned my original problem could be related to the original issue of this post.

[PTide]

Had it been corrupted, the FLR browser wouldn't have worked for that backup at all. I requested your case to be reviewed, so please work with the support engineer.

Thanks!

[nathano]

Is there any update on this issue? I have logged a case (07461323) about disks within a VM being backed up at different times.. not sure if that's related to this issue or not

[PTide]

Hi,

It's in the works - we have tracked down the root cause, and now we are working on getting it fixed.
I will update this thread when there will be news.

Thanks!

[Mario Rossi]

Hi, any news?
Are Proxmox VM backups made with Veeam safe or not?

[PTide]

The patch is undergoing final tests, so we are almost there.

Thanks!

[Gostev]

The hotfix is now available > https://www.veeam.com/kb4686

[beh]

@Gostev
Does this hotfix need a new chain/full-backup?

[Gostev]

No.

[blockbridge]

The hotfix is now available > https://www.veeam.com/kb4686

Great news, we will give it a go when we free up some cycles.

Best,
Blockbridge

[blockbridge]

Good news! Veeam backups with the new version are functional! Restored VMs passed our snapshot consistency tests. So, we can say that backup of a VM with a single disk is "point in time" (i.e., crash consistent) and has integrity when restored.

We also confirmed that previously taken backups were non-recoverably corrupt. Taking full backups after updating to the version with the fix makes sense.

We have a few more tests to run, but we wanted to keep everyone in the loop. So far, so good!

[Gostev]

As I already shared in my previous post, taking full backups is not required in this case.

Also, if you previously taken backups are "non-recoverably corrupt" then you're likely facing some other issue, perhaps a storage-based corruption. Because we did not see any cases of unrecoverable backups due to this issue either internally or in Customer Support in a few months since 12.2 was released.

[blockbridge]

Hi Gostev, I am following up and closing this out. Based on our testing, the corruption issues with the backup data have been resolved by the Veeam software update.

We ran the following test cases with full backups on lvm, zfs, and Blockbridge to prove the fix:
- I/O sequence analysis internal to a single disk
- I/O sequence analysis distributed across eight disks
- dm_integrity checking of a single disk

In each case, the restored contents were valid, and the data contents were correct. This should be sufficient to support Veeam in our customer environments.

Feel free to reach out if you need testing on future updates.