Data corruption on restore points - Case # 06355943

[bob808]

Created ticket with Veeam but no leads so far. Wondering if anyone else has seen this before.

Issue:
- Some restore points are randomly getting "corrupted" after being backed up successfully. This happened 5 times in the last 30 days that we know of for one VM.
- There has been at least one other VM in the same job with this issue as far as we know.
- I haven't seen this in any other jobs yet, but the only way to check is to check each restore point individually, so we don't yet know the full extent of the impact.

Symptoms:
- No errors or failures reported in the backup job.
- The Veeam backup validator reported no issues with the "corrupted" VBK files (https://helpcenter.veeam.com/docs/backu ... ml?ver=120)
- This tells me the VBK files are fine, but the contents of the VMDK files are corrupted.
- This does not happen all the time. Only some restore points are impacted.
- I've only seen this on two VMs so far, within the same job and the dates with corrupted restore points don't always coincide.

Restore Methods (This is a Windows 2019 Server with MS SQL)
1. Full VM Backup
- OS Will not load. VM Will boot up in Windows Recovery Mode as the OS volume is not recognized.

2. File Level Restore
- The Volumes letters are not recognized. Rather than c:\, d:\, etc.... ,the FLR appliance shows Volume1, Volume2, Volume3
- 2 Volumes mount fine, the rest are mounted in Veeam Backup Browser but show as empty with 0 objects.

3. SQL Application Restore
- Database fails to load.
- "MY_DATABASE ExplorerManagementService: Failed to get VM mount result (sessionId = 'd73d48af-39f6-4213-9fa0-xxxxxxxxxxx') | SQL metadata for database MY_DATABASE not found in restore point b479a6fd-be66-4ad1-9270-xxxxxxxxxx"

FLR Logs:
VeeamAgent.MountVm.Client.log
- Volume 0 and Volume 1 load fine. The rest of the volumes fail to load and have this error:

Code: Select all

[30.10.2023 13:09:45.956] <  7596> prx      | ERR |Failed to update reparse points for mounted volume 'C:\VeeamFLR\MY_VIRTUAL_MACHINE_47bf558e\Volume2'. Maximum count of points to update: '1000'.
[30.10.2023 13:09:45.956] <  7596> prx      | >>  |The volume does not contain a recognized file system.
[30.10.2023 13:09:45.956] <  7596> prx      | >>  |Please make sure that all required file system drivers are loaded and that the volume is not corrupted.
[30.10.2023 13:09:45.956] <  7596> prx      | >>  |Failed to open mounted volume: 'C:\VeeamFLR\MY_VIRTUAL_MACHINE_47bf558e\Volume2\'.
[30.10.2023 13:09:45.956] <  7596> prx      | >>  |--tr:Error code: 0x000003ed
[30.10.2023 13:09:45.956] <  7596> prx      | >>  |An exception was thrown from thread [7596].

[Gostev]

Do you use application-aware processing? As my only idea here would be VMware Tools quiescence messing things up.

[bob808]

Hi Gostev,

Yes, this is a backup job with application-aware processing that backs up and truncates MS SQL transaction logs.

I've asked the VM owner to upgrade VMWare tools and perform Windows updates. Still waiting to hear back.

Thanks,

Tim

[EskBackupGuy23]

Hello -

Was this ever resolved? I have a similar issue with trying to restore successfully backed up SQL databases on a 2 node physical cluster.
This part is what is similar: "MY_DATABASE ExplorerManagementService: Failed to get VM mount result (sessionId = 'd73d48af-39f6-4213-9fa0-xxxxxxxxxxx') | SQL metadata for database MY_DATABASE not found in restore point b479a6fd-be66-4ad1-9270-xxxxxxxxxx"

Thanks!

Ron

[Spex]

We also had this problem, case number 07123320, and could not find the cause. All backups of a particular Windows VM were performed correctly according to veeam, but when doing a file-level restore, drive letters were not shown for some disks. Sometimes disk C:, other times the DB disk was missing. Also, the VM could not be booted from surebackup.

We found that one of our two Windows backup proxies (both in SAN mode) kept causing the problem. A health check of the backups did not help to identify the problem. The proxy with the problem was investigated for all possible error sources (HW, Bios, SAN, Config, etc) without success and was identical to the other proxy. This problematic proxy backed up about 400 VMs daily - without ever generating such an error.

I automated the checking of all VM backups (Windows VMs only) by using a script with Publish-VBRBackupContent and comparing the number of mounted volumes with the information from vCenter.

Solution:
First we changed the agent for this VM to have working backups again and later we switched to Linux proxies (planned for other reasons) and the problem completely disappeared.

[EskBackupGuy23]

When you say "we changed the agent for this VM.." What do you mean? Did you remove and reinstall it? Use a different version?

I am having this same problem. Any help would be greatly appreciated.

Thank you!

Ron

[Spex]

Sorry for beeing unclear in my last post.
To have working backups again we switched to a in guest agent.
Verifyed this setup and no problems.
After that we exchanged our hardware proxies - as it was planed some time ago.
The new linux proxies worked as expected and so we stopped using the agent.
For control reasons we use a automated sure backup that tries to mount the disks and compares the number of mounts to our cmdb.