Veeam 4.1.2 corrupts replication and backups

[Andrey]

Hello,

I'm evaluating Veeam Backup and stumbled over an extremely bad software behaviour - any attempt to make backup or replication of Windows 2008 AD controller make either source and destination completely unusable. For sure I always enable Veeam VSS, but this doesn't help much. File system got corrupted.
Any attempt always leads to:
Creating snapshot
VSSFreezer: -805306316 Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
Cannot prepare the [NTDS] data to a subsequent restore operation.
Cannot process NTDS data.
Updating BCD failed.
Cannot get [BcdObject.Id="{9dea862c-5cdd-4e70-acc1-f32b344d4795}",StoreFilePath=""] object.
COM error: Code: 0xd0000034
-and --------------------
Removing snapshot
Failed to retrieve "SCSI (0:0) Hard disk 1" disk information. Check VM virtual hardware configuration.
Wrong number of extents - 21, descriptorFile "nfc://conn:psvm03,nfchost:ha-host,stg:4cc08c56-1efa2253-e35f-001b21651a9e@PSWAD01/PSWAD01-1-000001.vmdk"
Veeam Backup will attempt to remove snapshot during the next job cycle, but you may consider removing snapshot manually.
Possible causes for snapshot removal failure:
- Network connectivity issue, or vCenter server is too busy to serve the request
- ESX host was unable to process snapshot removal request in a timely manner
- Snapshot was already removed by another application
-----------------------------------
I've googled and also looked in these posts and figured out that possibly (possibly, to make a purchase decision I must to be sure 120%) that problem may be that Veeam sets "Enable VMWare tools quienscience" by default. But on another hand - documentation states that enabling Veeam VSS automatically disables VMWare tools VSS. None works. I'm totaly confused - why does software have options which lead knowingly to a total VM infrastructure destruction?
My configuration:
VMWare Essentials
2 ESXi 4.1 servers.
Veeam 4.1.2 on separate Windows XP station (for evaluation)
I would appreciate any suggestions on how to make proper backup and replication with Veeam software.

[joergr]

Please immediately open a support case with vmware, this has nothing to do with veeam. vcenter cannot commit your snapshots because somehow the extend (you did an extend to your vmfs lun latelay?) seems to be broken.

best regards,
Joerg

[Andrey]

Joerg,
Unfortunately we didn't have a VMWare support contract. Nor that I did any vmfs extend. This VM was converted from VMWare Server 2.0 to ESX and worked fine, till I attempted to make a backup. I understand that it has something to do with snapshots. But do you think I can fix it say - switch VM off and make cold backup/restore with Veeam?

[joergr]

Holy shit, boy, this is extreme stuff, china in your hand, and should be handled by vmware support. Anything i tell you is for your own risk i *really* suggest to get vmware support on this one. Now you asked for it, that it what i would do IN A TEST LAB: I would hot-clone the vm to another storage which i made sure is 100% safe. Then i would check out the clone in an isolated environment, turn it on, manually create and delete a snapshpot, see what it does. AFTER that i would shut down the source and undergo special tests with the source (storage, snaphots).

best regards,
Joerg

[Andrey]

Joerg,
I like your attitude
Thank you for your suggestion. I came from a world of raw iron where it was just supposed to work

[Gostev]

I have seen it before a few times when converting Workstation/Server VMs creates invalid disks (especially when you convert them into thin disks). Definitely not something I would recommend doing. With DC, it is so easy to create new DC VM directly on ESX, add to domain and sync it, transfer or roles to it from "original" DC, and decommision the original DC.

[Andrey]

I wish it would be a mere DC... Due financial restrains this server is also DHCP/DNS/File/Print and God knows what else.
Playing with a copy in a sandbox I would say that prior running Veeams backup file system is completely consistent. No event log complains(read or even yellow marks), vssadmin list writers shows no errors. Standard windows backup works flawlessly. Even ESX snapshot goes right... But, as soon as I start Veeam backup with Veeam VSS enabled I get OS into the unstable state - vssadmin list writers shows NTDS errors, many event log messages and as a final result - server unable to restart (Please inser bootable media and bla-bla-bla).
I'd like to understand what's going on before I jump the gun and start over new DC. What if it comes to the same?

[Gostev]

vssadmin list writers shows NTDS errors

Wow, looks like this server is quite borked. This another issue that you need to deal with as well, before trying to perform backup of this server with Veeam. Veeam VSS will not work if some VSS writers report errors.

snapshot goes right...

Well, snapshots with vSphere client does not really use any sort of quiescence at all, so you cannot compare it with snapshot created for backup. vSphere client is the same as running Veeam Backup job with both Veeam VSS and VMware tools quiescence unchecked. Of course, you do not want such backups for your DC, as backing up DC without VSS will result in USN rollback upon restore and DC will stop (assuming you have more than one DC).

[Andrey]

Well. I have no vss writer error before running a backup. I get them during backup and this is the issue I want to clarify.
Snapshots, understanding the implications I intensionaly checked quiescence for VMware during VMWare snapshots.
In fact I do have only one DC - no backup. I thought that having near real time replication with Veeam it would be unnecessary to keep extra license and/or hardware.

[Gostev]

Well. I have no vss writer error before running a backup. I get them during backup and this is the issue I want to clarify.

Well, this is expected as well as VSS writers do not work all the time, but during VSS snapshot only. Veeam, acting as backup application asks Microsoft VSS framework to prepare system for shadow copy > VSS framework kicks VSS writers > some writers fail and report errors > you see errors in the Event log > failed VSS writers state is set to Error > you see that with vssadmin list writers.

[Andrey]

Thank you. Looks like I have to reinstall DC and all s..t on it

[djcosonok]

Good day

I came across a similar problem with a P2V-d Windows 2003 SBS server. Everything ran fine until trying to take a Veeam Backup (this was Veeam 5.) The backup ran fine but failed to commit the snapshot with the warning:

Removing snapshot
Failed to retrieve "SCSI (0:0) Hard disk 1" disk information. Check VM virtual hardware configuration.
Wrong number of extents - 81, descriptorFile "nfc://conn:apollo,nfchost:host-1705,stg:datastore-7019@SOHO-SBS/SOHOSBS-000001.vmdk"
Veeam Backup will attempt to remove snapshot during the next job cycle, but you may consider removing snapshot manually.
Possible causes for snapshot removal failure:
- Network connectivity issue, or vCenter Server is too busy to serve the request
- ESX host was unable to process snapshot removal request in a timely manner
- Snapshot was already removed by another application

Then after manually committing the snapshot, the server starts reporting NTFS errors and corrupt file system errors, and falls into an unrecoverable state.

The fix was to do a V2V on a pre-backup copy of the server (had done a cold clone of the original uploaded server prior to it being backed up) and then after this the backup ran fine.

It looks like the problem was down to the original P2V being done using the coldclone.iso to a mapped drive on a laptop, this was then uploaded to the hosting environment (the P2V had not been done directly to the vCenter as this was not possible.) The V2V was done with the coldclone.iso to vCenter.

Cheers

[Alexey D.]

David, thanks for posting this explanation!