Snapshot File Issues and More

[shanes]

Im new to Veeam and have hit several issues which I need to get a better understanding about.
I have a client with a single ESX 4.0 server and 3 hosts, one of which is a SBS2008 server. Recently I had a issue whereby Veeam caused the server to create _1.vmdk files in which grew to the same size of the orginal vmdk file and of course this meant I run out of space.

Im not sure if this is a issue with 4.1 and have upgraded to 4.1.1 however I can not have this happen again! Also I have the backups running off to a Windows 7 PC and during this crash I had to restore the backup to a new ESX server I took onsite. My concern here is that during the restore, I found that it took 10 hours to restore only 240Gb of data to the replacement ESX server. This was way too long and Im wondering if there is another issue here or maybe the compression settings should be modificed or something else to cut that recovery time down.
Secondly even with VSS enabled and no errors on the Backup job, once the box was recovered the Server had lost the fact that it was a DC (critical for SBS) and all I could do was log into Safe mode... I had to rebuild the system and migrate data!

Can I get some infromation on this and possibel solutions moving forward on these issues... thanks

[Vitaliy S.]

Hello Shane,

Welcome to Veeam comminuty forums! I believe this is a snapshot file that you're talking about, because Veeam doesn't create any virtual disks, we create snapshots of the source VM while backup/replication job is performed. This snapshot file is used while original VMDK is being backed up. And the size of the snapshot depends on the actual data change on your SBS server during a backup job period.

If you're lacking of free space we do recommend you to encrease your source storage capacity.

Also could you tell us whether you were restoring to ESX or ESXi server? You may try uploading the same large file to that ESX host using vSphere client Datastore Browser and see if you have the same speed or not. Also here is a useful link about Restore Speeds: Best Practice for Optimal Restore Speed

As for restoring SBS 2008 DC please have a look at the existing threads below for more information:
DC restore, Active directory authoritative restore and Restore of all dcs in a domain

Thank you!

[shanes]

Thanks for the info, however the snapshots I understand but that is not what I am referring to. I had the base vmdk file called SBS01.vmdk as a thick disk, after veeam ran a while back it created a new file called sbs01_01.vmdk this started to grow up to the same disk size as the orignal sbs01.vmdk file, at the time the Veeam backup said it had failed but worked the next day. Then 2 weeks back Veeam failed again and it created another new file called sbs01_02.vmdk file. It grow to 90% of the orginal file and disk space ran out and the VM crashed. Note VM support stated that the only way to fix this is to clone the box! However as you can tell this would mean increasing disk space which I had to do.
So my question is why are these files created and retained, of course you can not remove them using VM snap manager as they dont exist in the console and if you remove the _1 or _2 file you are taken back to the date/information on the orginal vmdk file which now would be months ago.

[shanes]

Also reading up on the AD restore there is a theme that you must have a secondary AD controller for the AD to replicate back after a restore.
Of course with SBS you are generally only using the single AD controller!!!! So the question is can you recover the domain controller if there is ONLY ONE?
I see that the forums discuss this in minor detail but I find it strange that there is no clear answer on this question considering the product indicates you can restore AD successfully.

[Gostev]

Yes, there are no problems restoring AD domain controller without any problems if there is only one DC. This has not been discussed much because it is the simplest scenario which cannot have any issues even theoretically (unlike multi-DC environments where synchronization issues may appear if DC is not restored properly)...

Also I noticed you are talking about bulk restore speed in you initial post, here's a most good recent thread about restore speed which provides great insight on why the full restore is not so fast and what can be done to improve its speed: Veeam Restore Speed

[Gostev]

Noticed your other post about snapshots. The latest release (4.1.1) adds warning on the job when ESX fails to process our request to remove snapshot to make you aware about this potential issue. So you should be safe now.

[shanes]

Gostev,
based on your post where you say it should not have any issues.
On the restored SBS2008 server two restores now have had the same result.
"No logon servers are available for your request" when you try to boot into the SBS server the first. Booting into Safe mode it looks like all AD shares are gone ...

[Gostev]

Shane, I believe this is SBS 2008 specific issue. This happens due to the fact that it runs many other applications besides actual Active Directory domain controller. What happens is some services attempting to start while domain controller is not ready due to restore procedure, and so service accounts cannot perform logon, which results in this error. There should be no issues with the following DC reboots. If the issue persists through reboots, we would want to take a look at this in live, so please open a support case with us.

[shanes]

I understand this process however, the server has been rebooted many many times and i even waited a day before trying to log again with the server on. Reviewing the event logs in SBS during a safe mode boot shows no errors regarding services not starting from a previous boot. Also the server on first boot after the restore did not boot into safe mode which I believe it should have?

[Gostev]

Also the server on first boot after the restore did not boot into safe mode which I believe it should have?

This would indicate to me that Veeam VSS integartion is not enabled in the backup job settings, or it failed to process the VM but the option was selected to "continue backup even if VSS fails".

[shanes]

I checked that setting and the VSS was enabled and Logs from the job are below from the backup I restored. (anyway Ill try another restore and get back to you)
10 of 10 files processed
Total VM size: 280.00 GB
Processed size: 280.00 GB
Processing rate: 571 MB/s
Backup mode: NBD with changed block tracking
Start time: 8/03/2010 10:00:40 PM
End time: 8/03/2010 10:09:03 PM
Duration: 0:08:22

More importantly for me now thou is the orginal issue with the snapshot problem. I now have situation with a VM being triple its orginal size due to the snapshot playback/cleanup failing. I have been told that the only way to replay the files back to the orginal VM is to clone the system. Is this correct??? Of course due to it being triple the size now I dont have the space to clone.

[Gostev]

Shane, one thing that comes to my mind is using VMware Converter to clone the VM to another storage. But it would be best if you seek for advice on how best to resolve this situation from VMware support, as I am sure this is common issue they have seen many times and may have script, tools and procedures already available.

Also, now that you have upgraded to version 4.1.1, you will be getting a warning each time when snapshot removal fails, so you will not run into this situation again.

[shanes]

Ok, Im getting extremely annoyed wit this issue now. As cloning a box is NOT a solution you should need to do because a backup failed!!!!!

In short VMWARE have stated that unless there is tracking in Snapshot manager there is no way in which you can recover using any tools short of cloning the system. So I have checked snapshot manager and of course Veeam does not seem to update or track here. So VMWARE do NOT have a script that can be used to recover from here. Thus only solution is to Clone!!!!!

So this basically means the following:

If veeam fails for any reason and as you now say version 4.1.1 creates only a "Warning" and 4.1 just continues on, nothing is available to commit your VMDK_1 file back to the orginal VMDK file, which is what the Veeam product has created, as it has not updated snapshot manager and VMWARE can not assist as they need the record in Snapshot manager which is what all the VMWare backup products do and by VMWares support personal Veeam should also do!!!!!

If the above is all true then Im sorry but this is a MAJOR flaw that needs attention and saying that I have to go back to VMWARE to assit is not a solution either...
My issue is here is now I have to expand storage capacity to clone systems at my cost which is just crazy...

[Gostev]

If veeam fails for any reason and as you now say version 4.1.1 creates only a "Warning" and 4.1 just continues on, nothing is available to commit your VMDK_1 file back to the orginal VMDK file, which is what the Veeam product has created, as it has not updated snapshot manager and VMWARE can not assist as they need the record in Snapshot manager which is what all the VMWare backup products do and by VMWares support personal Veeam should also do!!!!

This is incorrect statement by VMware support person due to lack of knowledge on how 3rd party products integrate with VMware. For any 3rd party product, snapshot management is completely isolated to the two API call, CreateSnapshot and RemoveSnapshot. Everything else is handled by VMware (including updating snapshot database records, processing actual removal and so on). 3rd party products are not supposed, and moreover not provided by any means of affecting any part or step of the snapshot removal process.