VM Crashing due to snapshot issues

[varrus999]

Hi,

We're just upgraded to VEEAM 4.1 and have been running 3.1 until recently.

A major issue we are having is with the snapshot process. Every now and again I get a failed backup, and a VM crashes. Sometimes it's just a matter of powering it back on, but often there are snapshot config issues where the parent/child structure is broken between snapshot files. I have learned a bit about fixing these issues with VMware support...somtimes it's just a matter of editing/restoring the VMX file to point ot the original disk, other times it's more complicated. My bottom line, I need to find out why this happens. I don't think it's a VEEAM issue so much as an issue with our platform. VEEAM is simply using ESX technology to create/remove snapshots. Am wondering if anyone else has experienced this? I have spoken to VMware about it but haven't gotten far yet. I am working on the theory that this is a SAN or storage config issue - we are getting IO errors on various windows VMs in our environment and I have been trying to troubleshoot - my main difficulty is in isolating which performance counters to look at. If anyone else has experienced these issues while using VEEAM or other similar backup products, I'd love to hear from you.

Other VEEAM issues I am having:
Speed is dismal - I get anywhere from 12 to 50 MB/s, with most jobs running about about 25 MB/s. Any suggestions on how I could improve this? We are running direct SAN backups over fibre to a physical server (which is running VEEAM) with direct SCSI attached storage. I tried to run a backup job directly to a folder on C drive to see of the attached SCSI was causing a bottleneck, but the job ran just as slow. We are looking at running backups directly to a SAN volume to see if this helps. Does anyone know if VEEAM will be smart enough to move the data directly, rather than loop it through the backup server?

Is there a difference between VCB and vStorage API backup? Is one faster / safer than the other?

Does 'safe snapshot removal' apply to ESX 3.5 with the latest update? Does it matter if I check this option?

Should I use the option to quiesce the file system, or VSS, or both? VMware tells me we should quisce to take 'crash consistent backups', but I have also read that it will freeze the VM for so long that it may cause isues in highly transactive servers like SQL. The last thing I want to do is introduce more unstability in our environment.

Any insght would be helpful.

Thanks!

Mike.

[Gostev]

Hello Mike, you should use Veeam VSS for all Windows VMs to avoid crashes during snapshots. Other methods may indeed results in crashes, corruptions or inconsistence (both on source VM, and in backups), and this has been reported before. Moreover, if you ever have issues with Veeam VSS, we will be able to fully support you, because we own this code (unlike built-in quiescence mechanisms VMware provides).

Using Veeam VSS automatically disables VMware quiescence, but you can also disable VMware Tools quiescence specifically in the advanced job settings to prevent it from kicking in in case Veeam VSS fails for any reason.

As for your other questions:

From your description it sounds like bottleneck is either data retrieval speed from the storage, or lack of CPU on Veeam Backup server. Keep in mind though, that for iSCSI SAN, 40-50MB/s is "normal" full backup performance. It is only FC SAN which can give you much better job throughtput. With iSCSI SAN, you would want to run 3-4 jobs in parallel to fully utilize your SAN throughtput. We have some existing threads discussing iSCSI SAN backup performance, check them out for more information. Also check the session results, may be your job fails over to network because of SAN access issues?

The data will always "loop through" Veeam Backup server because it does actual data processing (compression/deduplication). But overall, the backup process will be completely LAN free in both scenarios (data will only be moved across SAN fabric, never hitting the network or touching your ESX hosts).

Your vStorage API question is covered in the stickied FAQ thread, in short - you really want to go with vStorage in all cases.

Safe snapshot removal feature is well covered in the User Guide, you don't really need to use it with ESX 3.5 U2 or later hosts, this is for legacy ESX versions only. This is why it is disabled by default.

Hope this helps!
Thank you.

[ceebee]

We have the same issue in terms of the speed being anywhere between 2 MB to 150 MB. I cant get any answer from Veeam one way or the other. I have given up and will be soon looking at other alternatives.
Gostev-
Can we do something about improving your documentation in terms of commonly found issues and a quick how-to on the most commonly used set-ups?

[Gostev]

Like online support KB?
Yes, I've been pushing for it very much lately

[ceebee]

Any idea on when this is going to be available?

[jbsengineer]

I have experienced the same VM crashing due so Snapshot problems as well. It has nothing to do with VSS as it's not the guest OS that crashes. The guest is powered down immediately due the underlying disk structure gets "mixed" up. The past few times this has happened I need to manually modify the VMX, VMSD, and VMSN files as the VM would say the disk is currupted.