Comprehensive data protection for all workloads
Post Reply
varrus999
Influencer
Posts: 20
Liked: never
Joined: Sep 01, 2009 2:32 am
Full Name: Mike Veneziale
Contact:

VM Crashing due to snapshot issues

Post by varrus999 »

Hi,

We're just upgraded to VEEAM 4.1 and have been running 3.1 until recently.

A major issue we are having is with the snapshot process. Every now and again I get a failed backup, and a VM crashes. Sometimes it's just a matter of powering it back on, but often there are snapshot config issues where the parent/child structure is broken between snapshot files. I have learned a bit about fixing these issues with VMware support...somtimes it's just a matter of editing/restoring the VMX file to point ot the original disk, other times it's more complicated. My bottom line, I need to find out why this happens. I don't think it's a VEEAM issue so much as an issue with our platform. VEEAM is simply using ESX technology to create/remove snapshots. Am wondering if anyone else has experienced this? I have spoken to VMware about it but haven't gotten far yet. I am working on the theory that this is a SAN or storage config issue - we are getting IO errors on various windows VMs in our environment and I have been trying to troubleshoot - my main difficulty is in isolating which performance counters to look at. If anyone else has experienced these issues while using VEEAM or other similar backup products, I'd love to hear from you.

Other VEEAM issues I am having:
Speed is dismal - I get anywhere from 12 to 50 MB/s, with most jobs running about about 25 MB/s. Any suggestions on how I could improve this? We are running direct SAN backups over fibre to a physical server (which is running VEEAM) with direct SCSI attached storage. I tried to run a backup job directly to a folder on C drive to see of the attached SCSI was causing a bottleneck, but the job ran just as slow. We are looking at running backups directly to a SAN volume to see if this helps. Does anyone know if VEEAM will be smart enough to move the data directly, rather than loop it through the backup server?

Is there a difference between VCB and vStorage API backup? Is one faster / safer than the other?

Does 'safe snapshot removal' apply to ESX 3.5 with the latest update? Does it matter if I check this option?

Should I use the option to quiesce the file system, or VSS, or both? VMware tells me we should quisce to take 'crash consistent backups', but I have also read that it will freeze the VM for so long that it may cause isues in highly transactive servers like SQL. The last thing I want to do is introduce more unstability in our environment.

Any insght would be helpful.

Thanks!

Mike.

Gostev
SVP, Product Management
Posts: 26880
Liked: 4359 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VM Crashing due to snapshot issues

Post by Gostev »

Hello Mike, you should use Veeam VSS for all Windows VMs to avoid crashes during snapshots. Other methods may indeed results in crashes, corruptions or inconsistence (both on source VM, and in backups), and this has been reported before. Moreover, if you ever have issues with Veeam VSS, we will be able to fully support you, because we own this code (unlike built-in quiescence mechanisms VMware provides).

Using Veeam VSS automatically disables VMware quiescence, but you can also disable VMware Tools quiescence specifically in the advanced job settings to prevent it from kicking in in case Veeam VSS fails for any reason.

As for your other questions:

From your description it sounds like bottleneck is either data retrieval speed from the storage, or lack of CPU on Veeam Backup server. Keep in mind though, that for iSCSI SAN, 40-50MB/s is "normal" full backup performance. It is only FC SAN which can give you much better job throughtput. With iSCSI SAN, you would want to run 3-4 jobs in parallel to fully utilize your SAN throughtput. We have some existing threads discussing iSCSI SAN backup performance, check them out for more information. Also check the session results, may be your job fails over to network because of SAN access issues?

The data will always "loop through" Veeam Backup server because it does actual data processing (compression/deduplication). But overall, the backup process will be completely LAN free in both scenarios (data will only be moved across SAN fabric, never hitting the network or touching your ESX hosts).

Your vStorage API question is covered in the stickied FAQ thread, in short - you really want to go with vStorage in all cases.

Safe snapshot removal feature is well covered in the User Guide, you don't really need to use it with ESX 3.5 U2 or later hosts, this is for legacy ESX versions only. This is why it is disabled by default.

Hope this helps!
Thank you.

ceebee
Influencer
Posts: 17
Liked: never
Joined: Dec 24, 2009 2:39 am
Full Name: Chet
Contact:

Re: VM Crashing due to snapshot issues

Post by ceebee »

We have the same issue in terms of the speed being anywhere between 2 MB to 150 MB. I cant get any answer from Veeam one way or the other. I have given up and will be soon looking at other alternatives.
Gostev-
Can we do something about improving your documentation in terms of commonly found issues and a quick how-to on the most commonly used set-ups?

Gostev
SVP, Product Management
Posts: 26880
Liked: 4359 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VM Crashing due to snapshot issues

Post by Gostev »

Like online support KB?
Yes, I've been pushing for it very much lately :)

ceebee
Influencer
Posts: 17
Liked: never
Joined: Dec 24, 2009 2:39 am
Full Name: Chet
Contact:

Re: VM Crashing due to snapshot issues

Post by ceebee »

Any idea on when this is going to be available?

jbsengineer
Enthusiast
Posts: 25
Liked: 3 times
Joined: Nov 10, 2009 2:45 pm
Contact:

Re: VM Crashing due to snapshot issues

Post by jbsengineer »

I have experienced the same VM crashing due so Snapshot problems as well. It has nothing to do with VSS as it's not the guest OS that crashes. The guest is powered down immediately due the underlying disk structure gets "mixed" up. The past few times this has happened I need to manually modify the VMX, VMSD, and VMSN files as the VM would say the disk is currupted.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], brent@viptsg.com, Egor Yakovlev, Google [Bot], RobTurk and 101 guests