V8 backups causing VMs to crash and restart?

[luckyinfil]

Prior to V8, I've never had issues with VMs restarting during a backup - The backup would either succeed or fail but the VM would remain running.

After upgrading to V8, i'm starting to see some failures affect the running state of the VM.

This is the error in the Veeam backup job:

12/16/2014 11:43:12 AM :: Queued for processing at 12/16/2014 11:43:12 AM
12/16/2014 11:43:12 AM :: Required backup infrastructure resources have been assigned
12/16/2014 11:43:18 AM :: VM processing started at 12/16/2014 11:43:18 AM
12/16/2014 11:43:18 AM :: VM size: 40.0 GB
12/16/2014 11:43:28 AM :: Creating VM snapshot
12/16/2014 11:43:39 AM :: Error: An error occurred while saving the snapshot: Unable to save snapshot file.
12/16/2014 11:43:39 AM :: Network traffic verification detected no corrupted blocks
12/16/2014 11:43:39 AM :: Processing finished with errors at 12/16/2014 11:43:39 AM


Looking in vsphere, I see this error on the VM:

Error message from vmsrv: An operation required the virtual machine to quiesce and
the virtual machine was unable to continue running.

The VM then shuts down and HA restarts it.


Anyone else having similar issues? I've opened a support ticket (Case # 00707139) already but generally find the veeam forums more helpful.

[foggy]

Are you able to snapshot this VM via vSphere Client?

[luckyinfil]

Yes. But if I rerun the job after I test this snapshot, it goes through fine. Something Veeam is doing is periodically crashing VMs, maybe whatever it's trying to do to quiecse the VM.

[tsightler]

I'm thinking VMware HA may be incorrectly detecting the VSS freeze as a crash and thus restarting the VM. Are you using VMware tools quiescence or Veeam AAIP for the job? Anything unusual about this VM, like perhaps some physical RDM attached to it or anything?

[tsightler]

Also, any chance you are low on free space at that moment in time (perhaps many VMs being in snapshot using the space)? I believe the error can occur if the system starts to take a snapshot but then does not have enough space to complete the operation. Basically, vCenter will return an error that it is unable to save the snapshot file (which is what Veeam is reporting), and since there wasn't enough space the VM is unable to leave it's quiesced state so the other errors follow.

[luckyinfil]

Also, any chance you are low on free space at that moment in time (perhaps many VMs being in snapshot using the space)? I believe the error can occur if the system starts to take a snapshot but then does not have enough space to complete the operation. Basically, vCenter will return an error that it is unable to save the snapshot file (which is what Veeam is reporting), and since there wasn't enough space the VM is unable to leave it's quiesced state so the other errors follow.

Not low on free space.

[luckyinfil]

I'm thinking VMware HA may be incorrectly detecting the VSS freeze as a crash and thus restarting the VM. Are you using VMware tools quiescence or Veeam AAIP for the job? Anything unusual about this VM, like perhaps some physical RDM attached to it or anything?

What's AAIP? It's just a standard Veeam backup job with most of the default settings. Nothing special about this VM, just a 2003 with regular VMDKs.. I've seen it on other machines that are also regular.

[VladV]

If you're on 5.5 check this http://kb.vmware.com/selfservice/micros ... Id=2096282

[tsightler]

Sorry, AAIP = Application Aware Image Processing. Basically, is the "Enable application-aware processing" option ticked on the "Guest Processing" tab of the job as opposed to leaving this off and relying on VMware tools quiescence.

I'd suggest looking at the VM logs and see what caused this error:
Error: An error occurred while saving the snapshot: Unable to save snapshot file.
This is almost certainly the "trigger", as if the vCenter snapshot operation fails with this error the VM will basically not resume from quiescence. If you browse out to VM directory on the datastore there should be some good logs from around that time that will have more info on exactly what went wrong that VMware couldn't create the snapshot file. I've never seen it be anything other than space or some weird VMware queiscence issues (like the old SYNC driver still being installed), but there might be other possibilities I don't know about. Or there could be some other cause from the Veeam side, but that message is from VMware so the VM logs might hold some extra clues.