Snapshot Commit with VSS causes abnormal VM unresponsiveness

[dougsoltesz]

This has been opened under case #00192127

The support engineer thought it would be a good forum post.

Problem:
Only when Enable Application-aware image processing is checked off for a backup job the VM snapshot commit takes minutes and the VM is not responsive during this time.

Background:
Veeam backups run fine with Application aware image processing disabled. Taking a snapshot only takes a second and releasing a snapshot takes minimal time with no noticeable VM downtime. As soon as VSS in enabled in the Guest processing tab, backups run the same as before, but the releasing (commit) of a snapshot takes significantly longer and the VM will not respond for 30-60 seconds. Disabling VSS in the Guest processing tab does not help. Once enabled this problem persists on a VM even if the backup job is deleted along with data and recreated.

Extra background:
Taking a snapshot with vCenter and committing it runs just fine on the same VM. Even if Quiesce guest file system is checked. The snapshot only takes a second. Also if the VM is powered off and Veeam backup is run the snapshot commit takes a considerable amount of time once VSS is enabled.

This occurs only when using vCenter Server (mine a virtual appliance), and not when connecting direct to the host.

My environment consists of:
ESXi host running 5.1.0 Build 1021289
vCenter Appliance running 5.1.0.5300 Build 947940

It would seem that since Veeam is not actually doing the snapshot commit, but merely calling a commit through vCenter, that the way the commit is called changes once VSS is enabled. Since VSS works when backups are directed to the ESXi hosts (and I'm guessing that those running vCenter 5.1 on Windows don't have this issue) that the issue must be with the vCenter appliance.

Has anyone else had problems like this?

Doug

[dellock6]

Hi Doug, I'm not sure VSS is involved at all during the commit of a snapshot, even if the snapshot itself was made using VSS. VSS is used to freeze I/O in a Windows compliant program to guarantee data consistency, but to my understanding VSS is not invoked during the commit of the snapshot itself.
Maybe the timeout during the snapshot removal is the "usual" stun/unstun problem coming out from VMware itself?

Luca.

[dougsoltesz]

Luca,

Thanks, but I doubt that this is the "usual" stun/unstun problem from VMware. If that was the case I should be able to reproduce this by triggering a stun/unstun from vCenter.

Snapshots taken/release from vCenter, the host, or Veeam before VSS is enabled all work fine. Having at most 2 seconds of unresponsive time. However, after VSS is enabled the unresponsive time only during a Veeam backup snapshot commit is 30-60 seconds. This is even evident if the VM is powered off.

VSS needs to trigger several functions on snapshot commit if Veeam is to truncate the transaction logs, etc in the VM. I am not saying that this is the fault of Veeam, I agree that it is a VMware issue, but is triggered by a specific API call that Veeam is making that is different than the call they make before VSS is enabled and different than that made by vCenter.

I will next try to do a packet capture, or switch to vCenter running on a Windows host. I just wanted to see if anyone else had my problem, a solution, or a something they wanted me to test against the vCenter Appliance before I move vCenter to Windows. (I'm assuming this will work fine with the Windows vCenter 5.1 otherwise, I would think there would be more people posting about this.)

Doug

[dellock6]

Yep, mine was only a supposition, no problem in beeing wrong.

VSS sadly is a hard beast to manage, some implementations are good, some are not, and it all depends on the software vendor offering it. I'm dealing for example right in this days with an Oracle 11g installation on Windows, and VSS does not work "out of the box" like in microsoft, you need to make adjustments in the database to make it work...

that's to say, we first need to find out which application you are dealing with (AD, Exchange, SQL...) and move from here. For example, your case seems to exclude another problem with VSS coming from high I/O VMs, since you said the same problem happens when the VM is powered of (this is even stranger...).

Luca.

[Gostev]

Looks like the "usual" stun/unstun problem from VMware, except more changed data to commit - resulting in commit taking longer, and causing associated problems. I would check out the snapshot file size before commit with and without VSS enabled.