Host-based backup of VMware vSphere VMs.
Post Reply
EBoucq
Service Provider
Posts: 263
Liked: 17 times
Joined: Jan 21, 2011 10:10 am
Full Name: Eric Boucq
Contact:

Snapshots take a while to be created

Post by EBoucq »

Hi,

Correct me if I'm wrong but I thought that a backup job with the guest quiescence enabled triggers a VMware snapshot "without the VM memory".
On a VMware 6.0 infra with a Veeam B&R 9.5 U3, I have a few VMs whose snapshots (during a Veeam Backup) take more than 2 minutes to be created, exactly like if the memory was included. In that same job, other VMs don't have this problem. When I do a test directly from the vCenter (snapshot without memory), I don't have this issue and the snapshot takes a few seconds to be created.
That's why I'm wondering if the issues wouldn't be on Veeam side...
Anyone already had this problem ?
Thanks.
Eric
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshots take a while to be created

Post by PTide »

Correct me if I'm wrong but I thought that a backup job with the guest quiescence enabled triggers a VMware snapshot "without the VM memory".
You are correct on this one - Veeam should not snapshot VM memory state if quiescence has been selected.

When I do a test directly from the vCenter (snapshot without memory), I don't have this issue and the snapshot takes a few seconds to be created.
During the test, did you select "Quiesce guest filesystem" checkbox in vSphere UI?

Thanks!
EBoucq
Service Provider
Posts: 263
Liked: 17 times
Joined: Jan 21, 2011 10:10 am
Full Name: Eric Boucq
Contact:

Re: Snapshots take a while to be created

Post by EBoucq »

Hi,

No, during the manual snapshot test, I didn't select "quiesce guest file system". Should I ?

Thanks,
Eric
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshots take a while to be created

Post by PTide »

Yes, please do so. The main idea behind my proposal is to test whether quiescing via VMware will also take that long. I suspect that the in-guest VSS writers might be the culprit.

Thanks!
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Snapshots take a while to be created

Post by Regnor »

Is your vSphere 6.0 on a current patchlevel?
I remember having similar problems which were solved patching the hosts.
Also check if on events inside the VMs regarding VSS timeout and if you have other backup jobs or similar tasks running.
EBoucq
Service Provider
Posts: 263
Liked: 17 times
Joined: Jan 21, 2011 10:10 am
Full Name: Eric Boucq
Contact:

Re: Snapshots take a while to be created

Post by EBoucq »

Ok, so when I do a snapshot test from the vCenter, with "Quiesce guest file system" enabled, it's super slow.
The version of vSphere is 6.0.0 4192238
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshots take a while to be created

Post by PTide »

That is, it's worth checking VSS writers events as per Regnor's advice.

Thanks!
EBoucq
Service Provider
Posts: 263
Liked: 17 times
Joined: Jan 21, 2011 10:10 am
Full Name: Eric Boucq
Contact:

Re: Snapshots take a while to be created

Post by EBoucq »

Thanks PTide.

I checked the VSS writers and didn't see any errors. FYI, it's a Sybase database in the VM so no VSS writer.
But, what I see in the system log are "system failed to flush data to the transaction log" warnings...
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshots take a while to be created

Post by PTide » 1 person likes this post

Such events are usually followed by a more detailed description, such as "device busy" or "A device which does not exist was specified", and ID (might be 140, 57,or some other). To me it seems that the error does not really have anything to do with Veeam, and that its probably worth opening a ticket with MS while awaiting here for other ideas from fellow users.

Thanks!
evilaedmin
Expert
Posts: 176
Liked: 30 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

Re: Snapshots take a while to be created

Post by evilaedmin »

EBoucq wrote: Jan 21, 2019 4:26 pm Thanks PTide.

I checked the VSS writers and didn't see any errors. FYI, it's a Sybase database in the VM so no VSS writer.
But, what I see in the system log are "system failed to flush data to the transaction log" warnings...
Are general VMware snapshot operations on reasonably busy VMs performing to expectations? Any general storage latencies which may be exacerbated by snapshot create/delete operations? Snapshot operations can amplify the amount of write IO. The Windows events you mention could be a symptom of underperforming storage.

Latencies are funny in that at 35 ms everything may appear to be normal, but just a few more % IO can cause 35ms to turn into 100, and everything falls apart. (made up example).
EBoucq
Service Provider
Posts: 263
Liked: 17 times
Joined: Jan 21, 2011 10:10 am
Full Name: Eric Boucq
Contact:

Re: Snapshots take a while to be created

Post by EBoucq »

Thanks for all your replies.
I'll keep searching as some VMs don't have this issue although having a snapshot created at the same time....

Eric
Post Reply

Who is online

Users browsing this forum: Google Feedfetcher and 87 guests