Discussions specific to the VMware vSphere hypervisor
Post Reply
EBoucq
Expert
Posts: 232
Liked: 15 times
Joined: Jan 21, 2011 10:10 am
Full Name: Eric Boucq
Contact:

Snapshots take a while to be created

Post by EBoucq » Jan 18, 2019 10:33 am

Hi,

Correct me if I'm wrong but I thought that a backup job with the guest quiescence enabled triggers a VMware snapshot "without the VM memory".
On a VMware 6.0 infra with a Veeam B&R 9.5 U3, I have a few VMs whose snapshots (during a Veeam Backup) take more than 2 minutes to be created, exactly like if the memory was included. In that same job, other VMs don't have this problem. When I do a test directly from the vCenter (snapshot without memory), I don't have this issue and the snapshot takes a few seconds to be created.
That's why I'm wondering if the issues wouldn't be on Veeam side...
Anyone already had this problem ?
Thanks.
Eric

P.Tide
Product Manager
Posts: 5260
Liked: 459 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshots take a while to be created

Post by P.Tide » Jan 18, 2019 11:35 am

Correct me if I'm wrong but I thought that a backup job with the guest quiescence enabled triggers a VMware snapshot "without the VM memory".
You are correct on this one - Veeam should not snapshot VM memory state if quiescence has been selected.

When I do a test directly from the vCenter (snapshot without memory), I don't have this issue and the snapshot takes a few seconds to be created.
During the test, did you select "Quiesce guest filesystem" checkbox in vSphere UI?

Thanks!

EBoucq
Expert
Posts: 232
Liked: 15 times
Joined: Jan 21, 2011 10:10 am
Full Name: Eric Boucq
Contact:

Re: Snapshots take a while to be created

Post by EBoucq » Jan 21, 2019 10:50 am

Hi,

No, during the manual snapshot test, I didn't select "quiesce guest file system". Should I ?

Thanks,
Eric

P.Tide
Product Manager
Posts: 5260
Liked: 459 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshots take a while to be created

Post by P.Tide » Jan 21, 2019 10:53 am

Yes, please do so. The main idea behind my proposal is to test whether quiescing via VMware will also take that long. I suspect that the in-guest VSS writers might be the culprit.

Thanks!

Regnor
Service Provider
Posts: 319
Liked: 62 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Snapshots take a while to be created

Post by Regnor » Jan 21, 2019 12:00 pm

Is your vSphere 6.0 on a current patchlevel?
I remember having similar problems which were solved patching the hosts.
Also check if on events inside the VMs regarding VSS timeout and if you have other backup jobs or similar tasks running.

EBoucq
Expert
Posts: 232
Liked: 15 times
Joined: Jan 21, 2011 10:10 am
Full Name: Eric Boucq
Contact:

Re: Snapshots take a while to be created

Post by EBoucq » Jan 21, 2019 4:09 pm

Ok, so when I do a snapshot test from the vCenter, with "Quiesce guest file system" enabled, it's super slow.
The version of vSphere is 6.0.0 4192238

P.Tide
Product Manager
Posts: 5260
Liked: 459 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshots take a while to be created

Post by P.Tide » Jan 21, 2019 4:18 pm

That is, it's worth checking VSS writers events as per Regnor's advice.

Thanks!

EBoucq
Expert
Posts: 232
Liked: 15 times
Joined: Jan 21, 2011 10:10 am
Full Name: Eric Boucq
Contact:

Re: Snapshots take a while to be created

Post by EBoucq » Jan 21, 2019 4:26 pm

Thanks PTide.

I checked the VSS writers and didn't see any errors. FYI, it's a Sybase database in the VM so no VSS writer.
But, what I see in the system log are "system failed to flush data to the transaction log" warnings...

P.Tide
Product Manager
Posts: 5260
Liked: 459 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Snapshots take a while to be created

Post by P.Tide » Jan 21, 2019 6:34 pm 1 person likes this post

Such events are usually followed by a more detailed description, such as "device busy" or "A device which does not exist was specified", and ID (might be 140, 57,or some other). To me it seems that the error does not really have anything to do with Veeam, and that its probably worth opening a ticket with MS while awaiting here for other ideas from fellow users.

Thanks!

evilaedmin
Expert
Posts: 104
Liked: 13 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

Re: Snapshots take a while to be created

Post by evilaedmin » Jan 21, 2019 7:45 pm

EBoucq wrote:
Jan 21, 2019 4:26 pm
Thanks PTide.

I checked the VSS writers and didn't see any errors. FYI, it's a Sybase database in the VM so no VSS writer.
But, what I see in the system log are "system failed to flush data to the transaction log" warnings...
Are general VMware snapshot operations on reasonably busy VMs performing to expectations? Any general storage latencies which may be exacerbated by snapshot create/delete operations? Snapshot operations can amplify the amount of write IO. The Windows events you mention could be a symptom of underperforming storage.

Latencies are funny in that at 35 ms everything may appear to be normal, but just a few more % IO can cause 35ms to turn into 100, and everything falls apart. (made up example).

EBoucq
Expert
Posts: 232
Liked: 15 times
Joined: Jan 21, 2011 10:10 am
Full Name: Eric Boucq
Contact:

Re: Snapshots take a while to be created

Post by EBoucq » Jan 22, 2019 10:01 am

Thanks for all your replies.
I'll keep searching as some VMs don't have this issue although having a snapshot created at the same time....

Eric

Post Reply

Who is online

Users browsing this forum: No registered users and 21 guests