Snapshot removal issues of a large VM

VMware specific discussions

[MERGED] Packets Dropping During Replication and Backup - ES

Veeam Logoby Benji.Sawyer » Fri May 15, 2015 1:25 pm

When a backup job is running we are getting notifications that our VM's are occasionally dropping a small number of packets (never more than 1,000). We are running Veaam 8.0.0.817 and ESXi 5.5.0.0 2068190. The drops happen throughout the time the backup or replication job is running and one VM may experience pack drop 2 or 3 times (it normally takes 30 minutes to run the backup and replication jobs).

I've seen some posting where this was kind of expected behavior in older version of ESXi, but nothing about v 5.5. Is this still something that is expected? Is it actually a problem? I don't think we are using VSS in our job set up (no exchange or SQL), but would that fix this (or should we be using it anyways as a best practice)?

Thanks for taking the time to read this and thanks for your help
Benji.Sawyer
Novice
 
Posts: 5
Liked: never
Joined: Fri May 15, 2015 1:13 pm

Re: Snapshot removal issues of a large VM

Veeam Logoby foggy » Fri May 15, 2015 2:21 pm

Do you probably see this during the snapshot commit operation?
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Snapshot removal issues of a large VM

Veeam Logoby Benji.Sawyer » Mon May 18, 2015 1:07 am

Foogy,

I'm going to assume you were directing that comment at me, since my question has been merged into this beast of a post (although I'm not sure it really should be). How would I go about checking to see if this occurred during a commit process? My gut instinct would be to say no since we'll get the message of dropped packets for the same VM several times (minutes apart) during a single backup, but if you could point me in the direction of how I would check to see if it does coincide with the commit I'll take a look.

Thanks
Benji.Sawyer
Novice
 
Posts: 5
Liked: never
Joined: Fri May 15, 2015 1:13 pm

Re: Snapshot removal issues of a large VM

Veeam Logoby v.Eremin » Mon May 18, 2015 8:36 am

Check the job session statistics and see whether the problem happens during step that is called "Removing VM snapshot". Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Snapshot removal issues of a large VM

Veeam Logoby Benji.Sawyer » Tue May 19, 2015 12:20 am

There is no timestamp in there for the individual steps, but it looks like the answer is sometimes. It might also be happening during the Remove step but it is definitely happening at other times even when the specific VM isn't actively being backed up.
Benji.Sawyer
Novice
 
Posts: 5
Liked: never
Joined: Fri May 15, 2015 1:13 pm

Re: Snapshot removal issues of a large VM

Veeam Logoby v.Eremin » Tue May 19, 2015 9:25 am

What do you mean by "when the specific VM isn't actively being backed up"? VMs sometimes drop packets, even when no backup job are running against them? Anyway, it might be worth trying to reproduce that by manually creating a snapshot manually, keeping it long enough (the time similar to the one backup job takes), deleting it and seeing whether the issue occurs. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Snapshot removal issues of a large VM

Veeam Logoby Benji.Sawyer » Tue May 19, 2015 3:08 pm

Sorry for the confusion.

What I mean is that the entire job runs for ~30 minutes. During this the individual VMs only run a for a few minutes each (depending on the size). So VM1 may be the first to back up and it will drop packets while it is actually backing up and it might drop some more later after it is finished but the entire job is still running (say VM 4 is actually backing up). Does that make sense?
Benji.Sawyer
Novice
 
Posts: 5
Liked: never
Joined: Fri May 15, 2015 1:13 pm

Re: Snapshot removal issues of a large VM

Veeam Logoby v.Eremin » Wed May 20, 2015 8:14 am

Got it. Though, once the backup is finished for particular VM, Veeam doesn't touch any longer, so, hardly packet drops might be related to VB&R in this case (if, of course, your time observations are correct).

Anyway, I'd still recommend performing manual test that I've described just to see whether the issue can or cannot be reproduced without Veeam presence.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Snapshot removal issues of a large VM

Veeam Logoby Benji.Sawyer » Fri May 22, 2015 3:20 pm

v.Eremin,
Thanks for the suggestion. When I did a manual snapshot it also experienced packet loss, so I'm guessing this is really a VMWare issues, instead of Veeam, right?
Benji.Sawyer
Novice
 
Posts: 5
Liked: never
Joined: Fri May 15, 2015 1:13 pm

Re: Snapshot removal issues of a large VM

Veeam Logoby foggy » Fri May 22, 2015 3:43 pm

Yes, since you can rule out Veeam B&R from the picture completely.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

[MERGED] Need to reduce stun time

Veeam Logoby unsichtbarre » Mon Jun 15, 2015 3:11 pm

I am looking for strategies to reduce snapshot stun time. On a 15 minute replication, it is perceived that the snapshot stun is an issue and I would like to consider any and all strategies for reducing it.

Any Ideas? Thanks,
-J
unsichtbarre
Enthusiast
 
Posts: 61
Liked: 14 times
Joined: Mon Mar 08, 2010 4:05 pm
Full Name: John Borhek

Re: Snapshot removal issues of a large VM

Veeam Logoby foggy » Mon Jun 15, 2015 4:00 pm

John, some hints on reducing snapshot commit time are discussed in this huge thread. Also, consider replicating from backups that does not involve snapshots of the production VMs at all.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Snapshot removal issues of a large VM

Veeam Logoby unsichtbarre » Mon Jun 15, 2015 4:39 pm

Even so, backup would have to run every 15 minutes to achieve desired RPO. Stun would remain an issue.

P.S. I thought that posting on such an old/long thread was discouraged.
unsichtbarre
Enthusiast
 
Posts: 61
Liked: 14 times
Joined: Mon Mar 08, 2010 4:05 pm
Full Name: John Borhek

Re: Snapshot removal issues of a large VM

Veeam Logoby foggy » Tue Jun 16, 2015 9:10 am

unsichtbarre wrote:Even so, backup would have to run every 15 minutes to achieve desired RPO. Stun would remain an issue.

Right, I missed the frequency thing. What kind of source storage do you have? Any chance of using storage snapshots for replication?

unsichtbarre wrote:P.S. I thought that posting on such an old/long thread was discouraged.

We always try to keep similar discussions in one place for more effective search.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Snapshot removal issues of a large VM

Veeam Logoby michael.johnson » Wed Jun 17, 2015 5:23 pm

We are noticing the same sort of issues with vm snapshots. The interesting for us is this just started happening when we updated vSphere with the latest updates (v 5.5.0, 2718055). It was about a month ago and ever since that time we have see this issue.

What version of vSphere 5.5 are you on? 2718055?

Thanks,
-Mike
michael.johnson
Lurker
 
Posts: 1
Liked: never
Joined: Wed Jun 17, 2015 5:14 pm
Full Name: Michael Johnson

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 10 guests