Discussions specific to the VMware vSphere hypervisor
Post Reply
yossi.sarusi
Lurker
Posts: 2
Liked: never
Joined: Dec 01, 2010 12:42 pm
Full Name: yossi sarusi
Contact:

vm loose pings when runing snapshot

Post by yossi.sarusi »

hi
I just want to know if this thing is a problem? or is it a regular thing with vmware?

when I run veeam job, the vm is loosing a ping or 2...
this thin happen on every vm in my farm.
if this is a problem, then please help me fix it


thanks.

Vitaliy S.
Product Manager
Posts: 24243
Liked: 1859 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vm loose pings when runing snapshot

Post by Vitaliy S. »

Hello Yossi,

This is a regular thing with VMware, you may reporduce this behaviour by playing with snapshot manually. Should more inormation be required, please have a look at this topic: Many VM's dropping packets losing pingstate during rep/back

By the way, what version of ESX(i) host are you running?

Thank you!

yossi.sarusi
Lurker
Posts: 2
Liked: never
Joined: Dec 01, 2010 12:42 pm
Full Name: yossi sarusi
Contact:

Re: vm loose pings when runing snapshot

Post by yossi.sarusi »

we use esx 4.1.0
so this behavior is OK?
what if it will loose connection to 2 nodes in a cluster environmental?
wont it cause the cluster to fall?

thanks

Vitaliy S.
Product Manager
Posts: 24243
Liked: 1859 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vm loose pings when runing snapshot

Post by Vitaliy S. »

I'm not saying that this behaviour is ok, but it is somewhat expected with VMware snapshots. VM can possibly lose pings on snaphsot consolidation/removal procedure while VM is stunned for final snapshot commit, but I do not see how this can cause entire cluster failure.

jcgpro
Lurker
Posts: 1
Liked: never
Joined: May 04, 2011 2:33 pm
Contact:

Ping Timeout Durring Veeam Backup

Post by jcgpro »

[merged]

For some reason I have one VM that will lose network connection while Veeam is deleting the Snapshot. It happens every night during the backup.
When I disable the Veeam Backup Job, the problem goes away.

MattG
Enthusiast
Posts: 39
Liked: never
Joined: Dec 22, 2010 3:50 pm
Full Name: MattG
Location: Philadelphia, PA
Contact:

Re: vm loose pings when runing snapshot

Post by MattG »

How many pings are you losing?

What version of Windows?

What type of storage?

I am experiencing up to a minute freeze when Veeam removes the snapshot (with or without VSS), using NFS and Windows 2008 R2. FYI I have installed the VMware NFS/CBT patch.

-Matt
Twitter: http://twitter.com/#!/matthewgraci

dekkar
Enthusiast
Posts: 38
Liked: 1 time
Joined: Jun 17, 2010 12:21 am
Full Name: Nathan Tong
Contact:

VM unresponsive after removing Veeam Snapshot

Post by dekkar »

[merged]

Hi, I sometimes have a problem when removing a snapshot from a VM.... The vm becomes unresponsive, and it takes around 2+ hours for the snapshot to remove. (gets to 95% then just sits there)


I presume that this is probably a esx issue, but thought that maybe someone here may have more info about it... Ive had a hunt around, and people have confirmed that it happens, but no one seems to know if its fixable.

I dont think its the SAN as a whole, as the other 20 VMs on it run fine.... it seems to be isolated to the particular VM, (or possibly LUN)...

Has anyone seen this before?


Thanks,
Dekkar

joergr
Expert
Posts: 386
Liked: 39 times
Joined: Jun 08, 2010 2:01 pm
Full Name: Joerg Riether
Contact:

Re: vm loose pings when runing snapshot

Post by joergr »

Hi dekkar,

-what ESX version is running this vm?
-what san type and what connection protocol?
-is it nfs (if yes, apply latest vmware patches)?
-is the vm under heavy i/o load during veeam backup? is there any backup guest inside the vm running in addition to the veeam backup and may it be this occurs exactly in the veeam backup window? remember, every change to the source vm has to be comitted when the snapshot is deleted.

And JFYI: There is no thing such as "veeam-snapshots", veeam simply tells the vmware vcenter system or esx machine to do the snapshot.

best regards,
Joerg

dekkar
Enthusiast
Posts: 38
Liked: 1 time
Joined: Jun 17, 2010 12:21 am
Full Name: Nathan Tong
Contact:

Re: vm loose pings when runing snapshot

Post by dekkar »

Hi Joerg, thanks for the reply....

I was pretty sure it wasnt a problem with Veeam.... we are running 4, and have an Equallogic iSCSI SAN.....

THe particular server is an exchange server, which does run a Arcserve backup at night as well. THe server is back now (after around 2 hrs of merging the snap) and it looks like in this case, someone had made a snapshot on the weekend and not merged it....

SO possibly it could also be that the Snapshot veeam made had to be merged into another snapshot.....

joergr
Expert
Posts: 386
Liked: 39 times
Joined: Jun 08, 2010 2:01 pm
Full Name: Joerg Riether
Contact:

Re: vm loose pings when runing snapshot

Post by joergr »

trust me, it is not a veeam issue, veeam only tells vmware to do or remove the virtual snapshot and vmware does...
Yep, please check there are no other snapshots around, that could delay the commit, too. But much much more important: For heavens sake please make sure no other backup guest inside the vm does it´s job exactly at the time when veeam backup does its job. The snapshout would grow SO huge it would take a hell of time to commit it.

best regards,
Joerg

marksmithuk
Enthusiast
Posts: 26
Liked: 3 times
Joined: Apr 15, 2011 2:15 pm
Full Name: Mark Smith
Location: Devon, UK
Contact:

Veeam Replication causing network outage

Post by marksmithuk »

[merged]

Hi

We are using Veeam Backup and Replication 5.0.2.230 on Windows Server 2008 R2 which connects to a VMWare ESXi 4.1.0 (Build 348481) and are having some issues when it replicates our File Server pretty much constantly.

The FIle Server is about 1.3TB so is pretty big and we have noticed than when the replication has finished and VMWare is commiting the SnapShot we are getting a few blips in network connectivity whereby Pings go through the roof and we also get some outage which is causing issues for some apps that connect to it.

Anyone got any ideas why this might be happening ?

Cheers
Mark

Alexey D.

Re: vm loose pings when runing snapshot

Post by Alexey D. »

Hello Mark, please refer to the second post of this thread, it should give you a clue. Thanks!

marksmithuk
Enthusiast
Posts: 26
Liked: 3 times
Joined: Apr 15, 2011 2:15 pm
Full Name: Mark Smith
Location: Devon, UK
Contact:

Re: vm loose pings when runing snapshot

Post by marksmithuk »

Hi Guys

I am running ESXi 4.1.0 Build 348481 as a host with a Windows Server 2008 R2 SP1 guest with File Services running, nothing else.

I have a couple of replication jobs happening at once but on different ESXi Hosts (this FS replication and an Exchange Server replication). We are planning on having this constantly replicate our whole VM Cluster to a seccond SAN at a remove site over a LES link but if it is causing issues during production hours I am not sure what else I could do really.

The replication is happening over a direct SAN connection with the help of the Microsoft iCSCI initiator.

I understand that all Veeam is doing is telling VMWare to create a snapshot and then commit the snapshot so I guess that it is more a VMWare issue rather than Veeam itself but as you guys seem to know so much about how everything links together I thought here would be a good place to start.

Mark

Alexey D.

Re: vm loose pings when runing snapshot

Post by Alexey D. »

Mark, please reach out our support team and send all the logs. I removed your last post because it contained a technical issue - and we can't provide you assistance across the forum due to the lack of information. Thanks for your understanding.

marksmithuk
Enthusiast
Posts: 26
Liked: 3 times
Joined: Apr 15, 2011 2:15 pm
Full Name: Mark Smith
Location: Devon, UK
Contact:

Re: vm loose pings when runing snapshot

Post by marksmithuk »

Fair enough I will give the tech support guys a call.

emachabert
Veeam Vanguard
Posts: 373
Liked: 165 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: vm loose pings when runing snapshot

Post by emachabert »

Hi everyone,

Just to give feedback on what we see on our side, it always good to know what happen to other people.

Running latest build of Veeam and latest one from VMware (4.1.0 381591) we are seeing the same ping lost. (This is the same since 4.0, and before Veeam 5.0)
All our VMs are Linux VM (Centos5), there are 70 VMs , runing on two FiberChannel/SAS SANs (HP MSA2000) on a HP C7000 enclosure.
All the VMs are clustered (heartbeat) for hot failover and all the stuff is monitored by a Zabbix server.

Each morning, when I open my email I have a dozen of email saying VMs were not reachable by ping (at the time of the Veeam Backup).
When I look at the heartbeat logs I can see those lines:
May 20 04:46:30 sso2 heartbeat: [22962]: WARN: 3 lost packet(s) for [sso1.xxx.com] [11185830:11185834]
May 20 04:46:31 sso2 heartbeat: [22962]: info: No pkts missing from sso1.xxx.com!

It has never caused any production issue, as you can see it happens within one second, which is below our cluster switch threshold.

Is seems to be related to the snapshot removal by the way.
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

MarcelB
Influencer
Posts: 10
Liked: never
Joined: Feb 04, 2010 10:32 am
Full Name: Marcel van den Berg
Contact:

Re: vm loose pings when runing snapshot

Post by MarcelB »

I have seen frozen VM's as well and pings lost as well. It happens when VMware commits the snapshot. For a VM which had lots of changes during the time the snapshot file was active (so during the runtime of the backup) it might take a while to commit the snapshot.
I wrote an article about the behaviour on my blog: http://up2v.wordpress.com/2011/05/09/kn ... or-backup/

Best is to run backup during offhours and be carefull when doing replication jobs during business hours. Issues are caused by the way VMware designed snapshots. It is not a Veeam issue.

tomnats
Lurker
Posts: 2
Liked: never
Joined: Aug 25, 2010 6:57 pm
Full Name: Tom Nats
Contact:

[MERGED] Loss of connectivity at the end of backup

Post by tomnats »

I have a customer that has a SQL Server that we're backing up for them. At the end of the Veeam backup, he loses connectivity to this VM from his Web server and gets notified. It's just for 5-10 seconds but it's affecting his customer experience.

We are using application processing with no issues. Here is the latest log:
6 of 6 files processed
Total VM size: 190.00 GB
Processed size: 190.00 GB
Processing rate: 103 MB/s
Backup mode: HOTADD/NBD without changed block tracking
Start time: 8/1/2011 8:00:43 AM
End time: 8/1/2011 8:32:10 AM
Duration: 0:31:27

Right at 8:32 is when he loses connectivity and it's also when the backup is ended and the snapshot is being removed. Is this a VMware thing with the snapshot or is there something I can do to prevent this?

Thanks for any help you can provide,
Tom

sunshineb
Novice
Posts: 6
Liked: never
Joined: Mar 16, 2011 7:18 pm
Contact:

Re: vm loose pings when runing snapshot

Post by sunshineb »

It is happening with us too. It looks like we are going to have to use Backup exec for our sql servers.

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vm loose pings when runing snapshot

Post by Gostev »

Some recommendations can be found in more current topic regarding this issue > Snapshot removal issues of a large VM

mbeckey
Influencer
Posts: 16
Liked: 2 times
Joined: Sep 26, 2012 1:31 am
Full Name: Mark Beckey
Contact:

[MERGED] Guest looses Ping towards the end of Replication jo

Post by mbeckey »

Two of our VM guest’s that are both database servers are getting replicated every hour and they are losing pings at the end of the backup job. Our clients who access the applications are reporting that they are getting logged out during the course of the day. So for a test we turned off replication yesterday morning and there haven’t been any issues since then. The servers are each around 500 GB in size.

Has anyone see issues like this in the past?

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 17 guests