vm loose pings when runing snapshot

[yossi.sarusi]

hi
I just want to know if this thing is a problem? or is it a regular thing with vmware?

when I run veeam job, the vm is loosing a ping or 2...
this thin happen on every vm in my farm.
if this is a problem, then please help me fix it


thanks.

[Vitaliy S.]

Hello Yossi,

This is a regular thing with VMware, you may reporduce this behaviour by playing with snapshot manually. Should more inormation be required, please have a look at this topic: Many VM's dropping packets losing pingstate during rep/back

By the way, what version of ESX(i) host are you running?

Thank you!

[yossi.sarusi]

we use esx 4.1.0
so this behavior is OK?
what if it will loose connection to 2 nodes in a cluster environmental?
wont it cause the cluster to fall?

thanks

[Vitaliy S.]

I'm not saying that this behaviour is ok, but it is somewhat expected with VMware snapshots. VM can possibly lose pings on snaphsot consolidation/removal procedure while VM is stunned for final snapshot commit, but I do not see how this can cause entire cluster failure.

[jcgpro]

[merged]

For some reason I have one VM that will lose network connection while Veeam is deleting the Snapshot. It happens every night during the backup.
When I disable the Veeam Backup Job, the problem goes away.

[MattG]

How many pings are you losing?

What version of Windows?

What type of storage?

I am experiencing up to a minute freeze when Veeam removes the snapshot (with or without VSS), using NFS and Windows 2008 R2. FYI I have installed the VMware NFS/CBT patch.

-Matt

[dekkar]

[merged]

Hi, I sometimes have a problem when removing a snapshot from a VM.... The vm becomes unresponsive, and it takes around 2+ hours for the snapshot to remove. (gets to 95% then just sits there)


I presume that this is probably a esx issue, but thought that maybe someone here may have more info about it... Ive had a hunt around, and people have confirmed that it happens, but no one seems to know if its fixable.

I dont think its the SAN as a whole, as the other 20 VMs on it run fine.... it seems to be isolated to the particular VM, (or possibly LUN)...

Has anyone seen this before?


Thanks,
Dekkar

[joergr]

Hi dekkar,

-what ESX version is running this vm?
-what san type and what connection protocol?
-is it nfs (if yes, apply latest vmware patches)?
-is the vm under heavy i/o load during veeam backup? is there any backup guest inside the vm running in addition to the veeam backup and may it be this occurs exactly in the veeam backup window? remember, every change to the source vm has to be comitted when the snapshot is deleted.

And JFYI: There is no thing such as "veeam-snapshots", veeam simply tells the vmware vcenter system or esx machine to do the snapshot.

best regards,
Joerg

[dekkar]

Hi Joerg, thanks for the reply....

I was pretty sure it wasnt a problem with Veeam.... we are running 4, and have an Equallogic iSCSI SAN.....

THe particular server is an exchange server, which does run a Arcserve backup at night as well. THe server is back now (after around 2 hrs of merging the snap) and it looks like in this case, someone had made a snapshot on the weekend and not merged it....

SO possibly it could also be that the Snapshot veeam made had to be merged into another snapshot.....

[joergr]

trust me, it is not a veeam issue, veeam only tells vmware to do or remove the virtual snapshot and vmware does...
Yep, please check there are no other snapshots around, that could delay the commit, too. But much much more important: For heavens sake please make sure no other backup guest inside the vm does it´s job exactly at the time when veeam backup does its job. The snapshout would grow SO huge it would take a hell of time to commit it.

best regards,
Joerg

[marksmithuk]

[merged]

Hi

We are using Veeam Backup and Replication 5.0.2.230 on Windows Server 2008 R2 which connects to a VMWare ESXi 4.1.0 (Build 348481) and are having some issues when it replicates our File Server pretty much constantly.

The FIle Server is about 1.3TB so is pretty big and we have noticed than when the replication has finished and VMWare is commiting the SnapShot we are getting a few blips in network connectivity whereby Pings go through the roof and we also get some outage which is causing issues for some apps that connect to it.

Anyone got any ideas why this might be happening ?

Cheers
Mark

[Alexey D.]

Hello Mark, please refer to the second post of this thread, it should give you a clue. Thanks!

[marksmithuk]

Hi Guys

I am running ESXi 4.1.0 Build 348481 as a host with a Windows Server 2008 R2 SP1 guest with File Services running, nothing else.

I have a couple of replication jobs happening at once but on different ESXi Hosts (this FS replication and an Exchange Server replication). We are planning on having this constantly replicate our whole VM Cluster to a seccond SAN at a remove site over a LES link but if it is causing issues during production hours I am not sure what else I could do really.

The replication is happening over a direct SAN connection with the help of the Microsoft iCSCI initiator.

I understand that all Veeam is doing is telling VMWare to create a snapshot and then commit the snapshot so I guess that it is more a VMWare issue rather than Veeam itself but as you guys seem to know so much about how everything links together I thought here would be a good place to start.

Mark

[Alexey D.]

Mark, please reach out our support team and send all the logs. I removed your last post because it contained a technical issue - and we can't provide you assistance across the forum due to the lack of information. Thanks for your understanding.

[marksmithuk]

Fair enough I will give the tech support guys a call.

[emachabert]

Hi everyone,

Just to give feedback on what we see on our side, it always good to know what happen to other people.

Running latest build of Veeam and latest one from VMware (4.1.0 381591) we are seeing the same ping lost. (This is the same since 4.0, and before Veeam 5.0)
All our VMs are Linux VM (Centos5), there are 70 VMs , runing on two FiberChannel/SAS SANs (HP MSA2000) on a HP C7000 enclosure.
All the VMs are clustered (heartbeat) for hot failover and all the stuff is monitored by a Zabbix server.

Each morning, when I open my email I have a dozen of email saying VMs were not reachable by ping (at the time of the Veeam Backup).
When I look at the heartbeat logs I can see those lines:
May 20 04:46:30 sso2 heartbeat: [22962]: WARN: 3 lost packet(s) for [sso1.xxx.com] [11185830:11185834]
May 20 04:46:31 sso2 heartbeat: [22962]: info: No pkts missing from sso1.xxx.com!

It has never caused any production issue, as you can see it happens within one second, which is below our cluster switch threshold.

Is seems to be related to the snapshot removal by the way.

[MarcelB]

I have seen frozen VM's as well and pings lost as well. It happens when VMware commits the snapshot. For a VM which had lots of changes during the time the snapshot file was active (so during the runtime of the backup) it might take a while to commit the snapshot.
I wrote an article about the behaviour on my blog: http://up2v.wordpress.com/2011/05/09/kn ... or-backup/

Best is to run backup during offhours and be carefull when doing replication jobs during business hours. Issues are caused by the way VMware designed snapshots. It is not a Veeam issue.

[tomnats]

I have a customer that has a SQL Server that we're backing up for them. At the end of the Veeam backup, he loses connectivity to this VM from his Web server and gets notified. It's just for 5-10 seconds but it's affecting his customer experience.

We are using application processing with no issues. Here is the latest log:
6 of 6 files processed
Total VM size: 190.00 GB
Processed size: 190.00 GB
Processing rate: 103 MB/s
Backup mode: HOTADD/NBD without changed block tracking
Start time: 8/1/2011 8:00:43 AM
End time: 8/1/2011 8:32:10 AM
Duration: 0:31:27

Right at 8:32 is when he loses connectivity and it's also when the backup is ended and the snapshot is being removed. Is this a VMware thing with the snapshot or is there something I can do to prevent this?

Thanks for any help you can provide,
Tom

[sunshineb]

It is happening with us too. It looks like we are going to have to use Backup exec for our sql servers.

[Gostev]

Some recommendations can be found in more current topic regarding this issue > Snapshot removal issues of a large VM

[mbeckey]

Two of our VM guest’s that are both database servers are getting replicated every hour and they are losing pings at the end of the backup job. Our clients who access the applications are reporting that they are getting logged out during the course of the day. So for a test we turned off replication yesterday morning and there haven’t been any issues since then. The servers are each around 500 GB in size.

Has anyone see issues like this in the past?