Snapshot removal issues of a large VM

[DavidReimers]

Nope, I did know Vitality meant VMware Support - typo on my part. I'll log a call with VMware.

[hakimbhungra]

Hi,

We are facing issue on Red hat VM while clearing snapshot after the backup and replication process, the server stops responding and takes full IOPS. The server will comeback after couple of minutes.

Regards,
Hakim. B

[veremin]

Hi, Hakim.

Typically, the problem of frozen VM during snapshot commit process is related to VMware rather than Veeam side. So, as a first step of investigation I would recommend reproducing this situation in order to understand by whom this issue should be addressed (VMware, Veeam).

You can take the snapshot of your production VM manually, keep the snapshot open for long enough time before deleting it, similar to time it takes to backup the VM, and see if you experience the aforesaid issue or not.

Thanks.

[hakimbhungra]

Dear All,

Below the vmware version in our environment.

Host: VMware 5.0
Veeam Backup & Replication : 7.0

Storage: SAN FC connected
Virtual Machine OS: Red Hat 6
Size of VM: 600 GB

I have tried on other windows machine but this problem didn't occur, we have already disabled the vmware quiesing option.

Regards,
Hakim. B

[hakimbhungra]

I faced the same issue today as the veeam job ran again, snapshot removal come to 50 percent and freezes and stops network. Veeam backup logs show snapshot removal took 2:30 minutes.

10/2/2013 1:00:16 AM :: Job started at 10/2/2013 1:00:15 AM
10/2/2013 1:00:16 AM :: Building VM list
10/2/2013 1:01:01 AM :: VM size: 580.0 GB
10/2/2013 1:01:01 AM :: Changed block tracking is enabled
10/2/2013 1:01:03 AM :: Preparing next VM for processing
10/2/2013 1:01:03 AM :: Processing 'moodleapp'
10/2/2013 1:01:05 AM :: All VMs have been queued for processing
10/2/2013 1:57:19 PM :: Load: Source 10% > Proxy 8% > Network 50% > Target 93%
10/2/2013 1:57:19 PM :: Primary bottleneck: Target
10/2/2013 1:57:19 PM :: Job finished at 10/2/2013 1:57:19 PM


Regards,
Hakim. B

[Vitaliy S.]

Hakim,

Veeam B&R doesn't commit VM snapshot, it just sends the API call to vCenter Server to run this procedure. Please review this thread for possible tweaks that can help you to workaround this VMware snapshot behavior.

Thanks!

[bennie1988]

Best Veeam support,

Firts i have read VMware : [FAQ] Frequently Asked Questions v6
But my following qestion is not in that Faq:

The specs what iam running:

2x Vmware vsphere 5.0

1st Host:

1x Windows 2003 file server
1x windows 2003 Mail server (Mdaemon)= Standby (replication)
1x Windows 7 32bit with veeam 6.5
Veeam jobs:
1 Replication Mailserver from host 2 to host 1 (15min)
1 Backup job Mailserver from host 2 to NAS (1.5 hour)
1 Backup job Mailserver from host 2 to NAS (24 hour)


Second Host:
1x Windows 2003 fileserver=Standby (replication)
1x windows 2003 Mail server (Mdaemon)
1x Windows 7 32bit with veeam 6.5
Veeam jobs:
1 Replication Fileserver from host 1 to host 2 (15min)
1 Backup job Fileserver from host 1 to NAS (1 hour)
1 Backup job FIleserver from host 1 to NAS (24hour)

The issue:

First of all i want to say that Veeam is a verry good backup solution for VMware!

Everytime that Veeam delete his Temp snapshot on Fileserver and Mailserver (in Replication and the backup jobs).
The server,s hangs from 80% til 100% of removing snapshot. So that means that there a 30 people waiting till the remove action is done. Because there Network share can not be acces. And the mail wil stop working.
Also they are working with a planning program that is running on the fileserver. Also that wil not work.
So the servers Hangs.

Is this a known issue?
Is this normal?
Can you guys help me to figure this out and resolve the problem?

Kind regards,
Ben de korver (Netherlands)

[veremin]

Hi, Ben. Most likely, this issue can be even reproduced without Veeam presence. As to potential workarounds on how this snapshot issue can be bypassed, kindly, take a look at the answers provided above; might be helpful.

Thanks.

[bennie1988]

Hey Guys,

I have done the following thing:

- Disable VMware Tools Sync driver on the VM: http://kb.vmware.com/selfservice/micros ... Id=1009886

Its in test fase now! ( I hope it wil work)

More tips here:
http://www.veeam.com/kb1681

Edit 4 okt 2013 at 17:05

After a call with the customer it seems that the problem has been resolved.
Next week i wil be back to update this post

Kind Regards,
Ben de Korver (Netherlands)

[maverick964_uk]

I have Veeam 6.5 update3 installed.

Maybe 2-3 times a week we get a backup that hangs due to being stuck on "removing snapshot..."
I would have thought Veeam would be more intelligent to report an error then continue to the next VM.

How can we get round this as I'm sure its not all Veeam's fault. Vcenter too...

thanks

[foggy]

Mike, please review this topic for some hints. Also, parallel processing, introduced in Veeam B&R v7, allows to significantly decrease the time VM performs on snapshot, which results in faster snapshot commit.

[michaelwright1971]

Hya,

I am using Veeam 7 on Vsphere 5.1 and having trouble backing up SQL server also!
The backup seems to go well as for data being backed up. But the transactions on the database seem to suffer. and ss other guys have stated the SQL server looses connectivity(the backup failed due to lack of space for the snapshot of drives also)
config as follows:
SQL server only vm on ESXI host connected to Iscsi storage. Veeam Backup installed on other ESXI host 2008 R2. and using bare metal Linux box with IScsi storage for NFS repository. all on same LAN

For the backup job I have selected all defaults except enabled: Application aware image processing & Guest file system indexing.

From what Ive read, is the following the best approach.

1.Snapshots different location.
2. make sure no other snapshots on vm.
3. disable CBT.
4. set snapshot for safe removal
5.Set backup type direct SAN access.
6. Backup SQL server during quiet time etc

cheers Mike


The backups have failed also due to

[Vitaliy S.]

Hi Mike,

This is not required:

3. disable CBT.
4. set snapshot for safe removal
5.Set backup type direct SAN access.

Check out this post for a short summary that might help.

Thank you!

[veremin]

3. disable CBT

You’d better not disable CBT, as in this case the VB&R will read the whole VM image in order to find changed blocks. As a result, the time of backup operation increases, as well as, the size the snapshot that needs to be committed.

Thanks.

[superdekster]

In my case snapshot removal of SQL Database Server took about 1 hour and we lost pings for ~5 min at the end of commit. Then I moved VM to another datastore with more IOPS available, now snapshot removal takes 3 min and no network lose at all.

Hope this helps.

[zak2011]

There seems to be again some strange issues with snapshot removal processes after the Veeam backup job. One of them being an exchange VM.
patch2 has been installed , however after the next consecutive backup job run, the snapshot has not been removed yet.
Are there still problems with snapshot removal after Veeam backs up?

[foggy]

Arun, as you might know, the snapshot commit issues are typically related to VMware rather than Veeam (to confirm that, you can take the snapshot of the VM manually and keep it open long enough before deleting).

Although parallel processing and some other improvements, introduced in Veeam B&R v7, allow to significantly decrease the time VM performs on snapshot (which should result in faster snapshot commit), this problem itself could still persist, depending on the actual environment and particular VM size and activity. A number of potential hints/workarounds on how to eliminate this issue can be found in this topic above.

[zak2011]

Thanks the reply, foggy.
I remmember addressing this issue like many others , when Veeam v6 was released. However after that a patch update was released which was actually fixed the issues for me.
Update from support was there was vcenter disconnection and VeeamTracker snapshot had some old entries of left over snapshots and due to that this happened again. ( I hav not closed the case yet).
I understand that snapshot commit issues are typically related to VMware, but what I was also told that Veeam would tell vcenter during next job run to remove the left over snapshot... unfortunatly that did not happen in my case.
Would really like to know why.
Snapshot was removed manually, which made the Exchange VM unresponsive and not pingable for 15-20 min ...this may have happened also due to lack of sufficient resources on the host.

[foggy]

So are you talking about snapshots left over after the backup process completes and not long VM snapshot commit issue discussed in this thread? Leftover snapshots should be automatically deleted during the next job run, so, please continue working with our support team on this.

[zak2011]

Thats right! Thanks.

[KiwiJJ]

Hi,
Has anyone managed to get backup\replication of a fairly busy SQL server working with changed block tracking turned on / off ?

If I have changed block tracking turned on then the SQL server freezes during commit and the application servers loose connection. So I have to leave it off. This also applies to my backups.

What I am trying to achieve is to replicate the SQL server over the internet to the DR site without having to send huge amounts of data over the internet. If I have changed block tracking off and replicate using incremental backups does Veeam still just send changed blocks using it's own processing even though this will take longer ?

thanks,

John

[foggy]

Yes, only changes will be copied over to the target site. However, it is not recommended to disable CBT, as apart from the longer backup/replication, it will result in even larger snapshot file to commit (VM will run on a snapshot much longer due to more time required to read the entire VM disks). Please review this thread for some other hints on how to optimize backups of a busy server.

[arnen2]

Hi !
I have one VM with 3.8 TB disks. I do an own backupjob on that VM. It takes less than 6 hours. The snapshot removal takes 24 minutes, when the server is not possible to reach. When the backup starts it is not any stop in the connection with the server.
Best Wishes
Arne Nilsson

[dellock6]

Hi Arne,
sounds like the usual commit problems in vSphere, what happens if you run a manual snapshot in vCenter, keep it open for 6 hours and then you delete it? My thought is you are going to see the same connection problem as you described, since it's not a Veeam problem.
to better understand what's happening, see this VMware article:
http://kb.vmware.com/selfservice/micros ... Id=1002836

Luca.

[Matts N]

Hello!
As a side note for those using VNX storage systems and are using VAAI, you could experience somthing like this too, consolidation (and other VAAI operations) take a very long time to finish. This is due to a "feature" in the firmware, an updated version that solves this was released December 13, IIRC. Not sure about the firmware levels, but a customer with a VNX5500 was helped by this.

// Matts

[cby]

Ah, the ole snapshot merge and stun nugget! We have hundreds of VMs backed up but only one complains and drops the network connection even if the apparent break in comms is a few seconds. It only occurs where there is a connection to another server, the one that gets stunned at snapshot merge time. In our case an ODBC connection via SQL server to an Oracle DB server.

The workaround is to avoid NFS mounts to the stunned server, or at least use NFS judiciously. Better still, create a keepalive (a la standard TCP or SSH configs) on the client side that connects to the stunned server so that it can tolerate network drops for a given period or number of attempts.

[Shaft]

I have just installed Veeam 7.0 on a vSphere 5.1 environment. I must say Veeam is working great for 95% of my VMs. However the other 10% are having trouble with creating and removing snapshots. This is not an issue with Veeam but a limitation on the use of snapshots where heavily used VMs may experience an extended stun due to snapshots.
Is there a way Veeam can backup without using snapshots? Or any other options?

Thanks for the help in advance.

I did search the fourms and found only one topic about it and says snapshots are required. so I guess my question is what are my options.
http://forums.veeam.com/veeam-backup-re ... t8675.html


Thanks again

[foggy]

Vince, you're correct in assumption that there's no way to avoid snapshots completely. However some hints that allow to avoid (or at least eliminate the effect of) snapshot removal issues can be found in this huge topic above. Thanks.

[myrdin]

Hi,

scenario:

V7 Patch 3 backing up a machine. As soon as Veeam sends command to remove the snapshot to Vcenter at the end of the backup run, the vm hangs all the way until the remove snapshot process is finished (about 10-20 seconds).

Facts:

- i had this problem in vmware and disabling CBT in VMWARE, fixed it. Removing snapshot IN VMWARE takes less than a second. Removing snapshot in VEEAM, 20 seconds delay while the VM is unresponsive.
- machine is virtually 0 MB change rate (this is for testing). So no IOPS etc problems on big drives
- not a cpu issue, already checked

Tried:

- forcing ndp instead of hotadd, no luck
- disable application aware option, no luck
- disable CBT in both Vmware and Veeam job, no luck,


any thoughts?

thanks

[myrdin]

Actually it seems to work, Vmware reenabled CBT on the VM (??) even if it was disabled before