Host-based backup of VMware vSphere VMs.
Post Reply
DavidReimers
Enthusiast
Posts: 52
Liked: 2 times
Joined: Sep 20, 2010 4:39 am
Full Name: David Reimers
Contact:

Re: Snapshot removal issues of a large VM

Post by DavidReimers »

Nope, I did know Vitality meant VMware Support - typo on my part. I'll log a call with VMware.
hakimbhungra
Influencer
Posts: 18
Liked: never
Joined: Oct 01, 2013 11:46 am
Full Name: hakimbhungra
Contact:

[MERGED] RedHat VM freeze while clearing Snapshot

Post by hakimbhungra »

Hi,

We are facing issue on Red hat VM while clearing snapshot after the backup and replication process, the server stops responding and takes full IOPS. The server will comeback after couple of minutes.

Regards,
Hakim. B
veremin
Product Manager
Posts: 20389
Liked: 2297 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: RedHat VM freeze while clearing Snapshot

Post by veremin »

Hi, Hakim.

Typically, the problem of frozen VM during snapshot commit process is related to VMware rather than Veeam side. So, as a first step of investigation I would recommend reproducing this situation in order to understand by whom this issue should be addressed (VMware, Veeam).

You can take the snapshot of your production VM manually, keep the snapshot open for long enough time before deleting it, similar to time it takes to backup the VM, and see if you experience the aforesaid issue or not.

Thanks.
hakimbhungra
Influencer
Posts: 18
Liked: never
Joined: Oct 01, 2013 11:46 am
Full Name: hakimbhungra
Contact:

Re: Snapshot removal issues of a large VM

Post by hakimbhungra »

Dear All,

Below the vmware version in our environment.

Host: VMware 5.0
Veeam Backup & Replication : 7.0

Storage: SAN FC connected
Virtual Machine OS: Red Hat 6
Size of VM: 600 GB

I have tried on other windows machine but this problem didn't occur, we have already disabled the vmware quiesing option.

Regards,
Hakim. B
hakimbhungra
Influencer
Posts: 18
Liked: never
Joined: Oct 01, 2013 11:46 am
Full Name: hakimbhungra
Contact:

Re: Snapshot removal issues of a large VM

Post by hakimbhungra »

I faced the same issue today as the veeam job ran again, snapshot removal come to 50 percent and freezes and stops network. Veeam backup logs show snapshot removal took 2:30 minutes.

10/2/2013 1:00:16 AM :: Job started at 10/2/2013 1:00:15 AM
10/2/2013 1:00:16 AM :: Building VM list
10/2/2013 1:01:01 AM :: VM size: 580.0 GB
10/2/2013 1:01:01 AM :: Changed block tracking is enabled
10/2/2013 1:01:03 AM :: Preparing next VM for processing
10/2/2013 1:01:03 AM :: Processing 'moodleapp'
10/2/2013 1:01:05 AM :: All VMs have been queued for processing
10/2/2013 1:57:19 PM :: Load: Source 10% > Proxy 8% > Network 50% > Target 93%
10/2/2013 1:57:19 PM :: Primary bottleneck: Target
10/2/2013 1:57:19 PM :: Job finished at 10/2/2013 1:57:19 PM


Regards,
Hakim. B
Vitaliy S.
VP, Product Management
Posts: 27368
Liked: 2798 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. »

Hakim,

Veeam B&R doesn't commit VM snapshot, it just sends the API call to vCenter Server to run this procedure. Please review this thread for possible tweaks that can help you to workaround this VMware snapshot behavior.

Thanks!
bennie1988
Novice
Posts: 8
Liked: 2 times
Joined: Jul 02, 2013 11:28 am
Full Name: NL
Contact:

[MERGED] : Veeam 6.5 patch 3 Remove temp snapshot issue

Post by bennie1988 »

Best Veeam support,

Firts i have read VMware : [FAQ] Frequently Asked Questions v6
But my following qestion is not in that Faq:

The specs what iam running:

2x Vmware vsphere 5.0

1st Host:

1x Windows 2003 file server
1x windows 2003 Mail server (Mdaemon)= Standby (replication)
1x Windows 7 32bit with veeam 6.5
Veeam jobs:
1 Replication Mailserver from host 2 to host 1 (15min)
1 Backup job Mailserver from host 2 to NAS (1.5 hour)
1 Backup job Mailserver from host 2 to NAS (24 hour)


Second Host:
1x Windows 2003 fileserver=Standby (replication)
1x windows 2003 Mail server (Mdaemon)
1x Windows 7 32bit with veeam 6.5
Veeam jobs:
1 Replication Fileserver from host 1 to host 2 (15min)
1 Backup job Fileserver from host 1 to NAS (1 hour)
1 Backup job FIleserver from host 1 to NAS (24hour)

The issue:

First of all i want to say that Veeam is a verry good backup solution for VMware!

Everytime that Veeam delete his Temp snapshot on Fileserver and Mailserver (in Replication and the backup jobs).
The server,s hangs from 80% til 100% of removing snapshot. So that means that there a 30 people waiting till the remove action is done. Because there Network share can not be acces. And the mail wil stop working.
Also they are working with a planning program that is running on the fileserver. Also that wil not work.
So the servers Hangs.

Is this a known issue?
Is this normal?
Can you guys help me to figure this out and resolve the problem?

Kind regards,
Ben de korver (Netherlands)
veremin
Product Manager
Posts: 20389
Liked: 2297 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin » 1 person likes this post

Hi, Ben. Most likely, this issue can be even reproduced without Veeam presence. As to potential workarounds on how this snapshot issue can be bypassed, kindly, take a look at the answers provided above; might be helpful.

Thanks.
bennie1988
Novice
Posts: 8
Liked: 2 times
Joined: Jul 02, 2013 11:28 am
Full Name: NL
Contact:

Re: Snapshot removal issues of a large VM

Post by bennie1988 » 1 person likes this post

Hey Guys,

I have done the following thing:

- Disable VMware Tools Sync driver on the VM: http://kb.vmware.com/selfservice/micros ... Id=1009886

Its in test fase now! ( I hope it wil work)

More tips here:
http://www.veeam.com/kb1681

Edit 4 okt 2013 at 17:05

After a call with the customer it seems that the problem has been resolved.
Next week i wil be back to update this post

Kind Regards,
Ben de Korver (Netherlands)
maverick964_uk
Expert
Posts: 102
Liked: 3 times
Joined: May 09, 2013 8:57 am
Full Name: Mike Lavery
Contact:

[MERGED] Snapshot removal hang

Post by maverick964_uk »

I have Veeam 6.5 update3 installed.

Maybe 2-3 times a week we get a backup that hangs due to being stuck on "removing snapshot..."
I would have thought Veeam would be more intelligent to report an error then continue to the next VM.

How can we get round this as I'm sure its not all Veeam's fault. Vcenter too...

thanks
foggy
Veeam Software
Posts: 21138
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy »

Mike, please review this topic for some hints. Also, parallel processing, introduced in Veeam B&R v7, allows to significantly decrease the time VM performs on snapshot, which results in faster snapshot commit.
michaelwright1971
Novice
Posts: 6
Liked: never
Joined: Nov 03, 2013 6:17 pm
Full Name: Michael Wright
Contact:

Re: Snapshot removal issues of a large VM

Post by michaelwright1971 »

Hya,

I am using Veeam 7 on Vsphere 5.1 and having trouble backing up SQL server also!
The backup seems to go well as for data being backed up. But the transactions on the database seem to suffer. and ss other guys have stated the SQL server looses connectivity(the backup failed due to lack of space for the snapshot of drives also)
config as follows:
SQL server only vm on ESXI host connected to Iscsi storage. Veeam Backup installed on other ESXI host 2008 R2. and using bare metal Linux box with IScsi storage for NFS repository. all on same LAN

For the backup job I have selected all defaults except enabled: Application aware image processing & Guest file system indexing.

From what Ive read, is the following the best approach.

1.Snapshots different location.
2. make sure no other snapshots on vm.
3. disable CBT.
4. set snapshot for safe removal
5.Set backup type direct SAN access.
6. Backup SQL server during quiet time etc

cheers Mike


The backups have failed also due to
Vitaliy S.
VP, Product Management
Posts: 27368
Liked: 2798 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Snapshot removal issues of a large VM

Post by Vitaliy S. »

Hi Mike,

This is not required:
3. disable CBT.
4. set snapshot for safe removal
5.Set backup type direct SAN access.
Check out this post for a short summary that might help.

Thank you!
veremin
Product Manager
Posts: 20389
Liked: 2297 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Snapshot removal issues of a large VM

Post by veremin »

michaelwright1971 wrote:3. disable CBT
You’d better not disable CBT, as in this case the VB&R will read the whole VM image in order to find changed blocks. As a result, the time of backup operation increases, as well as, the size the snapshot that needs to be committed.

Thanks.
superdekster
Influencer
Posts: 12
Liked: 6 times
Joined: Oct 23, 2013 9:32 am
Full Name: Aleksandr Alpatov
Contact:

Re: Snapshot removal issues of a large VM

Post by superdekster » 2 people like this post

In my case snapshot removal of SQL Database Server took about 1 hour and we lost pings for ~5 min at the end of commit. Then I moved VM to another datastore with more IOPS available, now snapshot removal takes 3 min and no network lose at all.

Hope this helps.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Case 00441952

Post by zak2011 »

There seems to be again some strange issues with snapshot removal processes after the Veeam backup job. One of them being an exchange VM.
patch2 has been installed , however after the next consecutive backup job run, the snapshot has not been removed yet.
Are there still problems with snapshot removal after Veeam backs up?
foggy
Veeam Software
Posts: 21138
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy »

Arun, as you might know, the snapshot commit issues are typically related to VMware rather than Veeam (to confirm that, you can take the snapshot of the VM manually and keep it open long enough before deleting).

Although parallel processing and some other improvements, introduced in Veeam B&R v7, allow to significantly decrease the time VM performs on snapshot (which should result in faster snapshot commit), this problem itself could still persist, depending on the actual environment and particular VM size and activity. A number of potential hints/workarounds on how to eliminate this issue can be found in this topic above.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Snapshot removal issues of a large VM

Post by zak2011 »

Thanks the reply, foggy.
I remmember addressing this issue like many others , when Veeam v6 was released. However after that a patch update was released which was actually fixed the issues for me.
Update from support was there was vcenter disconnection and VeeamTracker snapshot had some old entries of left over snapshots and due to that this happened again. ( I hav not closed the case yet).
I understand that snapshot commit issues are typically related to VMware, but what I was also told that Veeam would tell vcenter during next job run to remove the left over snapshot... unfortunatly that did not happen in my case.
Would really like to know why.
Snapshot was removed manually, which made the Exchange VM unresponsive and not pingable for 15-20 min :(...this may have happened also due to lack of sufficient resources on the host.
foggy
Veeam Software
Posts: 21138
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy »

So are you talking about snapshots left over after the backup process completes and not long VM snapshot commit issue discussed in this thread? Leftover snapshots should be automatically deleted during the next job run, so, please continue working with our support team on this.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Snapshot removal issues of a large VM

Post by zak2011 »

Thats right! Thanks.
KiwiJJ
Expert
Posts: 105
Liked: 2 times
Joined: Feb 16, 2010 8:05 pm
Full Name: John Jones
Location: New Zealand

[MERGED] V7 and SQL Server Replication

Post by KiwiJJ »

Hi,
Has anyone managed to get backup\replication of a fairly busy SQL server working with changed block tracking turned on / off ?

If I have changed block tracking turned on then the SQL server freezes during commit and the application servers loose connection. So I have to leave it off. This also applies to my backups.

What I am trying to achieve is to replicate the SQL server over the internet to the DR site without having to send huge amounts of data over the internet. If I have changed block tracking off and replicate using incremental backups does Veeam still just send changed blocks using it's own processing even though this will take longer ?

thanks,

John
foggy
Veeam Software
Posts: 21138
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy »

Yes, only changes will be copied over to the target site. However, it is not recommended to disable CBT, as apart from the longer backup/replication, it will result in even larger snapshot file to commit (VM will run on a snapshot much longer due to more time required to read the entire VM disks). Please review this thread for some other hints on how to optimize backups of a busy server.
arnen2
Novice
Posts: 3
Liked: never
Joined: Jul 22, 2011 7:07 pm
Full Name: Arne Nilsson
Contact:

Re: Snapshot removal issues of a large VM

Post by arnen2 »

Hi !
I have one VM with 3.8 TB disks. I do an own backupjob on that VM. It takes less than 6 hours. The snapshot removal takes 24 minutes, when the server is not possible to reach. When the backup starts it is not any stop in the connection with the server.
Best Wishes
Arne Nilsson
dellock6
VeeaMVP
Posts: 6165
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Snapshot removal issues of a large VM

Post by dellock6 »

Hi Arne,
sounds like the usual commit problems in vSphere, what happens if you run a manual snapshot in vCenter, keep it open for 6 hours and then you delete it? My thought is you are going to see the same connection problem as you described, since it's not a Veeam problem.
to better understand what's happening, see this VMware article:
http://kb.vmware.com/selfservice/micros ... Id=1002836

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Matts N
Enthusiast
Posts: 69
Liked: 15 times
Joined: Dec 27, 2010 10:41 am
Full Name: Matts Nilsson
Contact:

Re: Snapshot removal issues of a large VM

Post by Matts N » 1 person likes this post

Hello!
As a side note for those using VNX storage systems and are using VAAI, you could experience somthing like this too, consolidation (and other VAAI operations) take a very long time to finish. This is due to a "feature" in the firmware, an updated version that solves this was released December 13, IIRC. Not sure about the firmware levels, but a customer with a VNX5500 was helped by this.

// Matts
cby
Enthusiast
Posts: 97
Liked: 6 times
Joined: Feb 24, 2009 5:02 pm
Contact:

Re: Snapshot removal issues of a large VM

Post by cby » 1 person likes this post

Ah, the ole snapshot merge and stun nugget! We have hundreds of VMs backed up but only one complains and drops the network connection even if the apparent break in comms is a few seconds. It only occurs where there is a connection to another server, the one that gets stunned at snapshot merge time. In our case an ODBC connection via SQL server to an Oracle DB server.

The workaround is to avoid NFS mounts to the stunned server, or at least use NFS judiciously. Better still, create a keepalive (a la standard TCP or SSH configs) on the client side that connects to the stunned server so that it can tolerate network drops for a given period or number of attempts.
Shaft
Influencer
Posts: 17
Liked: never
Joined: Nov 05, 2013 3:06 pm
Full Name: Vince Rucolas
Contact:

[MERGED] Take Backup without Snapshot?

Post by Shaft »

I have just installed Veeam 7.0 on a vSphere 5.1 environment. I must say Veeam is working great for 95% of my VMs. However the other 10% are having trouble with creating and removing snapshots. This is not an issue with Veeam but a limitation on the use of snapshots where heavily used VMs may experience an extended stun due to snapshots.
Is there a way Veeam can backup without using snapshots? Or any other options?

Thanks for the help in advance.

I did search the fourms and found only one topic about it and says snapshots are required. so I guess my question is what are my options.
http://forums.veeam.com/veeam-backup-re ... t8675.html


Thanks again
foggy
Veeam Software
Posts: 21138
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Snapshot removal issues of a large VM

Post by foggy »

Vince, you're correct in assumption that there's no way to avoid snapshots completely. However some hints that allow to avoid (or at least eliminate the effect of) snapshot removal issues can be found in this huge topic above. Thanks.
myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

[MERGED] VM hangs while committing the snapshot

Post by myrdin »

Hi,

scenario:

V7 Patch 3 backing up a machine. As soon as Veeam sends command to remove the snapshot to Vcenter at the end of the backup run, the vm hangs all the way until the remove snapshot process is finished (about 10-20 seconds).

Facts:

- i had this problem in vmware and disabling CBT in VMWARE, fixed it. Removing snapshot IN VMWARE takes less than a second. Removing snapshot in VEEAM, 20 seconds delay while the VM is unresponsive.
- machine is virtually 0 MB change rate (this is for testing). So no IOPS etc problems on big drives
- not a cpu issue, already checked

Tried:

- forcing ndp instead of hotadd, no luck
- disable application aware option, no luck
- disable CBT in both Vmware and Veeam job, no luck,


any thoughts?

thanks
myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

Re: VM hangs while committing the snapshot

Post by myrdin »

Actually it seems to work, Vmware reenabled CBT on the VM (??) even if it was disabled before
Post Reply

Who is online

Users browsing this forum: No registered users and 37 guests