Snapshot removal issues of a large VM

[rich980]

Hi,

We use veeam 6.1 for our daily backups at the moment but want to start using replication as well.

We currently have 4 esxi hosts running vsphere 4.1 split into two clusters.

I have setup a replication job for one vm machine to replicate from one cluster to the other.

It works fine but I find the vm machine is briefly losing network connection during replication.

Is there anything I can do to elimate this, would adding backup proxies between the clusters help?

Thanks

[JWester]

How many seconds is "briefly"? Does the network outage happen exactly during snapshot removal?

[BlueMan]

If we were running 2 virtual machines, 1 File and Print, and 1 Digital Dictation VM, on the following server hardware:

1x ML350p Gen8, Dual Processor Capable, 1 x Intel E5-2620 (2.0Ghz) Six-Core, 2x4GB Rdimms, 4x300GB 15K SFF SAS, P420i/512 FBWC.
1 x HP NC365T - Network adapter 4 Port

And we setup Veeam to take snapshots of the server in office hours. Would users notice any degradation of server/application performance?

We would be backing up to the following NAS, most likely via iSCSI.

1 x QNAP TS-659 Pro+ Turbo NAS
4 x Hitachi Ultra 3TB A7K3000 Hard Drive

Is iSCSI the best way to back up to the above NAS box?

How often can you take snapshots with Veeam?

If I took a snapshot of a server with 600GB data on it, roughly how long would it take to create the snapshot on the NAS box?

Thanks,

[JWester]

If we were running 2 virtual machines, 1 File and Print, and 1 Digital Dictation VM, on the following server hardware:

1x ML350p Gen8, Dual Processor Capable, 1 x Intel E5-2620 (2.0Ghz) Six-Core, 2x4GB Rdimms, 4x300GB 15K SFF SAS, P420i/512 FBWC.
1 x HP NC365T - Network adapter 4 Port

And we setup Veeam to take snapshots of the server in office hours. Would users notice any degradation of server/application performance?

That depends.
If you don't have a reasonable IO load on your internal disks: Probably not.
Taking a Vmware snapshot doubles the number of accesses on the VM's disk blocks. That could slow down this specific VM and, due to the added stress on the data store, all other VMs.

We would be backing up to the following NAS, most likely via iSCSI.

1 x QNAP TS-659 Pro+ Turbo NAS
4 x Hitachi Ultra 3TB A7K3000 Hard Drive

Is iSCSI the best way to back up to the above NAS box?

My opinion, not Veeam specific:
If you have a dedicated storage LAN to the QNAP: Yes.
If you share the LAN than I would prefer a NAS protocol like CIFS.

How often can you take snapshots with Veeam?

You mean, how often you could take backups? As often as you need. But it stresses the disks.

If I took a snapshot of a server with 600GB data on it, roughly how long would it take to create the snapshot on the NAS box?

There's no snapshot on the NAS box as it's only for backup.

[tsightler]

If you have NFS storage and you utilize hotadd as backup mode, your VMs might be 'stunned' longer than usual (lose network connectivity)

Recently we had case opened with VMware SDK team and they will be releasing official document shortly. As of now they have explained it as an 'issue' with NFS locking 'mechanism'. Issue can be easily reproduced using vSphere client (snapshot VM, mount disks to backup proxy as independent non-persistent, dismount disks -> remove VM snapshot) on ESXi4/ESXi5

Workarounds:
1. Switch to network mode
2. Migrate backup proxy to the same ESXi host where you have your VM.

Does anyone have a pointer to KB article which documents this issue or not if a fix has ever been released?

[rkovhaev]

VMware released KB
http://kb.vmware.com/selfservice/micros ... Id=2033540
I don't think there is a fix...

[tsightler]

This KB article is also referenced from the following KB:

http://kb.vmware.com/selfservice/micros ... Id=2010953

The second KB offers a solution for VMs hosted on NFS datastores freezing when snapshots are removed. I don't know if this would really resolve the issue but it might be worth a try since the "fix" references the other issue as well.

[Wlacroix]

So we have been veeam users for a while, in a very simple way.

Backup job at night and be happy. We had an SQL drive fail and did some very fan-dangled stuff to get it back but lost 2 hours of data. The powers that be say this is unacceptable so.....

We used to run the FULL on the SQL server, but stuff would crash bad.....No stopping of services etc. This SQL holds about 700g of databases, some very busy, some not.

New SQL jobs as of October1 2012:

Full at night midnight every day of week,
Differential 8:30, 13:30, 18:30
Transaction backups at 4:30 - 6:30 at night every 15 minutes, excluding the differential backup times.
This backup runs to a drive on the SQL server.

New veem backup to grab F drive and push it to another location using veeam.

We want to run this job as often as possible, were moving anywhere from 1mb to 30gb of data. This is not an issue for us, we have fiber links for the core. They never peak out more than 550mbit. (2gig links)
We turned off compression to the lowest level to increase speed of moving data, about 1.2gb minute. Give or take. (we think the bottle neck is the storage system at the backup location)

The SQL server does live transactions on several applications, aprox 85 users 16 hours a day for 3-8 applications, across I think 4-5 different databases about 25 gigs a pop.

When we run this job during the day, our SQL backup jobs get all kinds of weird errors, users start calling the IT help line like mad.
The DB guys get errors just shortly after my snapshot removal process happens.

I need a little insight into what exactly this snapshot tool is doing. It does not make any sense to me why it would be killing SQL connections.
Were NOT backing up any live data at all, were backing up the backup drive. HOWEVER......

SQL might be in the middle of a job when my job happens, this is almost unavoidable.
How are others backing up major SQL servers?

Please help!
Regards,
Wally

6 hosts
IBM DS3500 ISCSI 48 disks split into raid 10 1 hot swap per cabinet
3 switches, 2 in the server room, ISCSI separation for redundancy, third ISCSI switch is in the backup room on the other side of building. 2gb fiber links from ISCSI-1 and 2 to ISCSI-3 switch.
IBM DS3300 12TB running Raid 5, 1 hot swap.

[Vitaliy S.]

Were NOT backing up any live data at all, were backing up the backup drive. HOWEVER......

Even if you do not backup live data you still snapshot the SQL Server VM which holds the backups, right? When you commit the snapshot some connectivity issues might be observed (VMware limitation), please look through this topic for more details and recommendation on how to proceed in this case.

[Wlacroix]

Even if you do not backup live data you still snapshot the SQL Server VM which holds the backups, right? When you commit the snapshot some connectivity issues might be observed (VMware limitation), please look through this topic for more details and recommendation on how to proceed in this case.

I have read the topic 3 or 4 times now and were looking at making a few changes. Its absolutely the VM side.

We have 2 jobs, one is a full that we run periodically to recover the "FULL" server. Every month ish. Anyone know how checking the "independent" check boxes will affect this full backup?

We're wondering if its better to do a file copy instead of a veeam backup of this server.
There are aprox 300 users hitting it in various SQL applications and its a tough one to backup.

[Vitaliy S.]

If you use independent disks for the system drives then these disks will not be snapshotted and these disks will not be affected by the snapshot commit operation. On top of that, try the tweaks mentioned on the 5th page of this topic, might help.

[zak2011]

Recently faced an issue with Sure backups for exchange. One of the VMs kept failing during the Sure backup job the error stated cannot open the vmfs volume. After further investigation with technical support, found out that the snapshot created by a previous Veeam backup job for exchange was not removed.Tracing back to a previous job history stated a snapshot removal warning for the VM and a retry was successful. All backups jobs after that time stated the backup job was successful and removal of snaphot succeeded.
I had no clue the exchange VM was running on a shaphot for 15 days. Perhaps one of the reasons this happened was there was insufficient space on the datastore. But if this is the case..why would Veeam continue to backup the VM and state that the removal of snapshot succeded. Shouldnt the backup fail if Veeam isnt able to remove the previous temporary snapshot?

[Vitaliy S.]

I had no clue the exchange VM was running on a shaphot for 15 days. Perhaps one of the reasons this happened was there was insufficient space on the datastore.

To be notified about these issues, please take a look at Veeam ONE which has predefined alarms for snapshots (snapshot size, snapshot age, number of snapshots and VM consolidation needed). These alarms can also be used in the free edition of Veeam ONE.

But if this is the case..why would Veeam continue to backup the VM and state that the removal of snapshot succeded.

Veeam B&R doesn't remove the snapshot, it just sends an API call to VMware asking to remove the snapshot, most likely the vCenter Server replied back that the snapshot was removed.

[zak2011]

Thanks for the recommendation.
Seems that connection to the VC has been lost right before job was going to send remove snapshot command.
However if a previous temp snaphot exists, isn't there a way that Veeam can send an API call to VMware to remove the existing temp Veeam snaphot before the next backup begins?

[Vitaliy S.]

If snapshot was not removed for some reason, the subsequent job will commit this snapshot before creating a new one. Need to take a look at the log files to see what happened in your case.

[zak2011]

Noticed that again the vcenter does not remove the veeam temp snapshot for one particular VM on the second backup job run.
The VM is 20GB in size on a datastore with 340GB free space.
According to the backup logs the vcenter lost connection intermittently during the backup run and immediately Veeam has done a retry of the backup job and it says the Removal of snapshot succeeded.
However the snapshot has not been removed. Should i raise this up with technical support?

[Gostev]

Sure.

[zak2011]

Technical support said that ,unfortunately retry didn't check any previously created snapshots and that they would confirm that behavior with R&D to understand if it's possible to implement additional snapshot pre-checks.

[badisoft]

Hello,

some days ago we have pathed one of our ESXi from Build 421000 to Build 821926. From this Time on we hav a significant problem:

-Every Backup an Replicationjob works fine
-At the end of the repl-Job the replicated Exchangeserver stops working for some minutes (while deleting Snapshots).
-We do not have this behavior bevor pathing vmware.

Any suggestions?

Thanks

Manfred

[skeeziix]

This is obviously a known issue there are multiple references to this issue, is this "bug" going to be addressed or is there any updates for this?

http://forums.veeam.com/viewtopic.php?f ... &start=135
http://forums.veeam.com/viewtopic.php?f ... 838#p69838
http://community.spiceworks.com/topic/2 ... kup?page=2

[Vitaliy S.]

Hi Erica, there are multiple different issues referenced in the links above. Snapshot consolidation process is solely controlled by VMware, so it's better to involve their technical team while troubleshooting the original issue of this thread.

[ZacTech]

Any news on this issue? I am being forced to look at different backup solutions because of this, at least for exchange and SQL servers. I understand its not Veeams problem, but Veeam based their entire software off of it so they should make it their problem . I run Vsphere 5.0 bstandard with 2x equallogic 6100s. One SAS and one SATA. I have forced exchange to the SAS and no luck, still lost pings and outlook issues. We're talking 1-3 pings, nothing major but enough to rile the users. I am wondering if using dells MPIO (psp MEM vs round robin) would help but I only have the standard version of vSphere so I can't. I tried changing the workingDir for one server to the mirrored 15k drives powering the host but had the same issue. Issue persists on latest eql firmware 6.0.2 and my next step is to try vSphere 5.1. I LOVE Veeam but this is making me dig up my virtually torched BackupExec licenses. Then I will have to do a full Veeam back to get an image followed by a full BackupExec so I can do transaction backups every 30 mins. Please advise with a magical solution that has eluded us.

[ZacTech]

Spoke with my dell rep about this and he suggested that delayed ACK and LRO be disabled:

http://en.community.dell.com/support-fo ... x#20008239

I will be doing this tonight and will post the results.

[ZacTech]

Things appear to be better, I still lose a ping, but only one. In addition to the ACK and LRO they recommend round robin iops be taken from 1000 to 3. The true test will be tomorrow in the day with a full load.

They key piece was from this thread:

http://communities.vmware.com/message/2078926

One thing about Delayed ACK, is that you have to verify that the change took place. If you just change it, it appears that only new LUNs will inherit the value. (disable) I find that, while in maint mode, removing the discovery address and any discovered targets (in static discovery tab), then disabling Delayed ACK and re-add in Discovery Address and rescan resolves this.

At the ESX console run: #vmkiscsid --dump-db | grep Delayed All the entries should end with ='0' for disabled.
Common causes of performance issues that generate that alert are:

1.) Delayed ACK is enabled.
2.) Large Recieve Offload (LRO) is enabled
3.) MPIO pathing is set to FIXED
4.) MPIO is set to VMware Round Robin but the IOs per path is left at default of 1000. Should be 3.
5.) VMs with more than one VMDK (or RDM) are sharing one Virtual SCSI adapter. Each VM can have up to four Virtual SCSI adapters.
6.) iSCSI switch not configured correctly or not designed for iSCSI SAN use.

Update: simulated a load by having veeam backups going on some of the hosts and SAN, and the issue of timeouts during the creation and release of the snapshot persists. I did noticed during a exstop then hitting n, that during the snaps my %DRPRX for the virtual nic of the VM skyrockets, both with e1000 and vmxnet3.

[mark.casemore]

I have a Veeam job to backup an SQL server, the past two days the backup has been succesful but the job gets stuck on removing the snapshot whereby it takes at least 12 hours to remove the snapshot and severely hampers the performance of the server i.e. users complaining that the server is slow. The server is quite large in size, about 700GB in total. When configuring a job for sql is there anything specific I should be doing ?

I cancelled todays job but in the virtual infrastructure client it still says it's removing the snapshot and has been like that for 6 hours and is only 48% completed. I have Veeam installed on a virtual server in the same cluster as the SQL box to keep everything together, I have also raised a call with Veeam so they can look at this but was wondering if anyone else had issues like this. I am running Veeam 6.5 patch 1 since this morning but haven't tried any backups yet to test.

[veremin]

Hi, Mark.

When configuring a job for sql is there anything specific I should be doing ?

Just enable Application-Aware Image Processing along with log truncation option.

As to the snapshot removal issue, please be aware that it’s VMware which is completely responsible for the process of creating and removing snapshot, meanwhile, Veeam is just playing the “requestor” role.

Thus, it stands to reason to reproduce this situation and see whether these issues should be addressed by Veeam or by VMware. So, you can take the snapshot of your production VMs manually, keep the snapshot open for long enough time before deleting it, similar to time it takes to backup the VM. And then trigger the snapshot commit operation to check if you experience the similar behaviour or not.

Hope this helps.
Thanks.

[mediomen27]

Hi,
How big should be the vmware virtual disk to have a snapshot without problems ?
I have vSphere with vmfs 5.54 and veeam 6.1.0.181
My question come from a previous experience with the same version of veeam and vmware 3.5 with vmfs 3 (I dont know the block size) and the following vm configuration:
- 2 virtual disks of about 1,46 TB
- 1 virtual disk of about 1,50 TB
- 1 virtual disk of about 600 GB
- 1 virtual disk of about 650 GB

at the end of snapshot I have got a freeze of 20 minutes.
Thus I ask what should be the maximum size of the disks to have not any freeze ?
thanks very much

[kjc3303]

All depends on Storage and it's ability to commit the snapshot as well as things such as amount of changes made since snap taken. I don't think you will find an exact answer(I know because I have been looking into it myself).
You may get more joy on a vmware forum as this underlying issue is a vmware problem, that said there's a lot of knowledgeable guys on here

I will keep an eye on this thread as it's something that effects me albeit on a smaller scale( a minute or so)! but when you want to replicate frequently and users lose network drives it can be a problem.

Kev

[Vitaliy S.]

Yes, Kevin is correct, it doesn't matter what block size you have on the VMFS, it's more about storage IO performance and the load you have on the VM.

[tsands]

ZacTech,

Did your final change resolve the issue?