VSS timeout with Exchange 2010

VMware specific discussions

Re: VSS timeout with Exchange 2010

Veeam Logoby mrt » Thu Nov 14, 2013 2:46 pm

Gostev wrote:
Jamie Pert wrote:The trouble is most clients have multiple VMs on one to SAN (which is obviously what's meant to happen), Exchange gets busier as time goes on and on the whole I/O load on the SAN grows and grows

I totally agree. We have also accepted this as the fact. We have built a prototype of a solid workaround for this issue, something that will work even with slowest production storage. But it touches quite a few things around restores, so we cannot release this as a patch... hopefully our next release timelines will give us enough time to include this fix.

I don't suppose the fix you mention here is part of v7 R2?
mrt
Enthusiast
 
Posts: 47
Liked: 2 times
Joined: Thu Feb 10, 2011 7:27 pm

Re: VSS timeout with Exchange 2010

Veeam Logoby soylent » Thu Nov 14, 2013 5:06 pm

mrt wrote:I don't suppose the fix you mention here is part of v7 R2?


There is this bullet point in the release notes:

Added ability for application-aware processing logic to detect passive Microsoft Exchange DAG database present on the VM, and process it accordingly.
soylent
Enthusiast
 
Posts: 48
Liked: 4 times
Joined: Wed Aug 01, 2012 8:33 pm
Full Name: Max Strzelecki

Re: VSS timeout with Exchange 2010

Veeam Logoby Gostev » Thu Nov 14, 2013 8:54 pm

No, that was to address a different issue with item-level recovery. As I've noted, the fix that we have developed for the issues discussed in this thread cannot be shipped as a part of the update due to the scope of changes required. And, this issue has nothing to deal with DAG anyway, and can affect standalone Exchange servers as well.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VSS timeout with Exchange 2010

Veeam Logoby james575 » Tue Nov 26, 2013 10:21 pm

KB1680 says that you have only 20 seconds to snapshot the VM between the start of the freeze and the unfreezing by design. If I do a regular snapshot creation in vCenter of our single Exchange 2010 SP3 server, it takes around 5 minutes, while the KB is suggesting it should take a couple seconds. I am dumbfounded by this--are people here finding snapshots typically take only a few seconds to create??? I have only worked with this one vSphere environment so figured a 5 minute snapshot is normal.
james575
Novice
 
Posts: 8
Liked: never
Joined: Tue Jun 18, 2013 4:07 pm

Re: VSS timeout with Exchange 2010

Veeam Logoby lobo519 » Wed Nov 27, 2013 3:50 pm 1 person likes this post

james575 wrote:KB1680 says that you have only 20 seconds to snapshot the VM between the start of the freeze and the unfreezing by design. If I do a regular snapshot creation in vCenter of our single Exchange 2010 SP3 server, it takes around 5 minutes, while the KB is suggesting it should take a couple seconds. I am dumbfounded by this--are people here finding snapshots typically take only a few seconds to create??? I have only worked with this one vSphere environment so figured a 5 minute snapshot is normal.

Do you have the option "Snapshot virtual machine's memory" selected when you do this manually?
lobo519
Expert
 
Posts: 292
Liked: 34 times
Joined: Wed Sep 29, 2010 3:37 pm

Re: VSS timeout with Exchange 2010

Veeam Logoby james575 » Wed Nov 27, 2013 4:12 pm

lobo519 wrote:Do you have the option "Snapshot virtual machine's memory" selected when you do this manually?


Yes. I have always have selected this for every snapshot I think I have ever made. After reading your question, I tried snapshotting the Exchange server with that option not selected--took two seconds! I always chose the memory option because I thought that would be the safer/better method, but it looks like I need to read up on that some. Thank you!

(As an informational aside for anyone reading this thread, our VSS writer error was fixed by turning on the log truncating for Exchange in the Veeam replication job.)
james575
Novice
 
Posts: 8
Liked: never
Joined: Tue Jun 18, 2013 4:07 pm

Re: VSS timeout with Exchange 2010

Veeam Logoby massimiliano.rizzi » Wed Nov 27, 2013 4:40 pm

Hello experts,

we are facing the exact same problem when processing a Windows SBS 2008 VM using Application-Aware Image Processing:

==================================================
Image
==================================================

Based on my understanding, in our scenario the time-out occurs before VMware even tries to create the snapshot, thus before the 20 seconds timeout kicks in.

Is my understanding correct ?

BTW, I also have opened a Support Case (# 00482742) in order to take a deeper look into this.

Thank you in advance for your support.
massimiliano.rizzi
Service Provider
 
Posts: 93
Liked: 10 times
Joined: Tue Jan 24, 2012 7:56 am
Full Name: Massimiliano Rizzi

Re: VSS timeout with Exchange 2010

Veeam Logoby elliott » Mon Dec 02, 2013 12:20 am 1 person likes this post

Yes, the problem is the VSS snapshot timeout.

To save you some time here, just turn off the VM requiring application aware processing (so the job will complete with a warning and you will have a crash-consistent backup). You can try all the suggestions in the thread, but nothing has worked for our clients. (About 80% of our clients who use SBS and have bought Veeam have this problem.) I've played with v7 trial and it doesn't fix it.

I hope one day Veeam will patch this. Plenty of blame game (It's all Microsoft's fault) but at the end of the day other backup software doesn't have this problem, so Veeam needs to find away to work around the issue.
elliott
Influencer
 
Posts: 11
Liked: 7 times
Joined: Wed Jul 06, 2011 12:43 am

Re: VSS timeout with Exchange 2010

Veeam Logoby Jamie Pert » Tue Dec 03, 2013 11:35 am

annoyingly we had to turn on circular logging on the exchange server and then disable application aware processing and log truncation - far from ideal, however at least we get backups.

In my experience Veeam backing up Exchange 2010 servers is not a great combination. I wish I had more time and more resources to truly get a full understanding as to at what point the 20 second timeout occurs. It's so frustrating to think that a 20 second windows causes all this, a 5 or 10 second increase to the allowance and we could all be happy
@jam1epert on Twitter
Jamie Pert
Enthusiast
 
Posts: 68
Liked: 2 times
Joined: Thu Jun 14, 2012 10:56 am
Location: twitter.com/jam1epert
Full Name: Jamie Pert

Re: VSS timeout with Exchange 2010

Veeam Logoby Gmc85 » Tue Dec 03, 2013 3:58 pm

On our exchange mailbox servers we were encountering the vss_ws_failed_at_freeze error.

Our mailbox databases were sitting on RDM's, we've since changed them over to normal vmware hard disks and now I've not had a failed backup since (Hurrah!).
Gmc85
Novice
 
Posts: 7
Liked: never
Joined: Wed Dec 28, 2011 1:05 pm
Full Name: George

Re: VSS timeout with Exchange 2010

Veeam Logoby tfloor » Tue Dec 03, 2013 6:49 pm

I have a support ticket because I see the same with remote SQL server
tfloor
Expert
 
Posts: 270
Liked: 14 times
Joined: Tue Jan 03, 2012 2:02 pm
Full Name: Tristan Floor

Re: VSS timeout with Exchange 2010

Veeam Logoby yaroon » Mon Jan 20, 2014 4:35 pm

Our exchange 2010 test servers were having problems doing the snapshot-create within 20 seconds. Turns out that the number of vdisks in our config greatly influences the time it takes to create a snapshot. Our config has 4 standard disks (OS, App&Data, Pagefiles, WSB-backup -now obsolete) and 8 Exchange datadisks (all VMDK). Taking a snapshot in VCenter takes 20+ seconds, and even when VM is powered down, it takes 15 seconds.
When removing virtual disks from the VM, this time is reduced. So I reshuffled the database partitions from 1 database-per-disk tot 4 databases per disk (using 4 partitions), ending up with 2 database disks instead of 8. As this is our Exchange test environment, database size was not an issue. (100GB) - but with the new vSphere5.5 which breaks the 2TB vmdk limit, this could work for larger db's as well.
yaroon
Novice
 
Posts: 5
Liked: never
Joined: Thu Jun 16, 2011 8:08 am
Full Name: Jeroen Beekhuis

Re: VSS timeout with Exchange 2010

Veeam Logoby Andreas Neufert » Fri Jan 24, 2014 9:02 am

james575 wrote:
lobo519 wrote:Do you have the option "Snapshot virtual machine's memory" selected when you do this manually?


Yes. I have always have selected this for every snapshot I think I have ever made. After reading your question, I tried snapshotting the Exchange server with that option not selected--took two seconds! I always chose the memory option because I thought that would be the safer/better method, but it looks like I need to read up on that some. Thank you!

(As an informational aside for anyone reading this thread, our VSS writer error was fixed by turning on the log truncating for Exchange in the Veeam replication job.)



Backup Systems do not use the add memory to snapshot option. The idea is to bring the disks in a consistent state (VSS).
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: VSS timeout with Exchange 2010

Veeam Logoby Andreas Neufert » Fri Jan 24, 2014 9:20 am 2 people like this post

Hello Everybody.

Just want to add again my 2 cents here.

Feedback from my customer was, that in most cases adding more vCPUs and RAM to the Server fixed this problem, because Exchange can faster process it´s consistency and you do not run into 20sec timeout (see above).

Reducing vdisk amount can help to process snapshots faster.
If your vcenter is too slow, it can help to add ESXi host as managed server and select the exchange server over this esxi host.
All Exchange disks needs to stay on fast disks. Do not add the OS disk to a datastore with many other OS disks from different VMs.

Here are my updated general recommendations for Echange/Exchange DAG at VMware. (Not all of these are related to the 20sec VSS timeout thing):

1.
Increase the DAG heartbeat time to avoid cluster failover (no reboot or service restart needed) cluster /prop SameSubnetDelay=2000:DWORD cluster /prop CrossSubnetDelay=4000:DWORD cluster /prop CrossSubnetThreshold=10:DWORD cluster /prop SameSubnetThreshold=10:DWORD
2.
Use new Veeam Storage Snapshot Feature (StoreVirtual/3PAR/VSA) if you can (after v7 release) => Reduces Snapshot Lifetime to some seconds => No load and problems at commit because of less data. (This option can be counter productive if you experience the 20 sec VSS timeout)
3.
If you have problems with cluster failover at Backup, one option is to backup DAG member(s) that hold only inactive databases (no cluster failover because of no active databases) (Logfile Truncation will be replicated by Exchange in whole DAG). This give you also the option to restart the server or services and Exchange process VSS consistency more faster afterwards. If you restart the services, take care that you wait long enough afterwards that also the VSS Exchange writers come up again, before you backup.
4.
To reduce Snapshot commit time (and to reduce data in the snapshot), try to avoid any changes at the backup time window (User, Background processes, Antivirus, ....). Also try to avoid that on all LUNs on the storage System itself (faster writes at snapshot commit).
5.
If you can not avoid many changes on block level at your backupwindow? Use Forward Incremental or if you need space Forward incremental with daily transform into rollbacks. Reverse Incremental took a bit longer than the other backup methods => longer snapshot livetime => more changes in the Snapshot to commit
6.
To reduce Snapshot lifetime and reduce amount of data to snapshot commit, use new parallel processing with enought ressources to backup all of your disks at the same time (after v7 release)
7.
To reduce backup time window and snapshot lifetime, use Direct SAN Mode with minimal needed disks connected at selected Proxy. If not possible use NBD mode with 10GbE. (Do not run Proxy in Autoselect mode). Disable VDDK Logging for Direct SAN Mode if your backups themself run stable (ask support for the registry key and consequences).
8.
Use actual VMware Versions (newest VADP/VDDK Kits with a lot of updates in it) and actual Veeam Versions (newer VDDK Integration). And install actual ESXi/vCenter patches!
9.
Use at minimum VMware vSphere 5.0 because of changes in the snapshot places and Background things.
10.
If you still facing problems, use faster disks for all of the VM disks (also the OS disk!!!)
11.
Less VM disks can help to reduce snapshot creation (and commit?) time.
12.
To avoid VSS timeouts (hardcoded 20 seconds at Exchange VSS writer), Check your vcenter load and optimize it (or use direct ESX(i) Connections for Veeam VM selection, so that the snapshot creation took less time.
13.
If you facing VM downtime because of Snapshot commit, maybe another option (unsupported from VMware) is to change VM Setting snapshot.maxConsolidateTime = "1" (in sconds) (see discussion above).
14.
If you see Exchange VSS Timeout EventLog 1296 => Change Log setting => Set-StorageGroup -Identity "<yourstoragegroup>" -CircularLoggingEnabled $false
15.
VSS Timeout Problems => Add more CPU/Memory to the VM
16.
Check your health an configuration of Exchange itself. I saw some installations where different problems ended up with a high cpu utilization at indexing sevice. This prevented VSS to work correct. Check also all other mail transport-cache settings. Sometimes the caches replicate shapdows of the mails over and over again and nobody commit them (if you have multible transport services together with firewalls between them).
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: VSS timeout with Exchange 2010

Veeam Logoby cliffm » Sun Feb 23, 2014 9:55 pm

We have had this problem for years and have had many weekends spent on it. We bought Veeam with 5 years of support so obviously we are sticking to it. I went through the while process of upgrading the entire infrastucture to faster, bigger better physical hardware and networks. No dice. The only thing I have ever found that works is to turn vCentre server off. If there is a vCentre server running anywhere on the networks the VSS times out. This is strange because no Veeam backups or replicas I have configured use vCentre server VIX connections, they are all configured directly through standalone hosts. So now my only use of vCentre server is to turn it on to do host upgrades. Then I turn it off. Then my backups and replicas work.
cliffm
Enthusiast
 
Posts: 37
Liked: 3 times
Joined: Fri Jun 03, 2011 12:41 am
Full Name: Cliff Meakin

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 7 guests