VSS timeout with Exchange 2010

VMware specific discussions

Re: VSS timeout with Exchange 2010

Veeam Logoby Andreas Neufert » Mon Feb 24, 2014 8:18 am

Thanks for this feedback. If I run into this situation with a customer again. I will check your solution out and try to find the root cause, because I can not imagine why disabling the vcenter solved your VSS timeout problems. But again, thanks for sharing.
Andreas Neufert
Veeam Software
 
Posts: 2198
Liked: 357 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: VSS timeout with Exchange 2010

Veeam Logoby Vitaliy S. » Mon Feb 24, 2014 11:07 am

cliffm wrote:The only thing I have ever found that works is to turn vCentre server off. If there is a vCentre server running anywhere on the networks the VSS times out. This is strange because no Veeam backups or replicas I have configured use vCentre server VIX connections, they are all configured directly through standalone hosts.

That's really strange, do you have both job logs to compare the procedures triggered with vCenter Server powered On and Off?
Vitaliy S.
Veeam Software
 
Posts: 19541
Liked: 1098 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VSS timeout with Exchange 2010

Veeam Logoby cliffm » Wed Feb 26, 2014 2:55 am

Hi Vitaly,

I went through this for weeks with support some 3 years ago. In the end we could not get VSS to work at all and I stumbled on the vCentre idea by accident. Support reproduced my environment and could not get it to work either. Last month I upgraded to ESXi5 and decided to try out vCentre 5 again, thinking maybe things are better now. They are certainly faster with bigger hardware. Bang, almost immediately I started getting VSS errors in SBS backups that have been reliable for years. The jobs are not configured through vCentre. I am again frustrated and at the point of turning it off, again. I really don't want to spend more time on this so in short no, I don't have the logs. Also have just moved B&R today to a new server, migrated the database and settings etc. and turned the old one off. I am having VSS failures so if they don't stop the vCentre is in mothballs again for me.
cliffm
Enthusiast
 
Posts: 37
Liked: 3 times
Joined: Fri Jun 03, 2011 12:41 am
Full Name: Cliff Meakin

Re: VSS timeout with Exchange 2010

Veeam Logoby cliffm » Wed Feb 26, 2014 3:01 am

And just to put the icing on the cake, I just turned off vCentre and my SBS is happily producing snapshots again.
cliffm
Enthusiast
 
Posts: 37
Liked: 3 times
Joined: Fri Jun 03, 2011 12:41 am
Full Name: Cliff Meakin

Re: VSS timeout with Exchange 2010

Veeam Logoby Vitaliy S. » Wed Feb 26, 2014 10:25 am

Well...if a pure standalone connection to ESXi hosts is used every time, then vCenter Server power state should not affect anything. I will ask our QC team if they've seen anything similar before, thanks for the feedback.
Vitaliy S.
Veeam Software
 
Posts: 19541
Liked: 1098 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VSS timeout with Exchange 2010

Veeam Logoby nmace » Fri Mar 07, 2014 9:40 pm

I have not read through all of this giant thread, but for 3 days my Exchange backups have failed with a "Error: VSSControl: Failed to freeze guest, wait timeout".

Here are the things I've already tried courtesy of http://www.veeam.com/kb1680.

I've rebooted Exchange (no effect), I've connected to the Exchange VM via the host and not vCenter (no effect), and I've done an Exchange backup using the built in Windows Backup Tool. That succeeded fine. Running "vssadmin list writers" reports that the Microsoft Exchange Writer has timed out. Per the logs in vCenter, all of my Exchange snapshot creation times are about 22-25 seconds, which is outside of the 20 second window. This started happening as we moved more and more people onto the Exchange VM from a physical Exchange server. Am I really looking at the prospect of needing faster storage? Would I be able to utilize Storage IO Control on the VM instead to make sure it's getting enough disk performance instead? We have Enterprise+ for this cluster's vSphere hosts.

Is there anything else I should be trying?
nmace
Enthusiast
 
Posts: 93
Liked: 12 times
Joined: Mon Jul 23, 2012 3:48 pm

Re: VSS timeout with Exchange 2010

Veeam Logoby Vitaliy S. » Sun Mar 09, 2014 3:06 pm

Moving this VM to a faster storage should definitely help. On a side note, seems like the latest recommendation from Andy can also help > vmware-vsphere-f24/vss-timeout-with-exchange-2010-t5760-225.html#p101378
Vitaliy S.
Veeam Software
 
Posts: 19541
Liked: 1098 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VSS timeout with Exchange 2010

Veeam Logoby kinger » Mon Mar 10, 2014 2:22 pm

Hello, im new in this forum however I have problem with costumer - he faced the VIX unfreeze timeout error. How Can I fix it?
kinger
Lurker
 
Posts: 1
Liked: never
Joined: Mon Mar 10, 2014 2:10 pm
Full Name: Wojtek

Re: VSS timeout with Exchange 2010

Veeam Logoby v.Eremin » Mon Mar 10, 2014 4:46 pm

Hi, Wojtek,

If I were you, I would analyse thoroughly the existing KB article regarding this matter, and see whether the tips mentioned there helps to solve VSS timeout issue. If nothing helped, I would probably start thinking about new faster storage.

Thanks
v.Eremin
Veeam Software
 
Posts: 13255
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: VSS timeout with Exchange 2010

Veeam Logoby anubis » Mon May 26, 2014 9:08 am

Hello,

regarding the article http://www.veeam.com/kb1680. Could you please confirm, if I understood the process right? :

1- Veeam requests (via admion share or VIX) the VSS service to prepare for a backup (freeze the disks).
2- The VSS service freezes the disks
3- Veeam/VMware creates a snapshot (using Vcenter/ESXi)
4- Veeam/Vmware notifies the Microsoft VSS service that it can unfreeze the disks.
5- Veeam copies the data from the Vmware Snapshot

Between the steps 2 and 4, we should have less than 20 seconds, or the Exchange Writer will unfreeze the disks aborting the process.

From the article : "So, between step 2.1(when countdown begins) and step 2.3 we have only 20 seconds. If Veeam doesn't freeze the VM in timely manner, Windows VSS subsystem does it by itself. Hence you get unfreeze error." . Shouldnt this read "Windows VSS subsystem unfreezes the disks by itself"?

This line is also listed in the solutions : "VMware snapshot takes longer than 20 seconds (hardcoded Exchange VSS Writer timeout)" .. This should be part of the cause, not the solution IMHO.
anubis
Lurker
 
Posts: 1
Liked: never
Joined: Mon May 26, 2014 8:49 am
Full Name: Anubis

Re: VSS timeout with Exchange 2010

Veeam Logoby Andreas Neufert » Tue May 27, 2014 10:35 pm

Hi Anubis,

thanks for the feedback.
You are right with the timeline...
1- => We will start an VSS requestor and this triggers the VSS writers.

I think the message came from Exchange VSS framwork itself and is bypassed to the Veeam UI.

I have created a good blog post with tips an tricks round about this:
http://neufert-at-veeam.blogspot.de/201 ... dated.html
Andreas Neufert
Veeam Software
 
Posts: 2198
Liked: 357 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: VSS timeout with Exchange 2010

Veeam Logoby sambooka » Thu Jun 12, 2014 2:36 pm 2 people like this post

I have been banging my head with this issue for 3 weeks now Case # 00572707. I consider the issue resolved in my case and am about to move on. This will not be a solution for everyone but maybe it applies to someone and will help. No one at veeam was able to confirm my hypotheses so I just did it. It worked.

Background:
Exchange 2010 SP2 RU5.
2 node DAG
All Active DBs on Node1
All passive DBs on Node 2
PUB folder on Node 1 and 2

VSS Freeze timeouts on back. No log tuncation.

Still failing. We see the failures are specifically with the PUB folder DB.
Tried a cagillion things. Reregistering the VSS dlls .. restarting services. Doubled the CPU on Node 2 (passive node). Pretty much everything in this thread. Spoke to Nutanix support. Nada.. Spoke with Veeam support. VMWare support says nothing wrong on their end.. talk to Veeam. Veeam Tier2 says it is a VSS issue.. try to contact MS. I have not contacted MS but have read that MS doesnt not support snapshoting technologies and they dont support NFS storage (Nutanix). So I removed the Pub Folder DB from the server. Full backup complete. Logs Truncated. Exchange Explorer for Veeam sees the stores. Perfect. I have not tried a full restore of the server but dont see why it wouldnt work.

TLDR: VSSTimeouts on passive node of Exchange Dag. Fix: Passive node had Pub folder DB for HA. Removed the pub folder. Resolved.
sambooka
Lurker
 
Posts: 1
Liked: 2 times
Joined: Thu May 22, 2014 3:19 pm
Full Name: Boo

Re: VSS timeout with Exchange 2010

Veeam Logoby Andreas Neufert » Thu Jun 12, 2014 6:28 pm

Thank you very much for figuring this out. Great work. I will let support know and update my recommendation list soon.
Also I will test it in my environment. Can you please share the configuration of the public folder dbs with me (screenshots or PS output) so that I have a higher chance to reproduce that. andreas.neufert(at)veeam.com. Thanks in advance...Andy
Andreas Neufert
Veeam Software
 
Posts: 2198
Liked: 357 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: VSS timeout with Exchange 2010

Veeam Logoby bairda » Sat Aug 09, 2014 9:31 pm

I have had the VSS timeout errors and the ONLY fix that worked was enabling a network connection between the Veeam B&R server and the problem exchange 2010 servers.

With this connection you must either disable the firewall or put in rules to pass the traffic. Far from a perfect solution, but it is the only way that the freezes would come in under 20 seconds and proceed with the application aware exchange backups via Veeam. This works via Vcenter VM add or via ESXi host add.

I love the Veeam product but I wish Veeam would fix this particular issue or just state that you need to put a network connection to the vB&R and the exchange server VM's.

Yes, MS needs to lengthen the Timeout on the freeze. But those are the cards that Veeam is currently dealt and the conditions it has to work under. The connection-less, VIX method of exchange snapshots for Veeam backups don't seem to work consistently.

BTW...this doesn't seem to impact any other backups except Exchange. Those can go without a network connection between the veeam B&R and the VM's.
bairda
Novice
 
Posts: 4
Liked: never
Joined: Fri Jan 21, 2011 7:38 pm
Full Name: Alex Baird

Re: VSS timeout with Exchange 2010

Veeam Logoby Andreas Neufert » Sat Aug 09, 2014 9:47 pm

Hi, did you tried to reverse Network/VIX prozessing as descriebed in my guide?
If not, B&R wait for network timeout first and then process VIX based processing.

If you only perform VIX processing, this is a very good idea in any way.

I saw some cool improvements in v8beta1 as well, if you have the chance to test it with your Veeam SE let us know here.
As you process VIX only, the registry setting for reversing processing order needs to be set as well.
Andreas Neufert
Veeam Software
 
Posts: 2198
Liked: 357 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: Dima P., Exabot [Bot], stehei and 29 guests