Host-based backup of VMware vSphere VMs.
Post Reply
habibalby
Veteran
Posts: 392
Liked: 33 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: VSS timeout with Exchange 2010

Post by habibalby »

waiting for your result @stormlord
stormlord
Novice
Posts: 6
Liked: never
Joined: Dec 15, 2011 1:40 am
Contact:

Re: VSS timeout with Exchange 2010

Post by stormlord »

same error on direct host connect.
I will move the datastore to an other san this weekend.
I come back.
murda
Enthusiast
Posts: 38
Liked: never
Joined: Dec 18, 2009 10:05 am
Full Name: murda
Contact:

Re: VSS timeout with Exchange 2010

Post by murda »

Well HP dataprotector has a nice way of backing up an Exchange 2010 DAG Cluster. Maybe Veeam should look at a same solution.

I will test Veeam Exchange DAG 2010 backup next week but I doubt it can directly contact the cluster to check on which server the mailboxes are residing.
I don't know how you backup a DAG in Veeam? Backup all the Exchange 2010 VM's??
This will get you in trouble sooner or later because you will miss eventually an exchange database that has been swapped from mail server during backup of your exchange servers...

Explain this to your boss if your backup has cheese holes (if this works like I think it does) :?:
elliott
Influencer
Posts: 11
Liked: 7 times
Joined: Jul 06, 2011 12:43 am
Contact:

Re: VSS timeout with Exchange 2010

Post by elliott »

OK despite my assurance deleting snapshots fixed my problem, it didn't..
Upgraded to V6 to test it, same issue...

I think I have found at least a partial cause:
I just had a look at what services aren't running.. it appears the "COM+ System Application" service is stopped whenever the backup fails. If i manually start this, then run the backup, it works. The service will then stop after a few minutes (normally, no logged errors)...

So why does the COM+ System Application service stop..? And why is it causing the Exchange VSS writer, and subsequently Veeam, to fail?...
elliott
Influencer
Posts: 11
Liked: 7 times
Joined: Jul 06, 2011 12:43 am
Contact:

Re: VSS timeout with Exchange 2010

Post by elliott »

And it appears the Veeam backup, or some VSS related task instigated by the Veeam backup, causes the COM+ service to stop. I can start it at midday, and it will continue to run until midnight when the backup begins. The backup will succeed if COM+ System Application serivice is running, but it will stop and not restart itself - subsequent backups (ie the next night's backup) will then fail until COM+ System Application is started again.
elliott
Influencer
Posts: 11
Liked: 7 times
Joined: Jul 06, 2011 12:43 am
Contact:

Re: VSS timeout with Exchange 2010

Post by elliott »

Seems to stop at about the same time as event 9608 (Application log, MSExchangeIS) is logged. I've set a scheduled task to "net start COMSysApp" on that event being logged, so we will see what happens...
awatkins31
Service Provider
Posts: 6
Liked: 1 time
Joined: Jul 14, 2009 8:00 pm
Full Name: Adam Watkins
Contact:

Re: VSS timeout with Exchange 2010

Post by awatkins31 »

Environment consists of:
Veeam 6.0.0.158 64 bit
Windows 2008 R2 SP1 physical machine for veeam backup server/DAS for repository
ESX 4.1 5 host Enterprise cluster using iSCSI SAN
Direct SAN access mode for backup method

Here are my observations on this topic and what I did to fix the issue.

I was having the intermittent issues with backing up Exchange 2010 SP1 on W2k8 R2. The Exchange 2010 jobs were failing with the veeam error:
Freezing guest operating system
Unfreeze error (over VIX): [Backup job failed.]

After a failed backup, within windows, vssadmin list writers was showing Microsoft Exchange Writer "timed out". This would not happen every time, but around 90-95% of the Exchange 2010 jobs were failing when using application aware processing (i.e. vss). Other machine types (domain controllers, file servers, etc.) backed up successfully using vss. Also, for the Exchange servers that were failing in veeam, I was able to use windows server backup to make successful backups.

I had tried all forum suggestions ranging from moving guest machines to faster SAN, moving vcenter (it is a vm) to faster SAN, increasing RAM on vcenter, running backups on machines that were on lightly loaded ESX hosts, etc.

My environment consists of a multi-tenant cluster that is segmented into multiple port groups/vlans. The backup server DID NOT have access to all of those vlans due to firewall rules. I did have one Exchange server that consistently backed up successfully that happened to be an internal company Exchange server that DID have network connectivity between the veeam backup server network and the Exchange server. Using that information along with a post in this forum topic by "sami.laihorinne" related to capturing packets between a guest OS and the veeam backup server, I decided to do some investigation into the network side of things.

I created firewall rules that allowed traffic between the Exchange servers that were failing and the veeam backup server. I installed wireshark on the veeam backup server, started a veeam job for backing up an Exchange server, and started a trace looking for traffic between the Exchange server being backed up and the veeam backup server. Lo and behold, after creating the firewall rules allowing traffic between the Exchange networks and the backup server, the backups started working successfully; during the vss freeze/thaw, wireshark was capturing packets that were going to and from the veeam backup server and the Exchange 2010 server. So, there appears to be some traffic in the VSS processing that was trying to be exchanged between the guest OS and the veeam backup server. When this traffic was blocked by the firewall rules, the VSS event for Microsoft Exchange was timing out. When the traffic was allowed, VSS processing was proceeding without error.

According to Veeam documentation, since veeam 5, a direct network connection between the veeam backup machine and the guest OS is not supposed to be required. I was having these same backup issues with Exchange 2010 using Veeam 5. NOTE: I have not done the same test on 5 as we are now running on 6, but I have to assume the same thing holds true (don't hold me to that though as I have not done the actual test). It seems, in my experience, that backing up Exchange 2010 in Veeam 6 DOES require a network connection between the guest OS and veeam backup server (at least to get 100% successful backups). Perhaps the code from version 4 is still there as a "fallback" for direct connection to guest OS for when VSS processing does not work properly in the newer versions.

I have now setup the firewall rules for all of the Exchange networks that were suffering from this issue (I have about 12 Exchange servers on this cluster) and have not had a failed backup on those Exchange servers. I did wireshark captures on at least 4 of the Exchange backups and all of them were showing traffic flowing between the guest Exchange machine and the veeam backup server during the vss processing.

I would prefer to NOT have firewall policies allowing this traffic between the guest OS and the veeam backup server, but at least it has gotten me to the point of having successful backups of all the Exchange servers. I had opened a support ticket on this in the past with no resolution, but for me, this is at least a work around until Veeam can help either fix the product so that it does not require direct access to the guest OS for Exchange backups or explain what I am doing wrong that is causing this. I plan to open a ticket in the near future so that I can share packet captures, etc. with Veeam support/engineering.

Hope this helps someone else.

Adam
Andreas Neufert
VP, Product Management
Posts: 7077
Liked: 1510 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VSS timeout with Exchange 2010

Post by Andreas Neufert »

Hi Adam,

just a small idea.
Please try to use the local administrator for VSS. (In the Job - VSS-Advanced button you can change that for each VM separately.
I think/hope that prevent your described network traffic. Please try it an post your results.

Because of some microsoft restrictions (UAC & Co) the VSS User Account have to be exactly "Servername\Administrator".
If you change the local Administrator name to a different name you will run in some problems.

CU Andy
Andreas Neufert
VP, Product Management
Posts: 7077
Liked: 1510 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VSS timeout with Exchange 2010

Post by Andreas Neufert »

@murda

Include all Exchange DAG members in one job.
Veeam works on a block level so every database is backed up from a DAG Server.
Because of our deduplication function the amount of data stored are acceptable. So no cheese whole.

If you have TBs of Exchange DBs.
I recommend to work in paris. Each DAG Pair holds the same DBs.

But if you decide to store DBs on random DAG members, you only need to take care that every DAG Server is backed up.

CU Andy
Andreas Neufert
VP, Product Management
Posts: 7077
Liked: 1510 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VSS timeout with Exchange 2010

Post by Andreas Neufert »

@all with snapshot freeze problems.
@all with DAG cluster pans

NFS Datastores => Install VMware fixes (symtom: snapshot freezes at snapshot delete)
SAN Datastores => Install latest VMware Versions and check your HBA/datastore access profile if it suites your SAN Storage (Dedicated/Rounrobin/...)
iSCSI Datastores => Install latest VMware Versions and check your HBA/datastore access profile if it suites your SAN Storage (Dedicated/Rounrobin/...) + Use a dedicated enterprise switch for iSCSI VMware traffic
Update your SAN/iSCSI/NAS Firmware (in case of VMware snapshot commit/delete VMware writes a large amount of random writes) I saw a lot of old firmwares that have problems with that.


Do you use Disk System based sync mirroring?
To check if this is the problem: Disable Storage System syncron mirroring (I saw some systems that perform not well beacues of firmware bugs)

To check out if your Disk/network environment have problems, you can use local disks to check this out.(Storage vmotion of all Volumes)

If you have DAG cluster pans, try the above and then increase the cluster heartbeat time. See:
http://technet.microsoft.com/en-us/libr ... S.10).aspx
http://www.extropy.com/forums/knowledge ... g-over-wan

And use NTP Servers for time sync on each VMware host and VM:
http://kb.vmware.com/selfservice/micros ... nalId=1318
awatkins31
Service Provider
Posts: 6
Liked: 1 time
Joined: Jul 14, 2009 8:00 pm
Full Name: Adam Watkins
Contact:

Re: VSS timeout with Exchange 2010

Post by awatkins31 »

Andreas, I will give that a try. As it currently stands, the username I am using is the domain administrator or an account with domain administrator rights set in veeam with that following syntax: "domainname\username". I will try "computername\administrator" for those Exchange servers where it is possible. Unfortunately, on a couple of machines, since they are hosted exchange, we are breaking Microsoft best practice and running the DC role and Exchange services on a single vm. In those instances, a local user account won't necessarily be allowed since an AD server does not have local user accounts any longer.

when I get a chance to change the accounts, do another backup and capture packets, I will post results.

Adam
Bribo
Novice
Posts: 4
Liked: never
Joined: Jan 01, 2006 1:01 am
Full Name: Brian Speerly
Location: Madisonville, LA
Contact:

Re: VSS timeout with Exchange 2010

Post by Bribo »

I've been following this thread with much interest. I don't think I have an Exchange server at all that I can backup with Veeam. We have multiple clients with different types of networks that cannot backup Exchange. We have a few that are in a Shared Datacenter like awatkins31, but the hosts do have connectivity between them, unlike his original setup, but we also have clients with 2 VMware hosts with DAS , no VC, Veeam server virtual or Physical depending, and they all fail with VSS errors in Veeam. I can back those servers up with Mozy, or with NTBackup, using VSS no problem. I think I'm going to have to stop selling Veeam, as if it cannot do the job, there are other solutions. I hope the Veeam support folks are taking this seriously. I haven't seen any of them posting in awhile, and when they did, they just seemed to point the finger at MS, but we are all spending money and staking our reputation on their software, we need to get a fix for this.
Vitaliy S.
VP, Product Management
Posts: 27371
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VSS timeout with Exchange 2010

Post by Vitaliy S. »

Bribo wrote:I hope the Veeam support folks are taking this seriously. I haven't seen any of them posting in awhile, and when they did, they just seemed to point the finger at MS, but we are all spending money and staking our reputation on their software, we need to get a fix for this.
Our technical support team does not monitor these forums, that is why we always ask to open a support ticket on any technical problem you might have.
Andreas Neufert
VP, Product Management
Posts: 7077
Liked: 1510 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VSS timeout with Exchange 2010

Post by Andreas Neufert »

I forgott in my previous posts to say the following:

Often Veeam VSS fails because there were an Agent from an other backup vendor present. This software often places his VSS skripts in the VMware Tools Script environment.
So our job starts this scripts, too ... and fail because of double vss commands.

Please have a look at the following places:
C:\Windows\<pre-freeze-script.bat>
C:\Windows\<post-thaw-script.bat>
C:\Program Files\VMware\VMware Tools\backupScripts.d\

CU Andy
Andreas Neufert
VP, Product Management
Posts: 7077
Liked: 1510 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VSS timeout with Exchange 2010

Post by Andreas Neufert »

Our Support Knowledge base can also be helpful:
http://www.veeam.com/kb_search_results.html/vss/All/1/
SHSDavid
Novice
Posts: 4
Liked: never
Joined: Sep 19, 2011 4:01 am
Contact:

Re: VSS timeout with Exchange 2010

Post by SHSDavid »

Andreas Neufert wrote:Hi Adam,

just a small idea.
Please try to use the local administrator for VSS. (In the Job - VSS-Advanced button you can change that for each VM separately.
I think/hope that prevent your described network traffic. Please try it an post your results.

Because of some microsoft restrictions (UAC & Co) the VSS User Account have to be exactly "Servername\Administrator".
If you change the local Administrator name to a different name you will run in some problems.

CU Andy
This solved the issues I was having :D
Andreas Neufert
VP, Product Management
Posts: 7077
Liked: 1510 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VSS timeout with Exchange 2010

Post by Andreas Neufert »

Hi everbody,

again a tip for DAG Users.

http://technet.microsoft.com/en-us/libr ... 51172.aspx
"You can specify only a name for the DAG and leave the Witness Server and Witness Directory check boxes cleared. In this scenario, the wizard will search for a Hub Transport server that doesn't have the Mailbox server role installed. It will automatically create the default directory and share on that Hub Transport server and use that server as the witness server."

If you try to start a DAG cluster member in the lab (SureBackup, Exchange Restore-Wizard).
You need a running
a) a CAS Server that holds no mailbox role
or
b) the Server which holds the manual selected Witness Server/Folder
(depends on your DAG Setup - see above)
befor you start a DAG member.

==================================
Example SureBackup Job:
Application Group:
1. AD Server
2. CAS Server without mailbox role or dedicated Witness Server/Folder (depending on your setup)
SureBackup Job:
Linked to one or more DAG Cluster member backup job
==================================
Example manual Restore Application Group:
1. AD Server
2. CAS Server without mailbox role or dedicated Witness Server/Folder (depending on your setup)
3. DAG Memebers
==================================

Without a running Witness Role you run in trouble if you try to start a DAG cluster memeber.
No Witness => No running cluster service after DAG Server boot => no running Exchange information store

If someone had other tip or workarounds for other Exchange related topics please feel free to post it here.

THX Andy
Dan_D
Lurker
Posts: 1
Liked: never
Joined: Mar 04, 2012 2:56 am
Contact:

Re: VSS timeout with Exchange 2010

Post by Dan_D »

Hi all,

I have just encountered the same issue with the Exchange Writer timing out after upgrading one of my first client sites to Veeam 6.0. In this case, I am a MSP and the client's site was very simple, several VM's runnign on ESXi 4.1 U2 with all patches etc

- SBS 2011
- Veeam Backup/Management Tools etc
- Sharepoint
- SQL
- VCenter

After the upgrade to 6.0 (and initially dealing with the silly issue where Veeam decides to backup to SysWow64 folder on C: of the Veeam server!), the Exchange Writer would always fail and timeout when backing up the SBS VM. I came across Adam's post in this read re: firewalls and decided to give that a shot. I allowed all traffic from the Veeam server to the SBS 2011 box and bingo, fixed the problem instantly. In this case, it is a very simple network topology - one flat subnet and all servers/PC's etc on that same subnet.

I'm not overly concerned with that firewall rule being in place in this particular organisation (would prefer that it didn't have to be the case, but still...)

In any case, happy that it is solved.

Thanks

Dan
singy2002
Service Provider
Posts: 66
Liked: 1 time
Joined: Mar 22, 2010 8:43 am
Full Name: Andrew Singleton
Contact:

Re: VSS timeout with Exchange 2010

Post by singy2002 »

After having this issue intermitently for months on several exchange servers I have just tried to create a firewall rule between the two networks performing the backups, the backup then immediately works.

The problem is I cannot leave all ports open between the backup boxes and the destination exchange servers (Hosting company)

What I am most puzzled about is why the job sometimes works (10%) without the ports open and mostly does not (90%) but yet when i open the ports it works every time.

The ports I have witnessed it using are very high and presumably random 35598, 22XXX, etc

Is there a defined list of what ports are used, as i dont recall seeing this before?
Andreas Neufert
VP, Product Management
Posts: 7077
Liked: 1510 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VSS timeout with Exchange 2010

Post by Andreas Neufert »

@dan. And @singy

What user are you using for VSS ?

Domain Admin ?

Try to use the lokal User "Administrator" and disable your added Firewall rules.

Give it a try ... Andy
Vitaliy S.
VP, Product Management
Posts: 27371
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VSS timeout with Exchange 2010

Post by Vitaliy S. »

singy2002 wrote:Is there a defined list of what ports are used, as i dont recall seeing this before?
You can find this list in our User Guide (pages 58-59).
singy2002
Service Provider
Posts: 66
Liked: 1 time
Joined: Mar 22, 2010 8:43 am
Full Name: Andrew Singleton
Contact:

Re: VSS timeout with Exchange 2010

Post by singy2002 »

Just to follow up on this.

I have tried both local and domain admin accounts, I have opened all ports between backup proxy/VM and the backup is still very intermittent as previously mentioned.
singy2002
Service Provider
Posts: 66
Liked: 1 time
Joined: Mar 22, 2010 8:43 am
Full Name: Andrew Singleton
Contact:

Re: VSS timeout with Exchange 2010

Post by singy2002 »

We have eventually got round this issue, not i might add with any of the suggested fixes.

The Veeam backup server was attempting to freeze the guest OS, this was timing out after the 59-60 seconds in every instance. After one fo my collegues found some other info we thought to try to add the Veeam server to the same VLAN as the Exchange server (Hosted environment) and then the backup worked each time.

We have a hosting domain containing all VMware infrastructure and backup servers and then customer domains within this. the exchaneg box in question was identical to 10 or so others that were all working with being on the same VLAN, but we tried to anyway. The free now takes 20 odd seconds so the 60 second timeout was just default as the server could not authenticate with the exchange box.

This is the only server were we have had to add VLAN access tot he Veeam box, other customer hsoted exchange noxes work without issue.
Andreas Neufert
VP, Product Management
Posts: 7077
Liked: 1510 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VSS timeout with Exchange 2010

Post by Andreas Neufert »

Hi Singy,

thanks for your post.
If you use the local Administratore "PCNAME\Administrator" for VSS, you need no network connection for VSS.
CU Andy
singy2002
Service Provider
Posts: 66
Liked: 1 time
Joined: Mar 22, 2010 8:43 am
Full Name: Andrew Singleton
Contact:

Re: VSS timeout with Exchange 2010

Post by singy2002 »

Despite being told this many times, that is not correct, we tried this outselves and with support and it made NO difference.

Please feel free to discuss this with your support department who will show it does not work. [ID#5181412] Exchange VSS failures
Andreas Neufert
VP, Product Management
Posts: 7077
Liked: 1510 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VSS timeout with Exchange 2010

Post by Andreas Neufert »

Hi Singy,

thanks again for your feedback. I´m not a support member and I do not know your environment, but wants to help here.
This is only a try but maybe it´s working. Maybe its because of some security settings in win 2008 r2.
Gave this a try, because that enables also the local things to connect to that share.
http://windowsarchitecture.wordpress.co ... 12/23/187/
If this dosn´t help, change it back.

I need some time to prepare my lab for a Exchange 2010. I will try this out and came back here in some days.

CU Andy
Andreas Neufert
VP, Product Management
Posts: 7077
Liked: 1510 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: VSS timeout with Exchange 2010

Post by Andreas Neufert »

If you have some space related problems with vss ... step 2 of this topic: Veeam Best Practises?, might be helpful.
KevinRFinken
Lurker
Posts: 1
Liked: never
Joined: Apr 24, 2012 4:00 pm
Full Name: Kevin R. Finken
Contact:

Re: VSS timeout with Exchange 2010

Post by KevinRFinken »

I was having this same problem and the fix for me was to start the COM+ System Application service on the mail servers. I changed the service to Automatic to hopefully prevent this problem in the future. Once that service was started the backup ran perfectly.
PedroVDP
Service Provider
Posts: 19
Liked: 2 times
Joined: Jan 18, 2012 9:10 am
Contact:

Re: VSS timeout with Exchange 2010

Post by PedroVDP »

Hey everyone,

I was fighting with my own Exchange DAG cluster being iffy in getting backed up but I was able to resolve it by shifting my backup window to a less busy period overall on the environment. Oddly enough this is smack in the middle of the day.

Searching high and low for an answer because I was fairly convinced the root cause was not Veeam I found information indicating that when you perform a VSS freeze on a DAG cluster, you have exactly 60 seconds to do what you need to do and initiate the unfreeze. If not, Exchange on its own will initiate the unfreeze and abort the backup.

This means you have 60 seconds from the the point in time VSS signals it is frozen to create a VMware snapshot and start the unfreeze. Try rescheduling your backups to where you know your VBR servers aren't under heavy load and where your VMware environment isn't either. It worked for me.
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: VSS timeout with Exchange 2010

Post by dellock6 »

Keep in mind that (it there are no new statement from Microsoft I do not know about) officially snapshots of Exchange 2010 are NOT supported:

http://www.thelowercasew.com/virtual-ma ... -supported

There are also tricks to let vmotion act gracefully on the DAG nodes, basically extending the DAG heartbeat timeout. I'm curious if there would be a similar registry hack to let VSS have more time to freeze the node itself.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Post Reply

Who is online

Users browsing this forum: Semrush [Bot] and 92 guests