VSS timeout with Exchange 2010

[DaxUK]

[merged]

Hi,

I am trying to backup a Exchange 2010 SP1 server with Veeam 5.02.230.

VMWare tools is installed on the system and I am using "Enable applications-aware image processing" in Veeam.

The backup works from time to time but mostly fails with the error:

Freezing guest operating system
Unfreeze error (over VIX): [Backup job failed.]

It seems to me by looking at the logs on the exchange server that the job may be clashing with the DAG replication.

Am I missing something I should be enabling in the backup task to work with DAG database groups?

Regards
Dax

[Stephan]

Hi Guys

Well I have the exact same problem as Rowdy who started this thread.

VMWare. Two very powerful hosts. And my "run of the mill" network built on VMWare. 2 DCs (one of them a file server), vSphere, an Exchange 2010 server, and a few more VMs like a WSUS server, a Mc Afee EPO server and a TMG server, but those are not applicable now.

Running VMWare 4.1 and Veeam 5.0.

My first backup went successful, each one after that fails with that same error:

Freezing guest operating system
Unfreeze error: [Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{a2b7f7fa-c7d9-40ac-a14c-bc85503cd358}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].]

My Windows event logs says this:

Exchange VSS Writer (instance 7b5ee970-1f72-455a-ac9f-172c4799e192) has prepared for backup successfully.
Information Store (4372) Shadow copy instance 1 starting. This will be a Full shadow copy.
Exchange VSS Writer (instance 1) has successfully prepared the database engine for a full or copy backup of database 'Mailbox Database 0685136214'.
Exchange VSS Writer (instance 7b5ee970-1f72-455a-ac9f-172c4799e192:1) has prepared for Snapshot successfully.
Information Store (4372) Shadow copy instance 1 freeze started.
Information Store (4372) Mailbox Database 0685136214: Shadow copy instance 1 freeze started.
Exchange VSS Writer (instance 7b5ee970-1f72-455a-ac9f-172c4799e192:1) has frozen the database(s) successfully.
Information Store (4372) Shadow copy instance 1 freeze ended.
Information Store (4372) Shadow copy instance 1 aborted.

After the failed backup I also get:

Writer name: 'Microsoft Exchange Writer'
Writer Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
Writer Instance Id: {a50dc7da-5f96-4056-ac5a-918c3c363406}
State: [9] Failed
Last error: Timed out

I installed the Windows Backup feature that also uses the same VSS Writers, and that backs up 100%, no problems, so I know the VSS Writers are working fine. Here are my logs when I back up using the plain windows backup:

Exchange VSS Writer (instance d17ecbd1-181a-441e-84e7-bd2ea6abf884) has prepared for backup successfully.
Information Store (4420) Shadow copy instance 1 starting. This will be a Copy shadow copy.
Exchange VSS Writer (instance 1) has successfully prepared the database engine for a full or copy backup of database 'Mailbox Database 0685136214'.
Exchange VSS Writer (instance d17ecbd1-181a-441e-84e7-bd2ea6abf884:1) has prepared for Snapshot successfully.
Information Store (4420) Shadow copy instance 1 freeze started.
Information Store (4420) Mailbox Database 0685136214: Shadow copy instance 1 freeze started.
Exchange VSS Writer (instance d17ecbd1-181a-441e-84e7-bd2ea6abf884:1) has frozen the database(s) successfully.
Information Store (4420) Shadow copy instance 1 freeze ended.
THIS IS WHERE VEEAM FAILES.
Exchange VSS Writer (instance d17ecbd1-181a-441e-84e7-bd2ea6abf884:1) has thawed the database(s) successfully.
Exchange VSS Writer (instance d17ecbd1-181a-441e-84e7-bd2ea6abf884:1) has processed the post-snapshot event successfully.

I don't have DAG. I 've never run Backup Exec on this network. The recources on the hosts and VMs are more than fine. Have tried moving the Exchange and vSphere to their own powerfull host. My exchange Logs and Data Store each have their own 15K RPM drives on our SAN, so those are fine. I've tried waiting till 10PM at night to run the backup to make sure there is zero activity on the network and exchange server . . . What do I do?!?!?

I have a support ticket open now 23 days and no luck so far . . .

[Vitaliy S.]

Hello Stephan,

Unfortunately, I do not have deep expertise in troubleshooting Exchange VSS writers problems, so I cannot offer much help here. But have you tried running both backup solutions one by one? Since we just issue an API call to trigger VSS Exchange writer, I would try to look through VSS writers debug logs or ask assistance from Microsoft team directly.

By the way, have you found any relevant information on this error ID - [0x800423f2]?

Thank you.

[Stephan]

Hi

That code is a timeout. This from Microsoft:

During snapshot creation, the freeze event suspends all program write activity for data consistency. For example, an SQL database must be in freeze mode for snapshot creation. The thaw event releases the program after successful snapshot creation. The snapshot creation process takes less than 60 seconds.

When the freeze event is called, a worker thread is created to issue a time-out after 60 seconds. If the thaw event is not called within 60 seconds, the worker thread issues a time-out.

This problem occurs if the thaw event is called before the worker thread is started. The worker thread issues a time-out after the thaw event is called and causes the VSS writer to fail.

I monitor Veeam and exchange logs side by side on two screens when trying to backup.

I start the backup at 9:53:36. At 9:55:27 the exchange server’s logs says “Information Store (4420) Shadow copy instance 2 aborted.” While Veeam’s still saying “Checking the license for the host 192.168.0.3 . . .

[Vitaliy S.]

Stephen, your issue needs to be investigated by reviewing all log files, which we cannot really do on these forums. So I suggest that you keep working with our support team and please engage guys from Microsoft to shed some light on why VSS writer fails here:

Information Store (4372) Shadow copy instance 1 freeze ended.
Information Store (4372) Shadow copy instance 1 aborted.

[joergr]

People,

as i am quite familiar with exchange 2010 with or without DAG Systems when it comes to VSS errors, let me list my experiences for the reasons of these errors:

a) 20% some third party vss agent runs on the same vm - please remove it
b) 20% general vss initialization problems, even higher focus than the exchange vss itself. Got to Computer/Disks/locate the Disk where the Database relies on, go enable volume snapshot scheduled copies, it will create one immideately for you, just delete it again and disable volume snapshot schedules again.
c) 60% (and most important) - STORAGE PERFORMANCE!!!! Just think about what happens when external vsphere data protection api kicks in (no matter if san, nbd or hot add): Not only an internal vss trigger is performed inside the guest, there ALSO (you always talk only about Microsoft and VEEAM - guys, you forgot looking at the vmware part) HAPPENS to be a vmware Snapshot on the host. Now, this snapshot is very resource consuming in the initial take (a) and in the consolidate (b) phase. Remember this. Now add the storage IOPS power and voila, you have multiple reasons for a nice vss timeout. Solution: These are often very high performing vm´s (exchange, sql, and so on). Locate them on very fast and best dedicated LUNs and be 100% sure to monitor the IOPS during the backup phase. With Equallogic you could do that very nicely with SAN Headquraters for example. Other SAN Vendors have their solutions. It would also be a good idea to monitor the esxi hosts performance data (latency!!!!!), either with vcenter itself or with veeam monitor for example.

c) happens to be THE reason for most of the vss timeout errors as of my personal experience.

best regards
Joerg

PS good lord, just back from copenhagen and got a very very VERY bad flu ;-(

[jingxi02]

I think Veeam should see enough feed back from the customer and the forums and Veeam should really find out why so many people is complaining the same problem. It CAN NOT be the problem in Mictosoft operating system. I was backing up a file server and everything was working fine for the last entire year. I just installed Exchange 2010 with service pack 1 and update rollup pack 5 for Exchange 2010 SP1 on it last week. it was only a fresh Exchange install. There is no email box created on the server yet. The database was empty. Ever since I installed the Exchange 2010, Veeam is start failing with guest unfreeze error which is VSS timed-out. To prove it isn't the VSS problem with the OS, I ran a native OS built-in Windows Backup on the server. It backed up successfully without any error. This pretty much isolated the VSS problem in the OS because the built-in Windows Backup app uses VSS as well. If this is VSS problem, it doesn't matter which backup application you use, it will fail the same way. Support with Microsoft pretty come to the same conclusion, If the built-in Backup app works, Microsoft support ends there because they will not diagnose 3rd party backup app.

Here is i'm guessing where the problem is.

1. Veeam server starts the backup process on an Exchange server
2. Veeam server sends an API command to the vCenter or ESX host
3. vCenter sends an API command to VMtools inside the VM Guest
4. VM Guest receives a API command and deploys the ondemand Veeam agent
5. The Veeam agent starts up and calls the VSS component in the OS
6. The OS VSS service starts up and freezing the application database for snapshot
7. Veeam agent confirms all VSS writers are in ready state and starts to run the backup
* here is the problem - I think after the VSS takes the snapshot on step 6 and the applications are ready to be backed up, the Veeam agent for some reason doesn't get the acknoledgement from the VSS component and sits there wait until the VSS timed-out. My guess is the Veeam agent for some reason doesn't read the state of the Exchange VSS writer. Even the Exchange VSS writer is in ready state, the Veeam just thinks it's not ready. When VSS timed-out and unfreezes the applicaiton database, the Veeam agent sees the VSS is no longer in freeze state and reture guest unfreeze error to the Veeam server.

The other possibility is when the VSS starts and takes snapshot, it freezes the Veeam agant as well and cause the the Veeam agent not able to collect the VSS writers state.

I've ticket opened with Veeam support for this issue for a few months. The support keeps beleive the problem is in the OS and asked me to tweak around the VSS components such as register the VSS dlls, removing other backup applications. However, none of the recommandation resolves the problem. i'm here to point out - this proble happens on all 4 existing Exchange 2007 and Exchange 2010 servers. It also happens on all other 6 fresh deployed test Exchange 2010 servers. It can not be the OS problem in 10 different servers. Veeam really need to put your engineers to work with Microsoft engineer to find out why your application has this type of failure. It works perfectly with Exchange 2003. Only Exchange 2007 and Exchange 2010 keeps failing.

My work around solution is the following.
1. Add an additional vDisk to the Exchange VM
2. Backs up your Exchange server to the extra vDisk
3. Use Veeam to backup the Exchange VM include all vDisk and ignore the application processing failure

This in the only way that I can ensure my Exchange servers to have consistent database when it comes to restore.

I hope Veeam really realize this critical problem and resolve it in the next release.

[Gostev]

Actually, Joerg has posted a perfect summary of this issue above. It's not all about Veeam and Microsoft here.

We ourselves are backing up our very own Exchange 2010 SP1 (serving few hundreds of mailboxes) with Veeam Backup, and do not have this issue. Also, according to the support statistics, the total amount of customers reporting this issue is very small. These facts are hard to ignore. Obviously, if it was the issue in the Veeam code, it would affect 100% of customers - instead of failing in just a few selected deployments.

I would agree that reasons provided with Joerg above are most likely reasons behind the issue. No matter how many test servers you deploy, 10 or 100, if they are all in the same environment and using the same storage, they will surely exhibit similar behavior.

That said, our support will continue working with every customers who are having this issue and try to assist with resolution. Unfortunately, we may be limited at resolutions options available to us (for example, we cannot suggest that customer changes the storage they are using).

[th83]

My issue with this seemed to be related to the database maintenance routine on the Exchange server. By default, Exchange 2010 does 24/7 ESE scanning, but some administrators set a window during off-hours for this (our window happened to overlap with the beginning of the Exchange backup). Create a custom schedule for each database to keep database maintenance from happening during the backup window, this cuts down on IOPS during that time.

[GDRBrian]

Having the same problem. Support says call VMware. Vmware says call Veeam. Forums say call Veeam support and Internet says who knows.

Is there a way to extend the timeout of the VSS? Veeam support says it is 2 min but our NTDS VSS snapshot is taking about 2min 15 seconds according to the logs.

I tried this but it seems to be different than what I am looking for: http://www.veeam.com/kb_search_results. ... meout/All/

[Vitaliy S.]

Hi Brian,

Not sure what other timeout you're referring to, but this is the only timeout you can configure with Veeam. There are no 2 min default timeouts, so it looks like there was a misunderstanding between you and our support engineer that took care of your case.

[Seaniboy]

We are getting the same problem with the Exchange 2010 and jobs failing loggin the following in Events -

Information Store (instance id) Shadow copy instance 1 aborted

We can backup fine with Windows Server Backup so i assume this must be a Veeam problem?

[Vitaliy S.]

Sean, you need to work with our support team directly, as it is hard to identify your Exchange issue without looking into backup job log files. Once you do that, I would appreciate it if you could update this topic with the resolution. Thank you.

[rwhitmer]

I have had this same problem with a customers network now for about 7 months. I have been working with Veeam support for all that time and have tried all of the suggestions that they had. They keep trying to redirect me to Microsoft as well. I think more people have this problem than you think Gostev. This has been very embarrassing to us as we recommended this product to this customer based on the dog and pony show we got from Veeam. I do admit that the other 20 some machines we are backing up/replicating for them seem to be working very well... but the Exchange 2010 server which is critical to their business fails with these VSS errors 95% of the time. I am to the point that I will probably have to uninstall Veeam and go with a different product because Veeam simply cannot give us an answer. I became a Veeam partner early in the process of this install for this customer... but I have not been able to recommend the product to any of my other customers at this point because of this issue. Please accept that this is Veeams responsibility to work with Microsoft engineers to truly get to the bottom of this issue. I really like your product (that is why I became a partner) but this issue will force me to use other products very soon.

[elliott]

OK,

I struggled with this issue for the past few days. one in 15 backups would work, the rest the VSS writer would time out freezing the OS.

You guys steered me in the right direction with IOPS/server load and the time it takes for the freeze to take place.... For me, the situation was as follows:

SBS2011 VM on a 15k SAS drive, with a SATA RAID5 being used to store snapshots (which were only used for Veeam in this case).

The issue was someone had logged in, created a snapshot called "test" which I then had not even looked for, so the VM was running on this test snapshot.

Deleted snapshot, flawless veeam performance. I didn't try relocating the snapshots to be on the same datastore as the VM.. this may have fixed it also... but for those with the same problem, try getting rid of the snapshots and see how you go.

[sami.laihorinne]

Hi,

I am experiencing the VSS error on one exchange 2010 but not on another. They are both on the same vmware servers and use the same storage. The difference is that on of the servers is on another network than the veeam backup server.
The backup server is on the same lan as the vmware hosts.
Trying to figure out whats going on i tcpdumped the traffic between veeam backup server and the working exchange 2010 machine.

It would seem that the exchange 2010 server is sending traffic to the veeam backupserver with info about the VSS snapshot:
The exchange server is 10.1.10.59 and my veeam backup server is 10.1.8.87.

7818 234.910977 10.1.10.59 10.1.8.87 SMB2 298 Create Response File: VeeamVssSupport\metadata\VSS

So, if you are backing up a server with no connection to the veeam backupserver this traffic is lost. In those scenarios where VSS fails does your exchange 2010 server have network connectivity to the veeam backup server? Is the backup server tightly locked down with a firewall?
In my case the VSS failing exchange2010 in dmz has no connectivity to the veeam backup server. How is VSS information communicated? It seems odd that it would be sent directly from the guest to the veeam backup server.

Sami

[sami.laihorinne]

Odd indeed, the same server is running vCenter so it is more likely that the VSS response is sent to vCenter. But that is also odd, why does the guest inform vcenter abot VSS snaps directly over IP?

[habibalby]

Well I did uninstall the Symantec Backup Exec remote Agent from our Exchange 2010 server and restarted the server. Ran 2 incremental jobs and they were successful without any issues. I will keep everyone posted if there is any other issue related to this.

Thanks Gostev

Hi,
I have the same issue. [ID#5156540]. Veeam was running fine on our Exchange Server 2007 and accidentally stopped the job. So, SnapShot were removed but Exchange VSS gives error and unable to freeze the VM again. I do have Symantec Exec agent installed and backup works fine.

@itcaptain you have uninstalled Symantec Backup agent from the Exchange server and after that Veeam Backup with VSS starts working fine?

Thanks,

[Gostev]

Correct.

[habibalby]

Okay, let me try it and post back the result.

Thanks,

[habibalby]

Hello,

[ID#5156540] VM Backup Stopped accidently & Backu [...]

I have good news:) It’s just came to my mind out of curiosity to amend the original job selection that was selected from vCenter, instead I selected the machine directly from the Host and removed the old VM that was added from vCenter, The Job started working again

Thanks,

[habibalby]

After 10 hours, Veeam has finished backing up the Exchange Server successfully.

So, the solution for the VSS_WS_FAILD_AT_FREEZ to modify the job instead of selecting the VM via vCenter, select it from the host directly. If it's not added to Veeam console, add it and modify the job, then delete the old VM from the backup job, then re-run the job. SUCCESS

[Vitaliy S.]

Hussain,

Glad that you've managed to resolve your issue, but please be aware that your workaround will not work when you decide to vMotion your Exchange server to another host. This will cause VM ID to change and thus will affect your existing backup/replication job.

Anyway, since your backup job has started to work through a standalone host, I would imagine that your vCenter Server might be starving on CPU/Memory resources. I believe allocating more resources or placing vCenter Server on the datastore with less load might also solve your initial issue.

Thanks.

[habibalby]

Hi,
Yes, I'm aware of that if the VM get vMotioned the job will fail, I will stick the vCenter and Mail Server VMs onto the same host.

[*]vCenter and Mail Server VM were on same host. [/*]

[*]vCenter VM on different LUN from different Diskpool on different Enclosure. "This will take out the Storage latency doubt. iSCSI Storage with MPIO and Round-Robin multipathing".[/*]

[*]vCenter configured with 2 vCPUs and 4 vRAM performance report for last one month in the below link[/*]
http://www.4shared.com/file/nFTtriD1/vC ... files.html

Thanks,
Hussain

[habibalby]

Today is Unfreez Error VM still on the same host and no-one change the job configuration... Why Yesterday it works today it doesn't.... Come On Veeam, contact Microsoft and do something for your customers.

Thanks,

[habibalby]

Second attempt it works fine Strange

Thanks,

[elliott]

Issue started happening again today.... :/

Not impressed.

[Gostev]

I assume nothing at all has changed on the Veeam B&R side in the past days, and it worked well before. Yet, you still tend to blame Veeam for Exchange VSS writer timing out to do its job Anyhow, please take a look at this recent post, might be related > Exchange 2003 VSS issues resolved by Veeam6

[habibalby]

Hi,
before I attempt to run it second time, I forgot that today I have added new vRDM Disks to Exchange and created new DBs as well. I refresh the VM selection by pressing recalculate bottom. When it initialized the new disks and did the calculation, the backup started working..

Veeam should do something regarding the Exchange VSS Timeout issue.

Thanks,

[stormlord]

Same unfreeze error here.
Exchange 2010 sp1,
Vcenter is not on same datastore.
No Backupexec agent.

I go try the direct host test,
I will come back for result.