One day Success, other day Failed

[Mike]

We experience that the backups often fail, but next day they backup with success.

It doesnt matter if we use VSS, or VMware tools.

With VMware tools enabled (VSS disabled) we often get the error:
Retrieving file list VCB error: Error: Other error encountered
: Snapshot creation failed: Operation timed out. An error occurred, cleaning up...


With VSS (for server with a running DB) enabled and VMware tools disabled:
Completing current object backup process VCB
error: No active paths found. Error: Failed to export the disk: The requested resource is in use An error occurred, cleaning up...

or sometimes:
Retrieving file list Failed
to perform Veeam VSS freeze: VMware tools status is "toolsNotRunning"

VM's arent stuck in snapshot modus, and vmware tools are running on all servers.
When a Job runs with 8 VM's in it, sometimes they are all succesfull, sometimes only one is succesfull (same job).
Cant find anything strange in the logs.... but thats also kinda hard to find if you dont know what youre looking for

[Gostev]

Hello, from the errors you are getting and the fact that they are intermittent it sounds like there are storage or infrastructure problems that affects VCB ability to work correctly. Generally, snapshot timeout and active path errors are pretty common problem in VI3 and they typically reflect either storage issues or high VC load (so unrelated to Veeam Backup - lower level). Our support may have more points for you on what things to check in your environment. But these errors definitely don't relate to VSS.

The last error and workaround are described in the existing thread: VMWare Tools not running?
This issue is also unrelated to Veeam Backup.

Hope this helps!

[donikatz]

Is there any way to work around VCB snapshot timeout issues? Any way to extend the timeout past 15 minutes? I have one VM that always has very large deltas and often takes longer than 15 minutes to remove the snapshot, causing the backup to fail. Thanks

[Gostev]

Doni, I will check that for you tomorrow morning.

[donikatz]

Thanks, I've also opened a case with VMware support

[Gostev]

I am not sure if this is timeout in VCB code, or in Veeam Backup code - this is what I need to check with devs on. Do you have the same issue when creating/removing snapshots using VMware Infrastructure Client?

[donikatz]

Seems to work fine from VIC. Did a couple snapshot/removals of an hour+ delta without timeout. But I'll have to keep trying it, since it's not consistent. I'm pretty sure the 15-min timeout is a VI3 thing, but it could be that something Veeam does is more-likely to cause it. So I'll be interested to see what both Veeam and VMware have to say about it. VMTN forums have some discussions about snapshot timeouts, but no obvious systematic workaround. Thanks! .dk

[donikatz]

VMware support explained how to change the default 15 min VirtualCenter timeout by editing the vpxd.cfg file. Notice they recommend changing it to 20 minutes for VCB snapshots, which seems to indicate it's not unusual for VCB snapshots to take longer to remove than manual VIC ones. I'm testing out a 25 min timeout and will post back with results. (I believe the max setting is 30 min.)

From VMware support:

VMware VirtualCenter 2.x has a hard coded 15 minutes timeout for any task.

The default 15 minutes timeout value can be changed in VMware VirtualCenter. If using VCB, the backup failed due to the time out, then check your vm and remove the Back up snapshot which has been left behind on them. The default value can be changed manually by modifying the vpxd.cfg file and adding the following under <vpxd> section :
<task>
<timeout>value</timeout>
</task>

The file is normally located in C:/Documents and Settings/All Users/Application Data/VMware/VMware VirtualCenter/vpxd.cfg The value is specified in seconds, default is 900 seconds Recommended value for VCB backup snapshots is 1200 seconds. VirtualCenter must be restarted after making the changes to the vpxd.cfg file.

[Gostev]

Thank you Doni - this is a very useful information.

[donikatz]

After some tests, so far so good. Performing manual and scheduled Veeam/VCB backups, no timeouts yet. The last job took 18 minutes to remove the snapshot, meaning it would have failed before the fix. I'll wait to see how it goes with tonight's regular schedule and monitor it over the next few days before making any conclusions, but if no more failures found, I'd suggest adding this to the FAQ. Thanks .dk

[donikatz]

A week of daily backups and not a single failure. Looks like problem solved!

[Gostev]

Thanks for update