Delays During Job Processing

[cecker]

Recently we have started to experience 10-15 minute delays within our jobs at the point when each VM is processed. This 10-15 minute delay is at the point after it logs the entry 'Using source proxy <backup server proxy name> and before it logs the line preparing guest for hot backup. This delay occurs during each VM within a job. This even occurs when there is only 1 job running. Has anyone else seen this?

[Vitaliy S.]

Hi Chris,

It's hard to say what causes this behavior without looking into the actual job debug log which should pinpoint the reason for this issue pretty quickly. Can you please contact our support team for assistance with logs review?

Btw, do you have any other proxy servers to try to process the VMs?

Thanks!

[cby]

Hi

Did the original poster (cecker) get to the bottom of this issue? We have recently started to see exactly the same problem. Before we start re-configuring SAN fabrics and zones I wondered if there was a resolution. We're running B&R 6.5.0.144 patch 3 on a physical W2k8 box 64-bit. And no, we have no other backup proxy to test.

Thanks

[tsightler]

I believe that this a known issue and that you should contact support to verify and they can help you resolve.

[cby]

Thanks, Tom, will do.

[cby]

In case someone encounters the same problem...

After looking through one of the Job.backup_jobname.Backup logs I noticed that between the 'Using source proxy VMware Backup Proxy' and 'Preparing guest for hot backup' stages there were attempts to remove 64 non-existent unique snapshots. Each attempt took approximately 30secs, hence the 30min delay (64 * 30secs => 30+mins) for each VM backed up in every job. This snapshot deletion process occurs for every VM to be backed up. When you have hundreds of VMs the problem is extremely serious.

Veeam support came back almost immediately (kudos to them!) stating that it's a known issue where a SQL Server table references a whole bunch of old snapshots and the backup process trawls through this list trying to delete them all when backing up every VM.

The fix is to delete the snapshot entries from the Veeam database:

Code: Select all

DELETE FROM [VeeamBackup].[dbo].[VmWareSnapshots]

or as in our case:

Code: Select all

DELETE FROM [VeeamEM].[dbo].[VmWareSnapshots]

Maybe it'll help someone else.

[veremin]

Yep, under certain circumstances VMwareSnapshots table might get filled with the references to the obsolete snapshots that are no longer present. This, in its turn, affects backup job performance. Cleaning the table should solve the issue, indeed.

Thank you for coming back and updating the topic with the found resolution; much appreciated.