Host-based backup of VMware vSphere VMs.
Post Reply
cecker
Novice
Posts: 3
Liked: never
Joined: May 25, 2010 1:12 am
Full Name: chris ecker
Contact:

Delays During Job Processing

Post by cecker »

Recently we have started to experience 10-15 minute delays within our jobs at the point when each VM is processed. This 10-15 minute delay is at the point after it logs the entry 'Using source proxy <backup server proxy name> and before it logs the line preparing guest for hot backup. This delay occurs during each VM within a job. This even occurs when there is only 1 job running. Has anyone else seen this?
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Delays During Job Processing

Post by Vitaliy S. »

Hi Chris,

It's hard to say what causes this behavior without looking into the actual job debug log which should pinpoint the reason for this issue pretty quickly. Can you please contact our support team for assistance with logs review?

Btw, do you have any other proxy servers to try to process the VMs?

Thanks!
cby
Enthusiast
Posts: 97
Liked: 6 times
Joined: Feb 24, 2009 5:02 pm
Contact:

Re: Delays During Job Processing

Post by cby »

Hi

Did the original poster (cecker) get to the bottom of this issue? We have recently started to see exactly the same problem. Before we start re-configuring SAN fabrics and zones I wondered if there was a resolution. We're running B&R 6.5.0.144 patch 3 on a physical W2k8 box 64-bit. And no, we have no other backup proxy to test.

Thanks
tsightler
VP, Product Management
Posts: 6035
Liked: 2860 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Delays During Job Processing

Post by tsightler »

I believe that this a known issue and that you should contact support to verify and they can help you resolve.
cby
Enthusiast
Posts: 97
Liked: 6 times
Joined: Feb 24, 2009 5:02 pm
Contact:

Re: Delays During Job Processing

Post by cby »

Thanks, Tom, will do.
cby
Enthusiast
Posts: 97
Liked: 6 times
Joined: Feb 24, 2009 5:02 pm
Contact:

Re: Delays During Job Processing

Post by cby » 1 person likes this post

In case someone encounters the same problem...

After looking through one of the Job.backup_jobname.Backup logs I noticed that between the 'Using source proxy VMware Backup Proxy' and 'Preparing guest for hot backup' stages there were attempts to remove 64 non-existent unique snapshots. Each attempt took approximately 30secs, hence the 30min delay (64 * 30secs => 30+mins) for each VM backed up in every job. This snapshot deletion process occurs for every VM to be backed up. When you have hundreds of VMs the problem is extremely serious.

Veeam support came back almost immediately (kudos to them!) stating that it's a known issue where a SQL Server table references a whole bunch of old snapshots and the backup process trawls through this list trying to delete them all when backing up every VM.

The fix is to delete the snapshot entries from the Veeam database:

Code: Select all

DELETE FROM [VeeamBackup].[dbo].[VmWareSnapshots]
or as in our case:

Code: Select all

DELETE FROM [VeeamEM].[dbo].[VmWareSnapshots]
Maybe it'll help someone else.
veremin
Product Manager
Posts: 20415
Liked: 2302 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Delays During Job Processing

Post by veremin »

Yep, under certain circumstances VMwareSnapshots table might get filled with the references to the obsolete snapshots that are no longer present. This, in its turn, affects backup job performance. Cleaning the table should solve the issue, indeed.

Thank you for coming back and updating the topic with the found resolution; much appreciated.
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 25 guests