Discussions specific to the VMware vSphere hypervisor
Post Reply
cecker
Novice
Posts: 3
Liked: never
Joined: May 25, 2010 1:12 am
Full Name: chris ecker
Contact:

Delays During Job Processing

Post by cecker » Mar 06, 2013 9:59 pm

Recently we have started to experience 10-15 minute delays within our jobs at the point when each VM is processed. This 10-15 minute delay is at the point after it logs the entry 'Using source proxy <backup server proxy name> and before it logs the line preparing guest for hot backup. This delay occurs during each VM within a job. This even occurs when there is only 1 job running. Has anyone else seen this?

Vitaliy S.
Product Manager
Posts: 22289
Liked: 1413 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Delays During Job Processing

Post by Vitaliy S. » Mar 06, 2013 11:56 pm

Hi Chris,

It's hard to say what causes this behavior without looking into the actual job debug log which should pinpoint the reason for this issue pretty quickly. Can you please contact our support team for assistance with logs review?

Btw, do you have any other proxy servers to try to process the VMs?

Thanks!

cby
Expert
Posts: 109
Liked: 6 times
Joined: Feb 24, 2009 5:02 pm
Contact:

Re: Delays During Job Processing

Post by cby » Jan 03, 2014 5:49 pm

Hi

Did the original poster (cecker) get to the bottom of this issue? We have recently started to see exactly the same problem. Before we start re-configuring SAN fabrics and zones I wondered if there was a resolution. We're running B&R 6.5.0.144 patch 3 on a physical W2k8 box 64-bit. And no, we have no other backup proxy to test.

Thanks

tsightler
VP, Product Management
Posts: 5293
Liked: 2146 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Delays During Job Processing

Post by tsightler » Jan 03, 2014 6:09 pm

I believe that this a known issue and that you should contact support to verify and they can help you resolve.

cby
Expert
Posts: 109
Liked: 6 times
Joined: Feb 24, 2009 5:02 pm
Contact:

Re: Delays During Job Processing

Post by cby » Jan 03, 2014 6:11 pm

Thanks, Tom, will do.

cby
Expert
Posts: 109
Liked: 6 times
Joined: Feb 24, 2009 5:02 pm
Contact:

Re: Delays During Job Processing

Post by cby » Jan 07, 2014 12:55 pm 1 person likes this post

In case someone encounters the same problem...

After looking through one of the Job.backup_jobname.Backup logs I noticed that between the 'Using source proxy VMware Backup Proxy' and 'Preparing guest for hot backup' stages there were attempts to remove 64 non-existent unique snapshots. Each attempt took approximately 30secs, hence the 30min delay (64 * 30secs => 30+mins) for each VM backed up in every job. This snapshot deletion process occurs for every VM to be backed up. When you have hundreds of VMs the problem is extremely serious.

Veeam support came back almost immediately (kudos to them!) stating that it's a known issue where a SQL Server table references a whole bunch of old snapshots and the backup process trawls through this list trying to delete them all when backing up every VM.

The fix is to delete the snapshot entries from the Veeam database:

Code: Select all

DELETE FROM [VeeamBackup].[dbo].[VmWareSnapshots]
or as in our case:

Code: Select all

DELETE FROM [VeeamEM].[dbo].[VmWareSnapshots]
Maybe it'll help someone else.

v.eremin
Product Manager
Posts: 16134
Liked: 1315 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Delays During Job Processing

Post by v.eremin » Jan 07, 2014 1:03 pm

Yep, under certain circumstances VMwareSnapshots table might get filled with the references to the obsolete snapshots that are no longer present. This, in its turn, affects backup job performance. Cleaning the table should solve the issue, indeed.

Thank you for coming back and updating the topic with the found resolution; much appreciated.

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests