-
- Novice
- Posts: 3
- Liked: never
- Joined: May 25, 2010 1:12 am
- Full Name: chris ecker
- Contact:
Delays During Job Processing
Recently we have started to experience 10-15 minute delays within our jobs at the point when each VM is processed. This 10-15 minute delay is at the point after it logs the entry 'Using source proxy <backup server proxy name> and before it logs the line preparing guest for hot backup. This delay occurs during each VM within a job. This even occurs when there is only 1 job running. Has anyone else seen this?
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Delays During Job Processing
Hi Chris,
It's hard to say what causes this behavior without looking into the actual job debug log which should pinpoint the reason for this issue pretty quickly. Can you please contact our support team for assistance with logs review?
Btw, do you have any other proxy servers to try to process the VMs?
Thanks!
It's hard to say what causes this behavior without looking into the actual job debug log which should pinpoint the reason for this issue pretty quickly. Can you please contact our support team for assistance with logs review?
Btw, do you have any other proxy servers to try to process the VMs?
Thanks!
-
- Enthusiast
- Posts: 97
- Liked: 6 times
- Joined: Feb 24, 2009 5:02 pm
- Contact:
Re: Delays During Job Processing
Hi
Did the original poster (cecker) get to the bottom of this issue? We have recently started to see exactly the same problem. Before we start re-configuring SAN fabrics and zones I wondered if there was a resolution. We're running B&R 6.5.0.144 patch 3 on a physical W2k8 box 64-bit. And no, we have no other backup proxy to test.
Thanks
Did the original poster (cecker) get to the bottom of this issue? We have recently started to see exactly the same problem. Before we start re-configuring SAN fabrics and zones I wondered if there was a resolution. We're running B&R 6.5.0.144 patch 3 on a physical W2k8 box 64-bit. And no, we have no other backup proxy to test.
Thanks
-
- VP, Product Management
- Posts: 6035
- Liked: 2860 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Delays During Job Processing
I believe that this a known issue and that you should contact support to verify and they can help you resolve.
-
- Enthusiast
- Posts: 97
- Liked: 6 times
- Joined: Feb 24, 2009 5:02 pm
- Contact:
Re: Delays During Job Processing
Thanks, Tom, will do.
-
- Enthusiast
- Posts: 97
- Liked: 6 times
- Joined: Feb 24, 2009 5:02 pm
- Contact:
Re: Delays During Job Processing
In case someone encounters the same problem...
After looking through one of the Job.backup_jobname.Backup logs I noticed that between the 'Using source proxy VMware Backup Proxy' and 'Preparing guest for hot backup' stages there were attempts to remove 64 non-existent unique snapshots. Each attempt took approximately 30secs, hence the 30min delay (64 * 30secs => 30+mins) for each VM backed up in every job. This snapshot deletion process occurs for every VM to be backed up. When you have hundreds of VMs the problem is extremely serious.
Veeam support came back almost immediately (kudos to them!) stating that it's a known issue where a SQL Server table references a whole bunch of old snapshots and the backup process trawls through this list trying to delete them all when backing up every VM.
The fix is to delete the snapshot entries from the Veeam database:
or as in our case:
Maybe it'll help someone else.
After looking through one of the Job.backup_jobname.Backup logs I noticed that between the 'Using source proxy VMware Backup Proxy' and 'Preparing guest for hot backup' stages there were attempts to remove 64 non-existent unique snapshots. Each attempt took approximately 30secs, hence the 30min delay (64 * 30secs => 30+mins) for each VM backed up in every job. This snapshot deletion process occurs for every VM to be backed up. When you have hundreds of VMs the problem is extremely serious.
Veeam support came back almost immediately (kudos to them!) stating that it's a known issue where a SQL Server table references a whole bunch of old snapshots and the backup process trawls through this list trying to delete them all when backing up every VM.
The fix is to delete the snapshot entries from the Veeam database:
Code: Select all
DELETE FROM [VeeamBackup].[dbo].[VmWareSnapshots]
Code: Select all
DELETE FROM [VeeamEM].[dbo].[VmWareSnapshots]
-
- Product Manager
- Posts: 20415
- Liked: 2302 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Delays During Job Processing
Yep, under certain circumstances VMwareSnapshots table might get filled with the references to the obsolete snapshots that are no longer present. This, in its turn, affects backup job performance. Cleaning the table should solve the issue, indeed.
Thank you for coming back and updating the topic with the found resolution; much appreciated.
Thank you for coming back and updating the topic with the found resolution; much appreciated.
Who is online
Users browsing this forum: Bing [Bot] and 25 guests