Why is "Preparing guest for hot backup" slow?

VMware specific discussions

Why is "Preparing guest for hot backup" slow?

Veeam Logoby MrSpock » Fri Oct 31, 2014 9:53 pm

I have several reverse incremental backup jobs with VSS. These jobs spend about 7-15 minutes each in the stage of "Preparing guest for hot backup". That is about 50 % of the backup time.

I have noticed that VeeamGuestAgent.exe is scanning through all user profiles (C:\Users) on the guest, and that will take some time as there are about 2000 profiles on each server. Not many users are logged on during backup. Wouldn't it be more efficient to only scan profiles that are or has been in use during backup? I am curious about this activity. What are the reasons for these comprehensive scans?

Best regards,

Johan

[Veeam Backup & Replication 7.0.0.871 / ESXi 5.5.0 / Windows Server 2008 R2 SP1]
MrSpock
Enthusiast
 
Posts: 34
Liked: 1 time
Joined: Fri Apr 24, 2009 10:16 pm

Re: Why is "Preparing guest for hot backup" slow?

Veeam Logoby Vitaliy S. » Sun Nov 02, 2014 4:59 pm

Hi Johan,

When you enable AAIP (application-aware image processing) the entire Guest OS is prepared for hot backup via triggering VSS writers/providers for all files and applications in the system, so there is no way to control what profiles to choose. BTW, do you have Guest OS files indexing option enabled?

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Why is "Preparing guest for hot backup" slow?

Veeam Logoby MrSpock » Mon Nov 03, 2014 7:43 am

Hi, Vitaliy.

Thanks for your reply. No, I have not indexing enabled. Would that make the preparation faster?

Best regards,

Johan
MrSpock
Enthusiast
 
Posts: 34
Liked: 1 time
Joined: Fri Apr 24, 2009 10:16 pm

Re: Why is "Preparing guest for hot backup" slow?

Veeam Logoby Vitaliy S. » Mon Nov 03, 2014 10:09 am

Not exactly, I was wondering if you have this option enabled or not, cause it might also take some time in the VM preparation process. Can you disable VSS for any test job and check how much time is spend for preparing any single VM? If the time period is noticeably reduced, then it seems like in-guest VSS process cannot complete any faster than you have now. Also if you would like to reduce the backup window, can you tell me what your bottlenecks stats are?
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Why is "Preparing guest for hot backup" slow?

Veeam Logoby MrSpock » Mon Nov 03, 2014 9:33 pm

Hi, Vitaliy.

I have done three full backups of the same machine, with different settings. I was a bit surprised that the native "quiescence" is just as slow as AAIP, but I suppose they both are creating a VSS snapshot? I noticed that the VMWare Snapshot Provider did a similar scan on the guest as the VeeamGuestAgent.

AAIP:

Image


VMWare Tools quiescence:

Image


None:

Image


Best regards,

Johan
MrSpock
Enthusiast
 
Posts: 34
Liked: 1 time
Joined: Fri Apr 24, 2009 10:16 pm

Re: Why is "Preparing guest for hot backup" slow?

Veeam Logoby Vitaliy S. » Tue Nov 04, 2014 12:45 pm

Yes, VMware Tools are also using VSS, but for the file system only, so it looks like your VM indeed required 9-10 minutes to prepare itself for hot backup. Also I see that source is reported as a bottleneck for your backup jobs. Can you please tell me what is performance of the source datastores when backup job is running? What are the read/write latency values?
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Why is "Preparing guest for hot backup" slow?

Veeam Logoby MrSpock » Tue Nov 04, 2014 7:30 pm

The datastores reside on two local 600 GB SAS drives (15.000 rpm) in RAID-1. The graph shows read latency on the datastore according to the vSphere Client. AAIP preparations start at the blue line and the disk read starts at the red line. Write latency is 0 as no writes are made on the source during backup. The load is very low on the host and the datastore during this backup test.

Image

The source is the bottleneck, as you say, but isn't that what to expect during a full backup?

I find it a bit odd with the latencies though. I assumed that the disk was read sequentially from the beginning to the end of the disk during a full backup. Shouldn't average latency be very close to zero then?

Best regards,

Johan
MrSpock
Enthusiast
 
Posts: 34
Liked: 1 time
Joined: Fri Apr 24, 2009 10:16 pm

Re: Why is "Preparing guest for hot backup" slow?

Veeam Logoby cby » Thu Nov 06, 2014 12:24 pm

Johan

Could this be the same rogue snapshot problem encounterd when there is a long delay between the 'Using source proxy...' and 'Preparing guest for hot backup...' stages?

If so, it's likely that the there are references in the Veeam table VmWare.Snapshots to non-existent snapshots.

This causes a 30 sec delay between every attempt in Veeam to access the non-existent snapshot files. It can't find the snapshot then moves onto the next one. And it happens for every job! So if you have 60 references it creates a 30 minute delay between the two stages 'Using source proxy...' and 'Preparing guest for hot backup...'. So if you have 40 backup jobs that adds 20 hours to the backup time!

Resolution:

Shutdown Veeam services

Fire up MS SQL Server Express/Management Studio 2005/2008 (or whatever version you're using) and execute the SQL:

Code: Select all
DELETE FROM VeeamEM.dbo.VmWareSnapshots


If it reports a number of rows deleted then there were references to rogue snapshots.

Hope that helps.
cby
Expert
 
Posts: 106
Liked: 6 times
Joined: Tue Feb 24, 2009 5:02 pm

Re: Why is "Preparing guest for hot backup" slow?

Veeam Logoby MrSpock » Thu Nov 06, 2014 1:16 pm

Hi, cby.

I don't think so. There is disk activity on the guest, which seems to be related to VSS. The VmWareSnapshots table do not exist in my Veeam database.

Thank you anyway!

Best regards,

Johan
MrSpock
Enthusiast
 
Posts: 34
Liked: 1 time
Joined: Fri Apr 24, 2009 10:16 pm

[MERGED] Preparing backup proxy for hot backup

Veeam Logoby Dennis_VW » Fri Jul 29, 2016 6:16 pm

Hi,

I recently installed a VM as VeeAm proxy because network mode failed and it solved by issue by running hotadd/hot backup for that VM.
Right now that veeam is runnning a full backup and the first 3 disks are backuped, and 2 disk are still being backuped.

Now for disk 6 and 7 it tells me: Preparing backup proxy NL-HQ-VEEAM01 for disk Hard disk 6 [hotadd]

It seems like it is stuck, because it is only running on 2 of the 4 available threads.
What can i do to have this backup continued?

The disks that are completed are still connected to the virtual VeeAm Proxy, is that a good sign or is something stuck??
Dennis_VW
Influencer
 
Posts: 12
Liked: never
Joined: Wed Feb 10, 2016 2:02 pm

Re: Preparing backup proxy for hot backup

Veeam Logoby Dennis_VW » Fri Jul 29, 2016 7:37 pm

Cancelled the job.
waited until snapshot was removed.
Restarted Proxy
Checked if disks where disconnected from proxy
Disabled automount (was enable, but on previous backup wasn't causing an issue)
updated to latest VMWare Tools (was 1 version behind) on VeeAm Proxy.

Started backup again and it seems to work, at least it started the first disk that hanged for couple of hours on the last run

If this fixed it, i'm just wondering why it wasnt able to start backuping the disks on the previous run?
Dennis_VW
Influencer
 
Posts: 12
Liked: never
Joined: Wed Feb 10, 2016 2:02 pm

Re: Why is "Preparing guest for hot backup" slow?

Veeam Logoby v.Eremin » Mon Aug 01, 2016 10:13 am

"Preparing backup proxy" line refers to Application Aware Image Processing functionality, more specifically, to runtime process that is installed inside guest OS and does all of the necessary steps.

The reasons why it is stuck vary dramatically. Therefore, please, open a support ticket, if the issue re-occurs.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin


Return to VMware vSphere



Who is online

Users browsing this forum: Bing [Bot] and 11 guests