Host-based backup of VMware vSphere VMs.
Post Reply
MrSpock
Service Provider
Posts: 46
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Why is "Preparing guest for hot backup" slow?

Post by MrSpock »

I have several reverse incremental backup jobs with VSS. These jobs spend about 7-15 minutes each in the stage of "Preparing guest for hot backup". That is about 50 % of the backup time.

I have noticed that VeeamGuestAgent.exe is scanning through all user profiles (C:\Users) on the guest, and that will take some time as there are about 2000 profiles on each server. Not many users are logged on during backup. Wouldn't it be more efficient to only scan profiles that are or has been in use during backup? I am curious about this activity. What are the reasons for these comprehensive scans?

Best regards,

Johan

[Veeam Backup & Replication 7.0.0.871 / ESXi 5.5.0 / Windows Server 2008 R2 SP1]
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Why is "Preparing guest for hot backup" slow?

Post by Vitaliy S. »

Hi Johan,

When you enable AAIP (application-aware image processing) the entire Guest OS is prepared for hot backup via triggering VSS writers/providers for all files and applications in the system, so there is no way to control what profiles to choose. BTW, do you have Guest OS files indexing option enabled?

Thanks!
MrSpock
Service Provider
Posts: 46
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Re: Why is "Preparing guest for hot backup" slow?

Post by MrSpock »

Hi, Vitaliy.

Thanks for your reply. No, I have not indexing enabled. Would that make the preparation faster?

Best regards,

Johan
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Why is "Preparing guest for hot backup" slow?

Post by Vitaliy S. »

Not exactly, I was wondering if you have this option enabled or not, cause it might also take some time in the VM preparation process. Can you disable VSS for any test job and check how much time is spend for preparing any single VM? If the time period is noticeably reduced, then it seems like in-guest VSS process cannot complete any faster than you have now. Also if you would like to reduce the backup window, can you tell me what your bottlenecks stats are?
MrSpock
Service Provider
Posts: 46
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Re: Why is "Preparing guest for hot backup" slow?

Post by MrSpock »

Hi, Vitaliy.

I have done three full backups of the same machine, with different settings. I was a bit surprised that the native "quiescence" is just as slow as AAIP, but I suppose they both are creating a VSS snapshot? I noticed that the VMWare Snapshot Provider did a similar scan on the guest as the VeeamGuestAgent.

AAIP:

Image


VMWare Tools quiescence:

Image


None:

Image


Best regards,

Johan
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Why is "Preparing guest for hot backup" slow?

Post by Vitaliy S. »

Yes, VMware Tools are also using VSS, but for the file system only, so it looks like your VM indeed required 9-10 minutes to prepare itself for hot backup. Also I see that source is reported as a bottleneck for your backup jobs. Can you please tell me what is performance of the source datastores when backup job is running? What are the read/write latency values?
MrSpock
Service Provider
Posts: 46
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Re: Why is "Preparing guest for hot backup" slow?

Post by MrSpock »

The datastores reside on two local 600 GB SAS drives (15.000 rpm) in RAID-1. The graph shows read latency on the datastore according to the vSphere Client. AAIP preparations start at the blue line and the disk read starts at the red line. Write latency is 0 as no writes are made on the source during backup. The load is very low on the host and the datastore during this backup test.

Image

The source is the bottleneck, as you say, but isn't that what to expect during a full backup?

I find it a bit odd with the latencies though. I assumed that the disk was read sequentially from the beginning to the end of the disk during a full backup. Shouldn't average latency be very close to zero then?

Best regards,

Johan
cby
Expert
Posts: 109
Liked: 6 times
Joined: Feb 24, 2009 5:02 pm
Contact:

Re: Why is "Preparing guest for hot backup" slow?

Post by cby »

Johan

Could this be the same rogue snapshot problem encounterd when there is a long delay between the 'Using source proxy...' and 'Preparing guest for hot backup...' stages?

If so, it's likely that the there are references in the Veeam table VmWare.Snapshots to non-existent snapshots.

This causes a 30 sec delay between every attempt in Veeam to access the non-existent snapshot files. It can't find the snapshot then moves onto the next one. And it happens for every job! So if you have 60 references it creates a 30 minute delay between the two stages 'Using source proxy...' and 'Preparing guest for hot backup...'. So if you have 40 backup jobs that adds 20 hours to the backup time!

Resolution:

Shutdown Veeam services

Fire up MS SQL Server Express/Management Studio 2005/2008 (or whatever version you're using) and execute the SQL:

Code: Select all

DELETE FROM VeeamEM.dbo.VmWareSnapshots
If it reports a number of rows deleted then there were references to rogue snapshots.

Hope that helps.
MrSpock
Service Provider
Posts: 46
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Re: Why is "Preparing guest for hot backup" slow?

Post by MrSpock »

Hi, cby.

I don't think so. There is disk activity on the guest, which seems to be related to VSS. The VmWareSnapshots table do not exist in my Veeam database.

Thank you anyway!

Best regards,

Johan
Dennis_VW
Influencer
Posts: 13
Liked: never
Joined: Feb 10, 2016 2:02 pm
Contact:

[MERGED] Preparing backup proxy for hot backup

Post by Dennis_VW »

Hi,

I recently installed a VM as VeeAm proxy because network mode failed and it solved by issue by running hotadd/hot backup for that VM.
Right now that veeam is runnning a full backup and the first 3 disks are backuped, and 2 disk are still being backuped.

Now for disk 6 and 7 it tells me: Preparing backup proxy NL-HQ-VEEAM01 for disk Hard disk 6 [hotadd]

It seems like it is stuck, because it is only running on 2 of the 4 available threads.
What can i do to have this backup continued?

The disks that are completed are still connected to the virtual VeeAm Proxy, is that a good sign or is something stuck??
Dennis_VW
Influencer
Posts: 13
Liked: never
Joined: Feb 10, 2016 2:02 pm
Contact:

Re: Preparing backup proxy for hot backup

Post by Dennis_VW »

Cancelled the job.
waited until snapshot was removed.
Restarted Proxy
Checked if disks where disconnected from proxy
Disabled automount (was enable, but on previous backup wasn't causing an issue)
updated to latest VMWare Tools (was 1 version behind) on VeeAm Proxy.

Started backup again and it seems to work, at least it started the first disk that hanged for couple of hours on the last run

If this fixed it, i'm just wondering why it wasnt able to start backuping the disks on the previous run?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Why is "Preparing guest for hot backup" slow?

Post by veremin »

"Preparing backup proxy" line refers to Application Aware Image Processing functionality, more specifically, to runtime process that is installed inside guest OS and does all of the necessary steps.

The reasons why it is stuck vary dramatically. Therefore, please, open a support ticket, if the issue re-occurs.

Thanks.
Post Reply

Who is online

Users browsing this forum: No registered users and 67 guests