Why is "Preparing guest for hot backup" slow?

[MrSpock]

I have several reverse incremental backup jobs with VSS. These jobs spend about 7-15 minutes each in the stage of "Preparing guest for hot backup". That is about 50 % of the backup time.

I have noticed that VeeamGuestAgent.exe is scanning through all user profiles (C:\Users) on the guest, and that will take some time as there are about 2000 profiles on each server. Not many users are logged on during backup. Wouldn't it be more efficient to only scan profiles that are or has been in use during backup? I am curious about this activity. What are the reasons for these comprehensive scans?

Best regards,

Johan

[Veeam Backup & Replication 7.0.0.871 / ESXi 5.5.0 / Windows Server 2008 R2 SP1]

[Vitaliy S.]

Hi Johan,

When you enable AAIP (application-aware image processing) the entire Guest OS is prepared for hot backup via triggering VSS writers/providers for all files and applications in the system, so there is no way to control what profiles to choose. BTW, do you have Guest OS files indexing option enabled?

Thanks!

[MrSpock]

Hi, Vitaliy.

Thanks for your reply. No, I have not indexing enabled. Would that make the preparation faster?

Best regards,

Johan

[Vitaliy S.]

Not exactly, I was wondering if you have this option enabled or not, cause it might also take some time in the VM preparation process. Can you disable VSS for any test job and check how much time is spend for preparing any single VM? If the time period is noticeably reduced, then it seems like in-guest VSS process cannot complete any faster than you have now. Also if you would like to reduce the backup window, can you tell me what your bottlenecks stats are?

[MrSpock]

Hi, Vitaliy.

I have done three full backups of the same machine, with different settings. I was a bit surprised that the native "quiescence" is just as slow as AAIP, but I suppose they both are creating a VSS snapshot? I noticed that the VMWare Snapshot Provider did a similar scan on the guest as the VeeamGuestAgent.

AAIP:




VMWare Tools quiescence:




None:




Best regards,

Johan

[Vitaliy S.]

Yes, VMware Tools are also using VSS, but for the file system only, so it looks like your VM indeed required 9-10 minutes to prepare itself for hot backup. Also I see that source is reported as a bottleneck for your backup jobs. Can you please tell me what is performance of the source datastores when backup job is running? What are the read/write latency values?

[MrSpock]

The datastores reside on two local 600 GB SAS drives (15.000 rpm) in RAID-1. The graph shows read latency on the datastore according to the vSphere Client. AAIP preparations start at the blue line and the disk read starts at the red line. Write latency is 0 as no writes are made on the source during backup. The load is very low on the host and the datastore during this backup test.



The source is the bottleneck, as you say, but isn't that what to expect during a full backup?

I find it a bit odd with the latencies though. I assumed that the disk was read sequentially from the beginning to the end of the disk during a full backup. Shouldn't average latency be very close to zero then?

Best regards,

Johan

[cby]

Johan

Could this be the same rogue snapshot problem encounterd when there is a long delay between the 'Using source proxy...' and 'Preparing guest for hot backup...' stages?

If so, it's likely that the there are references in the Veeam table VmWare.Snapshots to non-existent snapshots.

This causes a 30 sec delay between every attempt in Veeam to access the non-existent snapshot files. It can't find the snapshot then moves onto the next one. And it happens for every job! So if you have 60 references it creates a 30 minute delay between the two stages 'Using source proxy...' and 'Preparing guest for hot backup...'. So if you have 40 backup jobs that adds 20 hours to the backup time!

Resolution:

Shutdown Veeam services

Fire up MS SQL Server Express/Management Studio 2005/2008 (or whatever version you're using) and execute the SQL:

Code: Select all

DELETE FROM VeeamEM.dbo.VmWareSnapshots

If it reports a number of rows deleted then there were references to rogue snapshots.

Hope that helps.

[MrSpock]

Hi, cby.

I don't think so. There is disk activity on the guest, which seems to be related to VSS. The VmWareSnapshots table do not exist in my Veeam database.

Thank you anyway!

Best regards,

Johan

[Dennis_VW]

Hi,

I recently installed a VM as VeeAm proxy because network mode failed and it solved by issue by running hotadd/hot backup for that VM.
Right now that veeam is runnning a full backup and the first 3 disks are backuped, and 2 disk are still being backuped.

Now for disk 6 and 7 it tells me: Preparing backup proxy NL-HQ-VEEAM01 for disk Hard disk 6 [hotadd]

It seems like it is stuck, because it is only running on 2 of the 4 available threads.
What can i do to have this backup continued?

The disks that are completed are still connected to the virtual VeeAm Proxy, is that a good sign or is something stuck??

[Dennis_VW]

Cancelled the job.
waited until snapshot was removed.
Restarted Proxy
Checked if disks where disconnected from proxy
Disabled automount (was enable, but on previous backup wasn't causing an issue)
updated to latest VMWare Tools (was 1 version behind) on VeeAm Proxy.

Started backup again and it seems to work, at least it started the first disk that hanged for couple of hours on the last run

If this fixed it, i'm just wondering why it wasnt able to start backuping the disks on the previous run?

[veremin]

"Preparing backup proxy" line refers to Application Aware Image Processing functionality, more specifically, to runtime process that is installed inside guest OS and does all of the necessary steps.

The reasons why it is stuck vary dramatically. Therefore, please, open a support ticket, if the issue re-occurs.

Thanks.