Backup Browser slow for FLR from a VM disk with many files

[StanO]

Nice product. It's been a pleasure evaluating it thus far and I look forward to watching it grow.

I've been particularly pleased with the reliability of the backups. My restore testing is not yet complete, but it looks real good so far.

One area that does cause some concern is the observed slowness of the Backup Browser when performing a file level restore from a VM with over half a million files. After selecting the appropriate VM in the Backups context and selecting the desired restore point (a synthetic full, no rollback testing yet) and clicking Finish in the Restore Wizard window, it takes on the order of 10 minutes for the Backup Browser to open. When navigating to a sub-folder that contains the bulk of the files on the volume a similar delay is observed. I suspected backup compression settings as a contributing factor but dialing down to "Low" compression (with a completely new job) produced no noticeable change in behavior. Please note that this is strictly using the integrated restore functionality. Attempts to use the restore appliance resulted in a timeout message (100000 msec).

I noticed while waiting for the browser window and confirmed elsewhere in these forums that the VM's disks are mounted and assigned drive letters on the local host during restore operations and surmise that I could quickly perform a restore from these mounted volumes (they are available within 90 seconds of finishing the Restore Wizard dialog) and just close the Backup Browser later when it opens, but I'm wary of when these mounted volumes are actually in a ready state (can I use them as soon as I see them?).

I also noticed that there may be additional cataloging features forthcoming based on conversations in the wishlist thread. My current theory is that it's the cataloging being performed by the Backup Browser that causes this slowness, and so am hopeful some sort of background cataloging will remedy this situation. If it is indeed the case that the Backup Browser collecting file tree information causes this slowness, perhaps a wishlist item to have the option to not open the Backup Browser and rely solely on the mounted volumes for restores would provide an interim fix (if more robust cataloging is not imminent).

Great job on the product and the support from these boards.

Thanks,
Stan

[Gostev]

Hi Stan, thank you for your feedback. Where was your backup stored during your FLR testing, was it local on Veeam Backup server, or remotely? To me, it sounds like the issue may be due to backup storage performance here, or the fact that it is remote storage that you are accessing over the network.

I will need to check with the devs what triggers Backup Browser appearing, and whether you could use the volumes right after they appear under My Computer. Will do this next week after I return from vacation.

Thank you.

[StanO]

The backup storage is a locally attached SCSI RAID array with 14 spindles. I haven't ruled out the possibility of fragmentation being at least a contributing factor, but I think the likelihood that it is the main cause is low (other, smaller VMs within the same backup set do not exhibit the same behavior). I'll try to do a backup of the VM in question to a pristine volume on our SAN array (40 spindles) to confirm the behavior persists with fragmentation minimized.

I've observed read I/O rates from the backup storage on the order of 30-40 MB/second initially after the Restore Wizard dialog is completed. The read I/O rate drops to about 4-5 MB/second shortly before the mounted volumes appear under My Computer and stay at that level until the Backup Browser window appears. When delays are encountered while navigating within the Backup Browser window a 4-5 MB/second read rate is also observed. If I restore some directories with a significant number of files (about 9600 files comprising 4.5 GB) from one of the mounted volumes I observe read I/O rates on the order of single-digit MB/second, with occasional spikes into the teens.

Please enjoy your vacation. Based on the responsiveness you've demonstrated in this forum you deserve one and we can wait until your return.

Thanks,

Stan

[StanO]

Anton:

I hope you enjoyed at least part of your vacation.

Any word from the Devs on if the mounted volumes are viable for restore as soon as they appear within My Computer?

Thanks!

Stan

[Gostev]

Hi Stan, yes I talked to them yesterday, but forgot to update the thread, besides I had additional questions so I am not done with them yet on this issue

Anyhow, yes, the mounted volumes are viable for restore as soon as they appear within My Computer.

Delays in Backup Browser are also expected when you have very large amount of files (Backup Browser has to scan directory structure before displaying it, and remember that it deals with deduped backup file, not just plain VMDKs - backup content access engine has pretty significant overhead); however 10 minutes is not normal (and I am assuming in reality it is a bit less than that for you).

Thanks!

[StanO]

Anton:

It actually was slightly over 10 minutes (I timed several runs with a stopwatch provided by Riverbed - third mention of WAN acceleration in a week).

We're good with doing the restores from the mounted volumes, but if you'd like more specifics about the environment I've observed this in or want me to run more tests, I'd be happy to help out.

I did do a timing using a pristine LUN on a FC array with 40 spindles rather than the SCSI array using 14 spindles and observed the same operation completed in a little under 4 minutes (observed data rates of 10-12MB/sec rather than 4-5 MB/sec observed on the SCSI). I may try the same test with a volume formatted with a different allocation unit size (currently 64K) to see if that produces different results.

Thanks,

Stan

[StanO]

Just a quick follow up on this.

I did some additional testing with different block sizes and observed no change in the amount of time it took the Veeam Backup File Browser window to open (still took on the order of 10 minutes).

Through other work, we determined that the server we were using to perform the backups (and restores) was a bit underpowered for our needs (a Proliant DL380 G4 - Dual Xeon 3.0GHz single core with Hyperthreading). We reconfigured to have a newer Proliant BL460 (Single Xeon 2.833GHz quad core) run the backup jobs and not only did we see a significant improvement in backup rates (2-3x), but the time to open the Veeam Backup File Browser window dropped to less than a minute.

The takeaway for me was to think twice about re-purposing older servers to be the primary data movers.

Stan

[Gostev]

Stan, thanks for the update.