Entire VM restore speed for VMs with big disks can be increased?

[ibarizz]

Hello! Just coming here to bring out a doubt I'm currently having.

This is Veeam 12.
We’re experiencing slower than expected restore times for a specific VM.
The VM in question is a file server with nearly 10TB of data with only just 6 disks.
When performing an Entire VM Restore, we’re seeing recovery speeds of no more than 150 MB/s, resulting in a total restore time of almost 17 hours. We are using virtual proxies allowed to go through HotAdd transport mode and a network bandwith of 10Gbps.

We’ve noticed that other VMs, even larger ones that are not file servers, achieve much higher recovery speeds and consequently, shorter restore times (as an example, a VM of 12 TB placed in the same ESXi host, same proxies and transport mode, same hour and use in the network goes around 700 MB/s).

Given that we’re performing an Entire VM Restore, we’re not restoring individual files but rather the entire structure of the VM with its VMDKs. Therefore, it’s unclear why the large number of files on the file server would impact the restore speed.
The backup repository is the same one (it's an ExaGrid). And the configuration for each backup job is the same one.
I just want to make sure if there is any consideration for fileservers during Entire VM Restore.
My assumption is that the fileserver has HUGE disks, I mean, up to 3-4 TB disks and that's what delays the restore at most since only one proxy is assigned to those two disks and the speed doesn't go over 60 MB/s for each case. As for the other VM, it got more disks but not bigger than 1 TB, so basically more proxies can mount it's disks and proccess them in parallel. Is my assumption right?
Thank you!

[david.domask]

Hi ibarizz,

Thank you for the detailed explanation; a few comments and then some advice.

> Given that we’re performing an Entire VM Restore, we’re not restoring individual files but rather the entire structure of the VM with its VMDKs. Therefore, it’s unclear why the large number of files on the file server would impact the restore speed.

Can you elaborate on this statement a bit more? Are you suspecting that the contents of the virtual machine affect restore beyond just the total size of the VMDKs to be restored? The reason I ask is that your first statement is correct, the actual contents of the OS volumes doesn't matter during entire VM restore as there is no activity in the GuestOS on restore; all that happens is Veeam writes to the datastore via the VDDK library, and the datablocks for the VMDK are written.

I would suggest opening a Support case for this one; I do suspect there is some truth to what you're saying with regards to the proxies and disks, the workload likely is being spread out across multiple proxies, but I think there must be a bottleneck beyond just the proxies if it sits quite evenly at 60 MB/s per disk for your larger disks. Veeam Support can identify the true bottleneck from the logs and if necessary provide a test for the write speed during restore based on the transport mode. That is to say, for your VM with the larger disks, more proxies likely won't help as it's likely as you tell, fewer disks, fewer proxies/connections for writing disks in parallel, so you're likely bumping up against a bottleneck elsewhere, but it's best to let Support check the situation.

Please open a Support case and collect logs from the Veeam Backup and Replication server itself (use the 3rd radio option and select the Veeam server from the list); please mention the VMs you were testing with specifically as well as the approximate date/time of the test so the Support Engineer can jump right in with the research.

[ibarizz]

Well, I’m just returning to this topic…
The performance issue turned out to be a problem with the proxies due to a lack of resources, and somehow the traffic wasn’t directed to the 10Gbps network.
Nevertheless, it still surprises me that this Fileserver VM achieves a restore speed of 2GB/s (which is the expected speed), while the other larger VM reached a restore speed of 5GB/s (which is unusual). But hey, I’m satisfied with the final results!
Thank you for your kind assistance, David!