VMDK Shards

[ksl28]

Hi,

Ive been reading up on how Veeam and Rubrik handles large VMDK files, since we have several large VMs in our environment, that resides on standalone ESXi hosts.
Today we have a virtual linux VM on the same host, that acts as a hot-add proxy, but it seems that it only processes 1 VM disk per CPU = The job takes really long, even that it processes 3-400MB/s.

Rubrik have a awesome feature, where it splits large VMDK files into shards, and utilizes multiple streams per disk.
https://www.rubrik.com/blog/products-so ... protection

Does Veeam have anything similar to this?

[Andreas Neufert]

You compare 2 different read capabilities.

Our Hotadd processing runs at a higher queue depth than VMware or other vendors that integrate VMware standards use. So basically we send many requests to the storage controlers and the storage controller can decide which data they read first to the point that the storage controller reaches the optimal read speed. Usually this processing is the fastest processing for the VMware environments. You can check as well if the "Source" is the bottleneck in the job that use the hotadd proxy which would be then a strong indication that your VMware/storage backend is the bottleneck.

Regarding reading the same way as Rubrik. Since v11 you can enable this mode as well within Veeam.
From the v11 what´s new:

NBD multi-threading — The backup engine is now capable of establishing multiple NBD connections per
VMDK for better performance of network transport mode. At the same time, due to the low limit of max
NBD connections per ESXi host, there are reliability concerns associated with increasing the number of such
connections. While our resource scheduler tracks NBD tasks per host to ensure they remain within the limit,
we decided that a marginal performance benefit is not worth the risk of enabling this new behavior for our
entire customer base right away, as there might be external NBD connections too. However, you can use
the fully supported VMwareNBDConnectionsPerDisk (DWORD) registry value on the backup server to give
this functionality a try. Our internal testing showed that the best performance is achieved with two NBD
connections per disk. Please share your results in the Veeam R&D Forums to help us decide whether
to enable this functionality by default in future updates

In the end this means if you have a huge VMDK as you shared, you can enable this feature with NBD processing. For the overall backup speed of hundreds of VMs, this feature would not make any bigger difference on the total backup window.

I think the HotAdd mode (if source is shown as bottleneck) is the fastest way to backup this VM. You can as well check on the VM itself what disk speed you will get there, it is likely nearly the same speed as the HotAdd proxy will get.

[ksl28]

Hi,

Thanks for the very detailed description, it makes perfect sense
Just out of curiosity, if we use Hot-Add for backup and restore, it seems that it uses 1 CPU core on the linux proxy per VM disk it needs to backup or restore.
Is there a way to assign multiple CPU cores per VM disk, so that it processes it faster?

[Andreas Neufert]

No, there is no way to influence this as the Veeam agent processes just grab needed CPU and RAM resources for the incoming data. It is a matter how Windows handle processes.

Is the backup read throughput (random read because of Change Block Tracking) comparable with your normal disk speed?