vsan hotadd taking storage policy IOPs rate limit

[godfreyatlast]

During our lab testing a bottleneck was discovered due to vsan policy IOPs limit.

Example setup: Single VM backup with 2x disks, Proxy on same cluster w/ transport Auto (we want it to pick hotadd). In this backup config we saw the transfer speed for the VM run at 125MB/s and when one disk would complete the last disk would run at 62.5MB/s. Logs showed source at 99% (which was very helpful). I ended up correlating that to the Standard storage policy our clients run (we're a service provider, our stand tier is 2k IOPs). For vsan, this is 32k block size * 2000 IOPs, exactly 62.5MB/s. I change the storage policy to Performance, which is 4000 IOPs and the single disk backup speed increased to 125MB/s.

When hotadd adds the disk to the proxy the storage policy for the hotadd disk is listed “Datastore Default” and mimics its production storage policy. The “Datastore Default” policy also can't be changed while it is mounted to the Proxy.

From an infrastructure standpoint we’d prefer the backups run as quickly as possible. Especially for the sake of busy VMs when it come time to delete (merge) the vm snapshot, i.e. shorten the time a snapshot on vmware exist.

My question is, does Veeam have a way to elevate this limit we've discovered? Would veeam have some way of specifying a storage policy the moment of hotadd, or set the IOPs limit of the hotadd disk to unlimited?

I did test NDBSSL which seems to be unrestricted, its performance was slightly better than hotadd at our Standard storage policy, but much less performant than the Performance policy. NBDSSL hovered around 85MB/s, give or take 10MB/s. Ideally, hotadd unrestricted would be best.

[Gostev]

I doubt we can do this programmatically if the UI does not allow doing so. May be there's a way to specify it at hot add time though. @Andreas Neufert can we research this (or ask VMware)?

Have you considered just changing the "Datastore Default" policy to Performance? You can always specify the desired policy for newly provisioned VMs.

Another thing to check is the storage policy set on the backup proxy VM itself. I wonder if it propagates to all newly attached disks (or there's a way to make it do this)? Been a while since I touched VSAN...

[Andreas Neufert]

Will research with my Team.

[rennerstefan]

Another thing to check is the storage policy set on the backup proxy VM itself. I wonder if it propagate to all newly attached disks (or there's a way to make it do this)?

This is also what I would check first. @godfreyatlast can you check what policy i assigned to the proxy and also what policy is then assigned to the hot-add disks once the backup started.
You can check it like shown here: https://core.vmware.com/resource/vmware ... c6854-sub7

As Andreas wrote we will check as well.

[godfreyatlast]

Have you considered just changing the "Datastore Default" policy to Performance? You can always specify the desired policy for newly provisioned VMs.

Another thing to check is the storage policy set on the backup proxy VM itself. I wonder if it propagates to all newly attached disks (or there's a way to make it do this)?

Thanks for the reply!

1) If you look in the VM Storage Policies there is no "Datastore Default" to modify, there is " vSAN Default Storage Policy" but these seem to have no correlation to each other. If fact, in researching "Datastore Default" is a bit elusive from google in the context of backups. And the only place I've found reference in vcenter is - In vcenter > On a VM > Edit Settings > Hard Disk expanded > VM storage policy - In this Drop down, however if you select "Datastore Default" and save it will just revert to other items in the list (sometimes the the alphbetical top, sometimes the default vsan policy, possibly more that I haven't observed).

2) I've tested if the disks being hotadd'd to the proxy change speed when the Proxy OS disks change policy, they do not. The speed at which the VM disks backup is consistant with what the originating VMs VM Storage Policies was limited at.

[godfreyatlast]

This is also what I would check first. @godfreyatlast can you check what policy i assigned to the proxy and also what policy is then assigned to the hot-add disks once the backup started.
You can check it like shown here: https://core.vmware.com/resource/vmware ... c6854-sub7

As Andreas wrote we will check as well.

I've whitnessed this in two enviroments (company lab & a greenfield homelab), one where the policy of the proxy is in the higher performing tier, and one where the policy is the "vSAN Default Storage Policy" which is unlimited IOPs (all default settings) and not to be confused with "Datastore Default". Regardless, of what policy is configured on the proxy when the hotadd occurs the vm hostadd disk is set to "Datastore Default" and has consistantly matched the IOPs limit set by the original VMs disk policy. If that Policy is 4000 IOPs the throughput for the hotadd disk is 125MB/s, if i change it to 2000 then 64.5MB/s, I've even tested it down to 128 IOPS and speed followed suit down.

[Andreas Neufert]

Thanks for the follow up. We are analyzing this at the moment and will work with VMware on it considering a change at one of the next versions/updates.

[rennerstefan]

If fact, in researching "Datastore Default" is a bit elusive from google in the context of backups.

Hi again,
just to be sure... you are running it in a nested google environment or in your own VSAN?
Thanks

[rennerstefan]

Hi again,

we just reproduced what you do in our VSAN lab.
The "Datastore Default" has nothing to do with Google in our eyes.

In our lab the source VM had the policy "Source VM Test":




The Proxy had the policy "Proxy VM Policy" (1) and when the Hot-Add disk is attached (2), the disk also has the "Datastore Default" Policy set (like what you are seeing in your environment):




This "Datastore Default Policy is the one you set on the datastore level and is used if there is not other policy selected on the VMDK (disk) level:




So in your case you could set the data store default policy to one that is faster and make sure that all disk have a own one assigned.
I will also add the request to my list to check if there is a way to set the policy for the hot-add disk during a backup, but as it has been mentioned before in this thread, right now this is not possible.

Hope that helps.

Let us know.

[godfreyatlast]

Thanks @rennerstefan for the follow up.

By Google I meant "searching for information" not GCP, sorry for the confusion. All testing has been done on hardware based vsan enviroments.

I appriciate the clarification you found around Default Datastore in the context of vsan (I didn't know about that setting page for the vsan datastore), and also for confirming your test results against mine. Unfortunatly, in our case as a service provider every customer VM has a Storage Policy with IOPs limit specified on the vmdks, so any changes to the vSAN Default Storage Policy would make no difference.

I understand this not a function today, but would love to have this added in a future release. Do I need to take any steps to have this as a feature request?

[rennerstefan]

No, there is no need to take any steps.
All relevant people that track feature request are following here in the forum and I already added it to my list.

Thanks

[lharmer]

@rennerstefan was there any update on this feature request please?
As a service provider we have also just come across this when deploying Veeam to one of our vSAN environments.

Similar to @godfreyatlast, we would prefer backup (and restore) operations to not be restricted by the tenant IOPS limit during hotadd operations.

An additional side effect we believe may be happening is an impact on VM perfomance due to the IOPS limit being reached for the duration of the job. When realistically it's not actually tenant I/O causing the increase! Luckily we have backups only running overnight but it rules out being able to run more frequent jobs for customers.

Thanks.