Backing up simplivity cluster/disk latency

[matthewingram]

Is anyone else running simplivity and trying to backup the VM's via Veeam?

We deployed a small 3-node simplivity cluster with 25 VM's. When the veeam backup runs (NBD) the disk latency on the cluster skyrockets over 1000ms. Even when doing INC backups.

[HannesK]

Hello,
sounds your cluster is too slow... but it is possible to make the backup slower, too

storage latency control

Does that setting help?

Best regards,
Hannes

[matthewingram]

Is anyone else running simplivity and trying to backup the VM's via Veeam?

We deployed a small 3-node simplivity cluster with 25 VM's. When the veeam backup runs (NBD) the disk latency on the cluster skyrockets over 1000ms. Even when doing INC backups.

It shouldn't be. They are 3 nodes with all SSD. We have a case opened with HPE, but I figured other people must be running similar setup. Curious the note on those settings say not supported on vSAN volumes. This would be very similar to vSAN. I wonder why its not supported or what exactly is the support issue.

[HannesK]

vSAN is completely different technology compared with Simplivity. The only thing in common is that it is "hyper converged"

[Andreas Neufert]

Veeam in the end just performs VMware Snapshots and read data. You can easily simulate this by copying some data on your cluster. You should see same issue.

[vmtech123]

I had this issue during snapshot removal. direct SAN has solved it. I don't have Simplivity though so you may be in a different boat. Find out when the latency is up. Is it during snapshot creation, removal, or data copy.

[matthewingram]

sounds your cluster is too slow... but it is possible to make the backup slower, too

storage latency control

Does that setting help?

Is there a way to enable that on specific datastores? I see you can set thresholds per datastore, but it looks like its all on or all off?

[NightBird]

per datastore = Enterprise Plus License

The Set custom thresholds on individual datastores option is available in Veeam Backup & Replication Enterprise Plus Edition only.

[Andreas Neufert]

Latency control will not help inmost of the cases, as in these cases the hotadd cross host removal process cause it.

When you use HotAdd mode (check you job statistics, by selecting a VM there and then check disk backup entry for the mode (Network/HostAdd/DirectSAN/DirectNFS) and you VM runs on one host but the Proxy used for backup run on anothe VMware host, VMware will mount the VM disk snapshot state trough the other host to the Veeam Proxy. When this will be removed later after backup, you will see high latency in the disk backend. You can verify this by doing the same steps manually without Veeam.

So there are 2 other backup modes that I recommend to use:
1) use Network (NBD) mode, by changing the Proxy mode setting to "Network" (only). This will read all VMs through each host they belong to and you should not see these side effects.
2) Direct NFS mode. You need to give the Veeam Proxy access to the NFS backend. Please configure this only with a Simplivity Engineer that knows the system inside out to avoid configuration issues.

1) is likely a bit slower as all other backup methods, specifically if you compare single VM backup and restore. It scales pretty well with multiple backups in parallel accross the cluster. Direct NFS is the fastest backup method.

[matthewingram]

We are using NBD mode and seeing the issue.

[Andreas Neufert]

That is strange as only VMware data transport is involved then. Just the read IO load will cause this then and you will see same things happening when you add load to the cluster. I would generate same load pattern (Create manual snapshot, read 5% of the data within the VMs and delete the snapshot). If you see same issues then contact HPE to check what is happening in the cluster. Overall Veeam ONE (30day trial) can help to monitor latency and you can see if other operations will cause latency as well. There you can check what happens when you create manual snapshots and remove them after some minutes.