-
- Influencer
- Posts: 20
- Liked: never
- Joined: Oct 03, 2018 10:13 pm
- Contact:
Backing up simplivity cluster/disk latency
Is anyone else running simplivity and trying to backup the VM's via Veeam?
We deployed a small 3-node simplivity cluster with 25 VM's. When the veeam backup runs (NBD) the disk latency on the cluster skyrockets over 1000ms. Even when doing INC backups.
We deployed a small 3-node simplivity cluster with 25 VM's. When the veeam backup runs (NBD) the disk latency on the cluster skyrockets over 1000ms. Even when doing INC backups.
-
- Product Manager
- Posts: 14314
- Liked: 2890 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Backing up simplivity cluster/disk latency
Hello,
sounds your cluster is too slow... but it is possible to make the backup slower, too![Smile :-)](https://forums.veeam.com/images/smilies/icon_smile.gif)
storage latency control
Does that setting help?
Best regards,
Hannes
sounds your cluster is too slow... but it is possible to make the backup slower, too
![Smile :-)](https://forums.veeam.com/images/smilies/icon_smile.gif)
storage latency control
Does that setting help?
Best regards,
Hannes
-
- Influencer
- Posts: 20
- Liked: never
- Joined: Oct 03, 2018 10:13 pm
- Contact:
Re: Backing up simplivity cluster/disk latency
It shouldn't be. They are 3 nodes with all SSD. We have a case opened with HPE, but I figured other people must be running similar setup. Curious the note on those settings say not supported on vSAN volumes. This would be very similar to vSAN. I wonder why its not supported or what exactly is the support issue.matthewingram wrote: ↑Jul 30, 2019 4:21 am Is anyone else running simplivity and trying to backup the VM's via Veeam?
We deployed a small 3-node simplivity cluster with 25 VM's. When the veeam backup runs (NBD) the disk latency on the cluster skyrockets over 1000ms. Even when doing INC backups.
-
- Product Manager
- Posts: 14314
- Liked: 2890 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Backing up simplivity cluster/disk latency
vSAN is completely different technology compared with Simplivity. The only thing in common is that it is "hyper converged" ![Smile :-)](https://forums.veeam.com/images/smilies/icon_smile.gif)
![Smile :-)](https://forums.veeam.com/images/smilies/icon_smile.gif)
-
- VP, Product Management
- Posts: 6748
- Liked: 1408 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Backing up simplivity cluster/disk latency
Veeam in the end just performs VMware Snapshots and read data. You can easily simulate this by copying some data on your cluster. You should see same issue.
-
- Veeam Legend
- Posts: 235
- Liked: 134 times
- Joined: Mar 28, 2019 2:01 pm
- Full Name: SP
- Contact:
Re: Backing up simplivity cluster/disk latency
I had this issue during snapshot removal. direct SAN has solved it. I don't have Simplivity though so you may be in a different boat. Find out when the latency is up. Is it during snapshot creation, removal, or data copy.
-
- Influencer
- Posts: 20
- Liked: never
- Joined: Oct 03, 2018 10:13 pm
- Contact:
Re: Backing up simplivity cluster/disk latency
Is there a way to enable that on specific datastores? I see you can set thresholds per datastore, but it looks like its all on or all off?HannesK wrote: ↑Jul 30, 2019 6:01 am sounds your cluster is too slow... but it is possible to make the backup slower, too
storage latency control
Does that setting help?
-
- Expert
- Posts: 242
- Liked: 57 times
- Joined: Apr 28, 2009 8:33 am
- Location: Strasbourg, FRANCE
- Contact:
Re: Backing up simplivity cluster/disk latency
per datastore = Enterprise Plus License
The Set custom thresholds on individual datastores option is available in Veeam Backup & Replication Enterprise Plus Edition only.
The Set custom thresholds on individual datastores option is available in Veeam Backup & Replication Enterprise Plus Edition only.
-
- VP, Product Management
- Posts: 6748
- Liked: 1408 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Backing up simplivity cluster/disk latency
Latency control will not help inmost of the cases, as in these cases the hotadd cross host removal process cause it.
When you use HotAdd mode (check you job statistics, by selecting a VM there and then check disk backup entry for the mode (Network/HostAdd/DirectSAN/DirectNFS) and you VM runs on one host but the Proxy used for backup run on anothe VMware host, VMware will mount the VM disk snapshot state trough the other host to the Veeam Proxy. When this will be removed later after backup, you will see high latency in the disk backend. You can verify this by doing the same steps manually without Veeam.
So there are 2 other backup modes that I recommend to use:
1) use Network (NBD) mode, by changing the Proxy mode setting to "Network" (only). This will read all VMs through each host they belong to and you should not see these side effects.
2) Direct NFS mode. You need to give the Veeam Proxy access to the NFS backend. Please configure this only with a Simplivity Engineer that knows the system inside out to avoid configuration issues.
1) is likely a bit slower as all other backup methods, specifically if you compare single VM backup and restore. It scales pretty well with multiple backups in parallel accross the cluster. Direct NFS is the fastest backup method.
When you use HotAdd mode (check you job statistics, by selecting a VM there and then check disk backup entry for the mode (Network/HostAdd/DirectSAN/DirectNFS) and you VM runs on one host but the Proxy used for backup run on anothe VMware host, VMware will mount the VM disk snapshot state trough the other host to the Veeam Proxy. When this will be removed later after backup, you will see high latency in the disk backend. You can verify this by doing the same steps manually without Veeam.
So there are 2 other backup modes that I recommend to use:
1) use Network (NBD) mode, by changing the Proxy mode setting to "Network" (only). This will read all VMs through each host they belong to and you should not see these side effects.
2) Direct NFS mode. You need to give the Veeam Proxy access to the NFS backend. Please configure this only with a Simplivity Engineer that knows the system inside out to avoid configuration issues.
1) is likely a bit slower as all other backup methods, specifically if you compare single VM backup and restore. It scales pretty well with multiple backups in parallel accross the cluster. Direct NFS is the fastest backup method.
-
- Influencer
- Posts: 20
- Liked: never
- Joined: Oct 03, 2018 10:13 pm
- Contact:
Re: Backing up simplivity cluster/disk latency
We are using NBD mode and seeing the issue.
-
- VP, Product Management
- Posts: 6748
- Liked: 1408 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Backing up simplivity cluster/disk latency
That is strange as only VMware data transport is involved then. Just the read IO load will cause this then and you will see same things happening when you add load to the cluster. I would generate same load pattern (Create manual snapshot, read 5% of the data within the VMs and delete the snapshot). If you see same issues then contact HPE to check what is happening in the cluster. Overall Veeam ONE (30day trial) can help to monitor latency and you can see if other operations will cause latency as well. There you can check what happens when you create manual snapshots and remove them after some minutes.
Who is online
Users browsing this forum: No registered users and 70 guests