Host-based backup of VMware vSphere VMs.
Post Reply
matthewingram
Influencer
Posts: 20
Liked: never
Joined: Oct 03, 2018 10:13 pm
Contact:

Backing up simplivity cluster/disk latency

Post by matthewingram »

Is anyone else running simplivity and trying to backup the VM's via Veeam?

We deployed a small 3-node simplivity cluster with 25 VM's. When the veeam backup runs (NBD) the disk latency on the cluster skyrockets over 1000ms. Even when doing INC backups.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Backing up simplivity cluster/disk latency

Post by HannesK »

Hello,
sounds your cluster is too slow... but it is possible to make the backup slower, too :-)

storage latency control

Does that setting help?

Best regards,
Hannes
matthewingram
Influencer
Posts: 20
Liked: never
Joined: Oct 03, 2018 10:13 pm
Contact:

Re: Backing up simplivity cluster/disk latency

Post by matthewingram »

matthewingram wrote: Jul 30, 2019 4:21 am Is anyone else running simplivity and trying to backup the VM's via Veeam?

We deployed a small 3-node simplivity cluster with 25 VM's. When the veeam backup runs (NBD) the disk latency on the cluster skyrockets over 1000ms. Even when doing INC backups.
It shouldn't be. They are 3 nodes with all SSD. We have a case opened with HPE, but I figured other people must be running similar setup. Curious the note on those settings say not supported on vSAN volumes. This would be very similar to vSAN. I wonder why its not supported or what exactly is the support issue.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Backing up simplivity cluster/disk latency

Post by HannesK »

vSAN is completely different technology compared with Simplivity. The only thing in common is that it is "hyper converged" :-)
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backing up simplivity cluster/disk latency

Post by Andreas Neufert »

Veeam in the end just performs VMware Snapshots and read data. You can easily simulate this by copying some data on your cluster. You should see same issue.
vmtech123
Veeam Legend
Posts: 235
Liked: 133 times
Joined: Mar 28, 2019 2:01 pm
Full Name: SP
Contact:

Re: Backing up simplivity cluster/disk latency

Post by vmtech123 » 1 person likes this post

I had this issue during snapshot removal. direct SAN has solved it. I don't have Simplivity though so you may be in a different boat. Find out when the latency is up. Is it during snapshot creation, removal, or data copy.
matthewingram
Influencer
Posts: 20
Liked: never
Joined: Oct 03, 2018 10:13 pm
Contact:

Re: Backing up simplivity cluster/disk latency

Post by matthewingram »

HannesK wrote: Jul 30, 2019 6:01 am sounds your cluster is too slow... but it is possible to make the backup slower, too :-)

storage latency control

Does that setting help?
Is there a way to enable that on specific datastores? I see you can set thresholds per datastore, but it looks like its all on or all off?
NightBird
Expert
Posts: 242
Liked: 57 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Backing up simplivity cluster/disk latency

Post by NightBird »

per datastore = Enterprise Plus License

The Set custom thresholds on individual datastores option is available in Veeam Backup & Replication Enterprise Plus Edition only.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backing up simplivity cluster/disk latency

Post by Andreas Neufert »

Latency control will not help inmost of the cases, as in these cases the hotadd cross host removal process cause it.

When you use HotAdd mode (check you job statistics, by selecting a VM there and then check disk backup entry for the mode (Network/HostAdd/DirectSAN/DirectNFS) and you VM runs on one host but the Proxy used for backup run on anothe VMware host, VMware will mount the VM disk snapshot state trough the other host to the Veeam Proxy. When this will be removed later after backup, you will see high latency in the disk backend. You can verify this by doing the same steps manually without Veeam.

So there are 2 other backup modes that I recommend to use:
1) use Network (NBD) mode, by changing the Proxy mode setting to "Network" (only). This will read all VMs through each host they belong to and you should not see these side effects.
2) Direct NFS mode. You need to give the Veeam Proxy access to the NFS backend. Please configure this only with a Simplivity Engineer that knows the system inside out to avoid configuration issues.

1) is likely a bit slower as all other backup methods, specifically if you compare single VM backup and restore. It scales pretty well with multiple backups in parallel accross the cluster. Direct NFS is the fastest backup method.
matthewingram
Influencer
Posts: 20
Liked: never
Joined: Oct 03, 2018 10:13 pm
Contact:

Re: Backing up simplivity cluster/disk latency

Post by matthewingram »

We are using NBD mode and seeing the issue.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backing up simplivity cluster/disk latency

Post by Andreas Neufert »

That is strange as only VMware data transport is involved then. Just the read IO load will cause this then and you will see same things happening when you add load to the cluster. I would generate same load pattern (Create manual snapshot, read 5% of the data within the VMs and delete the snapshot). If you see same issues then contact HPE to check what is happening in the cluster. Overall Veeam ONE (30day trial) can help to monitor latency and you can see if other operations will cause latency as well. There you can check what happens when you create manual snapshots and remove them after some minutes.
Post Reply

Who is online

Users browsing this forum: Ivan239 and 88 guests