Discussions specific to the VMware vSphere hypervisor
Post Reply
matthewingram
Influencer
Posts: 18
Liked: never
Joined: Oct 03, 2018 10:13 pm
Contact:

Backing up simplivity cluster/disk latency

Post by matthewingram » Jul 30, 2019 4:21 am

Is anyone else running simplivity and trying to backup the VM's via Veeam?

We deployed a small 3-node simplivity cluster with 25 VM's. When the veeam backup runs (NBD) the disk latency on the cluster skyrockets over 1000ms. Even when doing INC backups.

HannesK
Veeam Software
Posts: 4042
Liked: 496 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Backing up simplivity cluster/disk latency

Post by HannesK » Jul 30, 2019 6:01 am

Hello,
sounds your cluster is too slow... but it is possible to make the backup slower, too :-)

storage latency control

Does that setting help?

Best regards,
Hannes

matthewingram
Influencer
Posts: 18
Liked: never
Joined: Oct 03, 2018 10:13 pm
Contact:

Re: Backing up simplivity cluster/disk latency

Post by matthewingram » Jul 30, 2019 6:05 am

matthewingram wrote:
Jul 30, 2019 4:21 am
Is anyone else running simplivity and trying to backup the VM's via Veeam?

We deployed a small 3-node simplivity cluster with 25 VM's. When the veeam backup runs (NBD) the disk latency on the cluster skyrockets over 1000ms. Even when doing INC backups.
It shouldn't be. They are 3 nodes with all SSD. We have a case opened with HPE, but I figured other people must be running similar setup. Curious the note on those settings say not supported on vSAN volumes. This would be very similar to vSAN. I wonder why its not supported or what exactly is the support issue.

HannesK
Veeam Software
Posts: 4042
Liked: 496 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Backing up simplivity cluster/disk latency

Post by HannesK » Jul 30, 2019 6:54 am

vSAN is completely different technology compared with Simplivity. The only thing in common is that it is "hyper converged" :-)

Andreas Neufert
Veeam Software
Posts: 3833
Liked: 687 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backing up simplivity cluster/disk latency

Post by Andreas Neufert » Jul 30, 2019 7:02 am

Veeam in the end just performs VMware Snapshots and read data. You can easily simulate this by copying some data on your cluster. You should see same issue.

vmtech123
Novice
Posts: 3
Liked: 5 times
Joined: Mar 28, 2019 2:01 pm
Full Name: Scott Patterson
Contact:

Re: Backing up simplivity cluster/disk latency

Post by vmtech123 » Jul 30, 2019 12:52 pm 1 person likes this post

I had this issue during snapshot removal. direct SAN has solved it. I don't have Simplivity though so you may be in a different boat. Find out when the latency is up. Is it during snapshot creation, removal, or data copy.

matthewingram
Influencer
Posts: 18
Liked: never
Joined: Oct 03, 2018 10:13 pm
Contact:

Re: Backing up simplivity cluster/disk latency

Post by matthewingram » Aug 03, 2019 3:00 am

HannesK wrote:
Jul 30, 2019 6:01 am
sounds your cluster is too slow... but it is possible to make the backup slower, too :-)

storage latency control

Does that setting help?
Is there a way to enable that on specific datastores? I see you can set thresholds per datastore, but it looks like its all on or all off?

NightBird
Service Provider
Posts: 178
Liked: 32 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Backing up simplivity cluster/disk latency

Post by NightBird » Aug 03, 2019 4:31 pm

per datastore = Enterprise Plus License

The Set custom thresholds on individual datastores option is available in Veeam Backup & Replication Enterprise Plus Edition only.

Andreas Neufert
Veeam Software
Posts: 3833
Liked: 687 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backing up simplivity cluster/disk latency

Post by Andreas Neufert » Aug 04, 2019 12:38 pm

Latency control will not help inmost of the cases, as in these cases the hotadd cross host removal process cause it.

When you use HotAdd mode (check you job statistics, by selecting a VM there and then check disk backup entry for the mode (Network/HostAdd/DirectSAN/DirectNFS) and you VM runs on one host but the Proxy used for backup run on anothe VMware host, VMware will mount the VM disk snapshot state trough the other host to the Veeam Proxy. When this will be removed later after backup, you will see high latency in the disk backend. You can verify this by doing the same steps manually without Veeam.

So there are 2 other backup modes that I recommend to use:
1) use Network (NBD) mode, by changing the Proxy mode setting to "Network" (only). This will read all VMs through each host they belong to and you should not see these side effects.
2) Direct NFS mode. You need to give the Veeam Proxy access to the NFS backend. Please configure this only with a Simplivity Engineer that knows the system inside out to avoid configuration issues.

1) is likely a bit slower as all other backup methods, specifically if you compare single VM backup and restore. It scales pretty well with multiple backups in parallel accross the cluster. Direct NFS is the fastest backup method.

matthewingram
Influencer
Posts: 18
Liked: never
Joined: Oct 03, 2018 10:13 pm
Contact:

Re: Backing up simplivity cluster/disk latency

Post by matthewingram » Sep 12, 2019 4:55 pm

We are using NBD mode and seeing the issue.

Andreas Neufert
Veeam Software
Posts: 3833
Liked: 687 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backing up simplivity cluster/disk latency

Post by Andreas Neufert » Sep 15, 2019 5:40 pm

That is strange as only VMware data transport is involved then. Just the read IO load will cause this then and you will see same things happening when you add load to the cluster. I would generate same load pattern (Create manual snapshot, read 5% of the data within the VMs and delete the snapshot). If you see same issues then contact HPE to check what is happening in the cluster. Overall Veeam ONE (30day trial) can help to monitor latency and you can see if other operations will cause latency as well. There you can check what happens when you create manual snapshots and remove them after some minutes.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], gpayne2019, restore-helper, wishr and 30 guests