Discussions specific to the VMware vSphere hypervisor
Post Reply
dcampregher
Service Provider
Posts: 152
Liked: 6 times
Joined: Mar 23, 2016 5:57 pm
Full Name: Diogo Campregher
Contact:

Verry slow restore do HPE StoreVirtual VSA

Post by dcampregher » Apr 04, 2019 12:01 pm

We are in face to a problem that is very weird. The Customer in case, use the HPE StoreVirtual VSA ( 02X HPE Server with 13 SSD disks, 2GB SAS Controller in RAID5, vSphere 6.5U2, volumes in NetworkRAID10, all the network is 10Gbps or more ).

The Veeam Server ( all in one ) is a VM in a dedicated local datastore with 20 disks SAS and a 2GB controller in RAID5.

All the backups are very fast. The IVMR, FLR, Restore VM Files and so on is very fast. But, when i try to restore the entire VM to the volumes in the VSA ( source volumes) the restore goes down to 1 or 2 Mbps and stay there.
If i try to restore the same VM ( same parameters of restore ) but to the local ESXi SSD Datastore ( out of the VSA but in the same ESXi ) the restore go up to 300MBps.

No matter what transport mode or any parameter in the Veeam that we change, if the restore is sent to the VSA Volumes the restore rate is very slow

I have a Tier-2 case to search a root cause. Any idea of how this could be happening ?

HannesK
Veeam Software
Posts: 4479
Liked: 563 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Verry slow restore do HPE StoreVirtual VSA

Post by HannesK » Apr 05, 2019 6:19 am

Hello,
could you support the case number? And just to clarify: you say that restore of VM files (large disks) is fast?

Best regards,
Hannes

Andreas Neufert
Veeam Software
Posts: 4037
Liked: 734 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Verry slow restore do HPE StoreVirtual VSA

Post by Andreas Neufert » Apr 05, 2019 6:57 am

If I understood correctly, you restore same VM twice on the same ESXi host, one restore is fast and one restore that goes to StoreVirtual is slow. Can you please check with a disk test tool (1MB block size) the write speed? As well you can try to clone a VM from the fast storage to the StoreVirtual to see if you achieve same problematic speed or if it is faster.

As well, please check at the restore statistics what mode is used for restore (NBD-Network/HotAdd/DirectSAN) and if it is different when you restore it to the other faster storage.

HannesK
Veeam Software
Posts: 4479
Liked: 563 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Verry slow restore do HPE StoreVirtual VSA

Post by HannesK » Apr 05, 2019 7:07 am

my understanding is the same. Support can help with the test tool (VixDiskLib), they use it quite often.

dcampregher
Service Provider
Posts: 152
Liked: 6 times
Joined: Mar 23, 2016 5:57 pm
Full Name: Diogo Campregher
Contact:

Re: Verry slow restore do HPE StoreVirtual VSA

Post by dcampregher » Apr 05, 2019 12:59 pm

Hi!
The case number is 03474350.

The restore to the local datastore run this:
05/04/2019 09:28:02 Restoring Hard disk 2 (150,0 GB) : 103,0 GB restored at 300 MB/s [hotadd]
05/04/2019 09:26:51 Restoring Hard disk 1 (50,0 GB) : 44,5 GB restored at 185 MB/s [hotadd]

The restore to the VSA datastore run this:
05/04/2019 09:57:55 Restoring Hard disk 1 (50,0 GB) : 251,0 MB restored at 802 KB/s [hotadd]
05/04/2019 09:57:55 Restoring Hard disk 2 (150,0 GB) : 175,0 MB restored at 721 KB/s [hotadd]

The VM clone from the local datastore to the VSA datastore get 12 minutes.

All the operations ( restores and clone ) from the same ESXi host and same parameters for the two restores.

Andreas Neufert
Veeam Software
Posts: 4037
Liked: 734 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Verry slow restore do HPE StoreVirtual VSA

Post by Andreas Neufert » Apr 05, 2019 1:16 pm

Looks like the VSA is just slower. At HotAdd we mount the restore VM disk to the Veeam Proxy running on the host and then we write directly to this disk the data. As you had good speed at the other storage, the Veeam infrastructure is not the bottleneck.

Do you have opened a ticket at HPE to look into this performance issue?

Andreas Neufert
Veeam Software
Posts: 4037
Liked: 734 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Verry slow restore do HPE StoreVirtual VSA

Post by Andreas Neufert » Apr 05, 2019 1:19 pm

Checked the support case. There is a statement there that the issue was recreated with Windows File copy within the VM, so you have a performance issue on the storage system.

dcampregher
Service Provider
Posts: 152
Liked: 6 times
Joined: Mar 23, 2016 5:57 pm
Full Name: Diogo Campregher
Contact:

Re: Verry slow restore do HPE StoreVirtual VSA

Post by dcampregher » Apr 05, 2019 1:21 pm

There is a detail.

The problem is not fixed in one VSA volume. It can happen now in the volume01 and tomorrow not, migrating the problem to the volume 03 and so on....

And this difference from 300 to 1 MBps ? It seems not a performance issue but the blocked performance by anything .....

03474350

Andreas Neufert
Veeam Software
Posts: 4037
Liked: 734 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Verry slow restore do HPE StoreVirtual VSA

Post by Andreas Neufert » Apr 05, 2019 1:26 pm

Yes, but there is nothing we could do differently with the volume. During the support case the same issue happend independantly reproducable with Windows Fiele copy to a StoreVirtual volume. So it is important that HPE will have a look to identify what will cause this issue within the storage.

dcampregher
Service Provider
Posts: 152
Liked: 6 times
Joined: Mar 23, 2016 5:57 pm
Full Name: Diogo Campregher
Contact:

Re: Verry slow restore do HPE StoreVirtual VSA

Post by dcampregher » Apr 05, 2019 1:45 pm

The "exact" issue not. During the file transfer with Windows Explorer we see the transfer ratio with high performance but with variable and in some time it stop by a fews seconds, i understand you when you tell that there is a problem with storage system and because this there is a another engineer ( our engineer ) to do an assessment in the environment.

In fact, the restore when sent to the VSA volume show na weird behavior, but different than the file copy. The restore in major cases go slow down to 1 Mbps and in less cases it run at 160~200MBps.

We will proceed with the vdisklib test and test with another tools.

Thanks.

Andreas Neufert
Veeam Software
Posts: 4037
Liked: 734 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Verry slow restore do HPE StoreVirtual VSA

Post by Andreas Neufert » Apr 05, 2019 2:35 pm

My guess is that when we do a restore we transport a lot of data and therefore bringing the storage under pressure, which end up in a high latency (seconds instead of milliseconds). You can check this at the VMware Disk monitoring.

dcampregher
Service Provider
Posts: 152
Liked: 6 times
Joined: Mar 23, 2016 5:57 pm
Full Name: Diogo Campregher
Contact:

Re: Verry slow restore do HPE StoreVirtual VSA

Post by dcampregher » Apr 05, 2019 2:40 pm

We have the Veeam One monitoring the environment.

During the very slow restore, there is nothing running ( very low I/O and troughput) in the VSA Volume and the response time is very low. It seems the data don't sent with velocity to the volume or there is aanything dropping the communication.

Gostev
SVP, Product Management
Posts: 25309
Liked: 3763 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Verry slow restore do HPE StoreVirtual VSA

Post by Gostev » Apr 07, 2019 7:33 pm

There's no such thing as "slow velocity sending": next block is sent immediately after the storage confirms that it has received the previous block. Your performance testing with raw storage clearly shows that Veeam performance is only limited with the actual storage performance. Which makes it a VSA-specific issue, so I recommend you open a support case with HPE, to have them take a look at the internal storage performance metrics during the periods of slow down. Thanks!

Post Reply

Who is online

Users browsing this forum: Brad.linch and 15 guests