Verry slow restore do HPE StoreVirtual VSA

[dcampregher]

We are in face to a problem that is very weird. The Customer in case, use the HPE StoreVirtual VSA ( 02X HPE Server with 13 SSD disks, 2GB SAS Controller in RAID5, vSphere 6.5U2, volumes in NetworkRAID10, all the network is 10Gbps or more ).

The Veeam Server ( all in one ) is a VM in a dedicated local datastore with 20 disks SAS and a 2GB controller in RAID5.

All the backups are very fast. The IVMR, FLR, Restore VM Files and so on is very fast. But, when i try to restore the entire VM to the volumes in the VSA ( source volumes) the restore goes down to 1 or 2 Mbps and stay there.
If i try to restore the same VM ( same parameters of restore ) but to the local ESXi SSD Datastore ( out of the VSA but in the same ESXi ) the restore go up to 300MBps.

No matter what transport mode or any parameter in the Veeam that we change, if the restore is sent to the VSA Volumes the restore rate is very slow

I have a Tier-2 case to search a root cause. Any idea of how this could be happening ?

[HannesK]

Hello,
could you support the case number? And just to clarify: you say that restore of VM files (large disks) is fast?

Best regards,
Hannes

[Andreas Neufert]

If I understood correctly, you restore same VM twice on the same ESXi host, one restore is fast and one restore that goes to StoreVirtual is slow. Can you please check with a disk test tool (1MB block size) the write speed? As well you can try to clone a VM from the fast storage to the StoreVirtual to see if you achieve same problematic speed or if it is faster.

As well, please check at the restore statistics what mode is used for restore (NBD-Network/HotAdd/DirectSAN) and if it is different when you restore it to the other faster storage.

[HannesK]

my understanding is the same. Support can help with the test tool (VixDiskLib), they use it quite often.

[dcampregher]

Hi!
The case number is 03474350.

The restore to the local datastore run this:
05/04/2019 09:28:02 Restoring Hard disk 2 (150,0 GB) : 103,0 GB restored at 300 MB/s [hotadd]
05/04/2019 09:26:51 Restoring Hard disk 1 (50,0 GB) : 44,5 GB restored at 185 MB/s [hotadd]

The restore to the VSA datastore run this:
05/04/2019 09:57:55 Restoring Hard disk 1 (50,0 GB) : 251,0 MB restored at 802 KB/s [hotadd]
05/04/2019 09:57:55 Restoring Hard disk 2 (150,0 GB) : 175,0 MB restored at 721 KB/s [hotadd]

The VM clone from the local datastore to the VSA datastore get 12 minutes.

All the operations ( restores and clone ) from the same ESXi host and same parameters for the two restores.

[Andreas Neufert]

Looks like the VSA is just slower. At HotAdd we mount the restore VM disk to the Veeam Proxy running on the host and then we write directly to this disk the data. As you had good speed at the other storage, the Veeam infrastructure is not the bottleneck.

Do you have opened a ticket at HPE to look into this performance issue?

[Andreas Neufert]

Checked the support case. There is a statement there that the issue was recreated with Windows File copy within the VM, so you have a performance issue on the storage system.

[dcampregher]

There is a detail.

The problem is not fixed in one VSA volume. It can happen now in the volume01 and tomorrow not, migrating the problem to the volume 03 and so on....

And this difference from 300 to 1 MBps ? It seems not a performance issue but the blocked performance by anything .....

03474350

[Andreas Neufert]

Yes, but there is nothing we could do differently with the volume. During the support case the same issue happend independantly reproducable with Windows Fiele copy to a StoreVirtual volume. So it is important that HPE will have a look to identify what will cause this issue within the storage.

[dcampregher]

The "exact" issue not. During the file transfer with Windows Explorer we see the transfer ratio with high performance but with variable and in some time it stop by a fews seconds, i understand you when you tell that there is a problem with storage system and because this there is a another engineer ( our engineer ) to do an assessment in the environment.

In fact, the restore when sent to the VSA volume show na weird behavior, but different than the file copy. The restore in major cases go slow down to 1 Mbps and in less cases it run at 160~200MBps.

We will proceed with the vdisklib test and test with another tools.

Thanks.

[Andreas Neufert]

My guess is that when we do a restore we transport a lot of data and therefore bringing the storage under pressure, which end up in a high latency (seconds instead of milliseconds). You can check this at the VMware Disk monitoring.

[dcampregher]

We have the Veeam One monitoring the environment.

During the very slow restore, there is nothing running ( very low I/O and troughput) in the VSA Volume and the response time is very low. It seems the data don't sent with velocity to the volume or there is aanything dropping the communication.

[Gostev]

There's no such thing as "slow velocity sending": next block is sent immediately after the storage confirms that it has received the previous block. Your performance testing with raw storage clearly shows that Veeam performance is only limited with the actual storage performance. Which makes it a VSA-specific issue, so I recommend you open a support case with HPE, to have them take a look at the internal storage performance metrics during the periods of slow down. Thanks!