Quick migration from storage to storage slow vs vMotion - SAN/HotADD

[leviu]

Tl;DR: Quick migration from one storage system to another using in SAN/HotAdd in any combinations of source/destiantion reaches only 90MB/s vs vMotion which reaches 250MB/s (so do the backups). How come?

Latest 11a patch here.
The logs are in support 05251738, which is closed due to network issues on that particular server. Ended up using another backup server to transfer the VM in using the Linux proxy in HotAdd mode at 75MB/s.

We have a physical server that has single 8G fiber connection via a fiber switch to 2 storage systems: 3par 7200 and MSA 2040. There are 3 ESXI hosts and a vCenter. Each of ESXI hosts is connected with dual 8G fiber connections. The network connection is only 1GB on the physical backup server and 4 x 1GB on ESXI hosts. There is a single fresh Linux backup proxy (Debian 11) on one of the ESXI hosts. All LUNs are exported in RW mode to the backup server.

I wanted to migrate a couple of big VMs (5TB) from the 3par to MSA. We don't have storage vMotion for live migrations, but for performance tests I've migrated a few test VMs using storage vMotion when they were turned off to see the expected performance on the target storage (MSA). I got around 250MB/s - looking and the MSA performance graphs in its UI and doing some simple math on how long the VM took to be moved vs its size.

Then I've tried a Quick Migration to transfer one of the big servers and got 75MB/s using the physical backup server in direct SAN mode. I've tried offloading the source to the Linux proxy and it got to 90MB/s. However, this is nowhere near the capabilities of the storage systems as evident by the offline vMotion storage migration test. Its not like the VM is under a heavy IOPS load.
This smells like just the limits of the 1Gbps copper network connection of the physical backup server. Albeit, I've tried to add another NIC and created Windows NIC teaming to get 2Gbps, but that did not help.

How come the Quick Migration is so much slower? Everything is using 8G fiber for storage access - be it the physical backup server with direct SAN or the backup Linux proxy via HotAdd (via the ESXI hosts fiber connection).
Please help me comprehend where my understanding of how Veeam components work gone wrong...

EDIT: Backups happily run at 360MB/s (CBT) processing rate from the MSA storage system.

[PetrM]

Hello,

First of all, Veeam uses native VMware vMotion and Storage vMotion whenever it is possible and fails over to its proprietary technology only if native VMware methods cannot be used. You can find more information on this page. Therefore, the first step would be to understand why does Veeam make failover to its own mechanism?

Speaking about data processing ratio, it's essential to determine the "bottleneck", the slowest stage in data processing conveyor, for instance data read from production storage or data transfer over the network link, or data write to the target. But this question is difficult to analyze over the forum posts without remote access and debug logs so I'd leave the opportunity to troubleshoot this technical issue to our support team. As far as I see the case 05251738 is closed, do you have another support case for this issue or may I ask you to explain what was the conclusion provided by support engineer?

Thanks!

[leviu]

Should I open a new case for the performance issues?

The issue in case 05251738 was that the Quick Migration was out right failing (after running for a bit), which eventually we discovered was caused due to server networking issues.
However, the support engineer did address my complaints about the slow speeds during a live session (but not entirely). He suggested to split the load between the Linux proxy (HotAdd) and physical backup server (SAN), which bumped the speed for 75 MB/s to 90 MB/s - that is a good improvement, but nowhere near 250 MB/s VMWare transfer can do.

The stats presented for a Quick Migration that started with source HotAdd and destination in SAN mode and failed over to NBD mode (then I stopped it) processed 390 GB:

Code: Select all

Busy: Source 71% > Proxy 14% > Network 77% > Target 80%  - Bottleneck: Target

Another job with the same config that failed over to NDB after a couple of hundred GB:

Code: Select all

Busy: Source 69% > Proxy 16% > Network 81% > Target 77%  - Bottleneck: Network

Another job that was HotAdd for source and destination and did not fall over to NDB mode (failed due to connection issues) processed 160 GBs:

Code: Select all

Busy: Source 73% > Proxy 26% > Network 72% > Target 65%  - Bottleneck: Source

In none of these sample did the speed exceed 100 MB/s.

Since we don't have the enterprise license for VMWare there is no vMotion for live VMs. Hence, I just tick the box during a Quick Migration for Veeam to use its own means. Albeit there was one quick migration where it somehow managed to do use vMotion on a live VM - that is still a mystery to me.

[PetrM]

Hello,

I think you should open another case for performance troubleshooting in order to understand if there is an option to accelerate the process. According to the statistics above, the "bottleneck" floats between Source and Network when SAN mode is used on target.

I'd suggest to check two things with the support team:
1. Is it possible to make data read faster in HotAdd mode or to use another transport mode for read?
2. Is it possible to optimize data transmission over the network channel between Data Movers? Maybe this setting might help to optimize performance in your case.

However, I assume that even in the best case the processing rate would be just a bit higher than 100 MB/s. The "bottleneck" will be shifted to Target from Source/Network but percentage ratios of all of these stages are in the same range now: 65 % - 80 %. Anyway, it's just an assumption and more precise technical analysis is required for the final conclusion.

Thanks!