Linux Backup Proxy Multipath Direct storage access transport mode

[JaySt]

i've recently went back from direct san performed by a single physical proxy/repository to using two virtual proxies running on two seperate ESXi hosts sending data over 10Gbit networking to the same repository. Job runtimes were better/shorter( some by alot! (90 mins to 30 mins incrementals), administration is easier etc.
Currently evaluating pro's and cons, but i think hotadd/virtual proxies is winning.

[nt25]

Well, I have 32 virtual proxies, so my final goal was to reduce management overhead by replacing these vms with a few physical boxes

[HannesK]

Hello,
thanks for bringing up the vixDiskLib.transport.san.whitelist option. I will ask R&D to check again. Yes, the config files you mention (*vmc*) sound more like for VMware Cloud on AWS (VMC).

so my final goal was to reduce management overhead by replacing these vms with a few physical boxes

that's a valid goal. I cannot estimate how long an improvement on our side could take... so my question would be: do you really max out the 16 / 20 / 32 / 40G single link connection? 32 Proxies sounds like you have thousands of VMs. Using Windows might also be a workaround.

Best regards,
Hannes

[JaySt]

sorry, perhaps i lost it somewhere, but why would Windows be a workaround? to reach which goal?

[HannesK]

windows does multipathing automatically as far as I see

[HannesK]

another workaround if the storage is "integrated" with Veeam: Backup from Storage Snapshots also uses multi-pathing on Linux

[emachabert]

Hannes,

I have been troubleshooting this for 2 days long and what we see is that it is not using mutlipath "every time". Seems like sometimes it does and some time it doesn't.

We use the throughput graph from the FC fabric to be sure.

Setup: Apollo 4200 running RHEL 9.1, Direct SAN FC16gbs to Alletra 9080 with backup from storage snapshot enabled.

During backup, multipathd creates the DM device and everything is good at mutlipath level.
Some times it uses the multipath and we have trafic on both FC link.
Some times only one link is used and the other sees no trafic.

We will open a support case.

[JaySt]

"During backup, multipathd creates the DM device and everything is good at mutlipath level."

interesting! i'm not using backup from storage snapshot (just direct SAN), so i have not yet seen the creation of DM devices during backup.
I was not aware of the DM creation logic of Veeam begin there, but makes sense with BfSS.
The fact that it sometimes uses DM devices in your case (so it is possible somehow...) makes me wonder even more why the DM is not used during direct SAN backups (without BfSS) in my case.

[bct44]

Thank you @emachabert, i missed this information. I will check that tomorrow morning with my colleague, i don't have storage snapshot neither.

[emachabert]

Dm devices are notre created by veeam.
It is the role of the multipathd daemon to create the device manager managed devices.
The multipathd create a virtual device that takes care of multipathing using the policies configured in the multipath.conf (type of distribution, failover, checker type etc)

My only doubt here is what really happens when I see that both links a fully used. Either it is the Veeam process that is reading from the dm device as we all would like it to be in that thread or it is in fact simply the fact the backup job is spread on many datastore and I then get many spnapshos mounted for backup and it randomly get their access distributed over the two links.

We clearly see a different pattern between Linux and Windows proxies where windows parh are evenly used and maxed out

Support case is open and we will know more about it soon.

[JaySt]

ah i understand: multipathd policies create the DM devices when the snapshots appear for the OS during BfSS.

And yes, i also think it would be more consistent with behavior i saw (and read here) to suspect the multple links being used because you're hitting multiple datastores, and not because multipathd is doing its thing on a single one of them.

[emachabert]

Hannes, here the case number in case you have time to look at it #06287720.

[emachabert]

Some graph to explain:

When starting a job spanning many datasore, at start you might say: perfect, both path are evenly used, and then it shows a non uniform pattern.

One job


Another one:


And when you launch a job targeting one datastore, only one path is used (green)

[HannesK]

Hello Eric,
thanks for the screenshots. I will check internally!

Best regards,
Hannes

[emachabert]

Support case found the root cause and solved the issue with a registry key to make the path discovery timeout bigger.

This is great because this is what we internally thought while observing the proxy, we were thinking the Veeam process was not waiting long enougth to have the multipathd make its work. Especially with big jobs mouting many snapshots on the proxy.

Code: Select all

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Name: LinuxProxyDiskPathQueryTimeoutSec
Value: 10 - Decimal

Here are the graph after adding the registry key.



A stable 3300MB/s on FC network during the backup.

[JaySt]

Great! good news something can be done about that using the registry value and this confirms some Veeam/mpath logic being used.
Im left with figuring out why my direct SAN (non-BfSS) iSCSI setup is not using the DM devices correctly then.

[emachabert]

Give a try to the registry key

[HannesK]

Eric: glad to hear that there is a solution! We are discussing to increase the timeout.

JaySt: please remember that multipathing for BfSS does not imply multipathing to work for direct-san. For direct-SAN, we use VDDK. For BfSS the "advanced data fetcher".

[JaySt]

ah yes, ADF. i forgot about that one! makes sense now.
So coming back to Windows vs Linux:
Does/should ADF also provide the same benefits on Windows using MPIO?

[HannesK]

yes, multipathing on Windows should work for direct SAN and BfSS via built-in Windows functionality (the Multipath I/O feature needs to be enabled in the Windows Server configuration). If you see something else, please provide more details / support number (separate thread would be great, because this thread is about Linux multipathing )

LUN masking needs to be pre-configured. There is no special logic on our side to do that. We only use Windows native functionality.

[JaySt]

it's more that i'd like to discover which logic is used in each situation (Linux vs Windows, BfSS vs non-BfSS).
For Linux, i now know that non-BfSS processes will use VDDK and not use mpath devices, so single path is used. Using BfSS on Linux will use ADF and therefor use mpath devices.
On Windows, i now conclude that MPIO will be used in all cases, so multiple paths used regardless of the use of BfSS.
(please correct if wrong).

[coolsport00]

Hello all. I created a new post with somewhat similar questions as is noted in here. But, as this post main topic was multipathing with iSCSI on a Linux server configured with DirectSAN, I thought it best to create a separate topic as mine is mostly about **configuration** for DirectSAN. Would be great to get your input in my post though..URL is as follows:
vmware-vsphere-f24/specifically-how-to- ... 91508.html

Really appreciate it!
Shane