Any plans to use Provisioning Stack in ESXi?

[Mehnock]

Hi,

I was just reading about the provisioning stack in ESXi and thought that would be a great stack for Veeam NBD traffic, has anyone tried forcing traffic down this stack (with prior setup of the provisioning stack)?

Also, is it possible to put a 10Gb NIC in the management network portgroup and leave the original NIC in there. Will the NBD traffic during a backup use this faster connection?

[PetrM]

Hi Christopher,

In this article we can see that provisioning stack has nothing to do with NBD:

Supports the traffic for virtual machine cold migration, cloning, and snapshot migration. You can use the provisioning TCP/IP to handle Network File Copy (NFC) traffic during long-distance vMotion.

But if you want to route NBD traffic over a dedicated network, perhaps you could have a look at this note and try the proposed method which should work starting from vSphere 7.

Thanks!

[Mehnock]

Yes I read the documentation and thought this would be a good improvement for VBR to use this stack but really all I needed was to be able to isolate my backup traffic and fortunately I am using vSphere 7 so I just configured my backup network using the way you suggested in the article. We'll see how it goes tonight.

[NickKulkarni]

Bumping an old thread. I can see ESXi 7.x implemented a VMK service for vSphere Backup NFC where the older ESXi 6.0 only had vSphere Replication NFC. Question I have is do I have to enable the VMK service for vSphere in ESXi 6.x for Veeam to use it for VM Restore?

see here https://vnote42.net/2022/08/10/how-to-i ... n-vsphere/

The reason I ask is I have already successfully isolated backup traffic to a separate network and NICs using a Custom TCP stack and a VMK on a separate vSwitch. I have also managed to put provisioning on this with a separate VMK. The increase in performance over the old situation (everything via the management ip and default TCP stack) is massive. What worries me is that during a recent full VM restore I noticed Veeam was using the management TCP stack with its built in (but not documented as far as I know) throttling to around 10MBps on the 1Gb management network.

Will enabling the vSphere Replication NFC service via a seperate VMK do the same thing for Veeam or is this a feature request we need for the next release of Veeam?

[PetrM]

Hi Nicholas,

Did you try selecting the same proxy in the restore wizard that was used for the backup job ("Pick proxy to use") ?

Thanks!

[NickKulkarni]

Interesting, no I left that on Automatic. Will experiment and see if that fixes it. Opened a support case and got an interesting reply too.

Let me answer your query.
"Does full VM restore use either the vSphere replication or vSphere replication NFC services and is this the reason Veeam is not using the backup IP range for restores? "
- In Veeam Backup & Replication, full VM restores typically utilize the NFC protocol, which relies on the management network by default. This is why Veeam might not be using your dedicated backup network IP range for restores.
- To ensure that VBR uses your custom backup network for restores, you can configure the vSphere Backup NFC option on the VMkernel adapter used for restore operations. This setting directs restore traffic through the specified network, bypassing the management network.
-------------------

[NickKulkarni]

I have tried the trick of assigning the custom tcp/ip stack to a VMK and enabling the vsphere replication and NFC replication services. It cannot be done.
https://www.techcrumble.net/2017/11/cre ... mkernel%29.

Not supported for those services, only the default, provisioning and vmotion stacks can be used for these services.

[NickKulkarni]

have confirmed this is still the case with Broadcom. see https://techdocs.broadcom.com/us/en/vmw ... twork.html and that it is the same through to Version 9. Based on this it would be a really nice feature of Veeam if they could force traffic from a restore via the Provisioning stack to allow traffic isolation away from the default management stack.

[PetrM]

Hello,

Could you please provide a support case ID? I’m not sure that this statement is entirely accurate:

in Veeam Backup & Replication, full VM restores typically utilize the NFC protocol

Restore job utilizes the same transport modes as the backup process, so if you have managed to isolate backup traffic for backup, I see no reason why it would not work with the same proxy for restore. What is the result when you use the same proxy for restore as you do for the backup job?

Thanks!

[NickKulkarni]

Hi, no problems Veeam Support - Case # 07554912 - does VBR use the provisioning stack in ESXi 6.x for full VM restore?
It would be good if we could sort this out in the same way as we did with the Backup isolation. The penalty for using the Management stack is severe.

[NickKulkarni]

Hello,

Could you please provide a support case ID? I’m not sure that this statement is entirely accurate:


Restore job utilizes the same transport modes as the backup process, so if you have managed to isolate backup traffic for backup, I see no reason why it would not work with the same proxy for restore. What is the result when you use the same proxy for restore as you do for the backup job?

Thanks!

Hi Petr,

not sure what relevance the proxy is to the restore as all proxies are in effect equal in both configuration and advantages. i.e. there is one Linux based VM proxy on each ESXi host. The thinking being that each host would have a hot add proxy for backup use by the VBR server. Each VM being backed up resides for both compute and storage on the same physical ESXi host. Storage is a local HDD storage on the ESXi host so there is no SAN involved. Having a VM proxy on each host for hot add should, in theory, give maximum backup efficiency I thought. I am getting good throughput during my backups.

The problem is only during a restore.

To explain the physical network topology involved.

The Primary and Secondary backup repositories are iSCSI extents on NAS servers. These use Microsoft's ISCSI Initiator on the VBR server and are showing as local drives on my VBR Server which is a VM on one of the ESXi hosts.

Primary repository is FreeNas on an HP Server chassis connected via static LAG over a pair of 1GBe Ethernet connections on the same physical switch stack as the ESXi hosts. This Freenas is where all the backups go to during normal backups.

The secondary repository is over a low performance (300MBps) WiFi link to another building (the offsite part of the strategy) so is dealt with by a Backup Copy job from the primary repository. We therefore do not restore directly from the Secondary unless it is absolutely vital i.e. in genuine disaster recovery scenario so its performance issues can safely be ignored.

As our "restore from" repository is the same as our "backup to" repository using exactly the same proxies in each case I would expect performance during a restore to be the same as a backup no matter which proxy I used because the physical locations are the same and the proxies are identical on all three ESXi hosts.

based on https://helpcenter.veeam.com/docs/backu ... ml?ver=120 which you linked to I would expect Virtual Appliance mode to be the default as the ISCSI extent is seen as local disks. The Matrix of transport modes says Local Storage: Install a VMware backup proxy on a VM on every ESXi host. This is I believe exactly what I have done.

In the test case I restored a full VM from primary repository to the same Physical ESXi host that my VBR VM resides on. I would therefore assume that apart from the connection to the FreeNas the Hot Add proxy would have direct access to local storage on the ESXi host and would benefit from this.
I hope this makes sense Petr, I would value any input you have on this. Thank you
Nick

[PetrM]

Hi Nicholas,

Many thanks for the detailed description of your infrastructure!

You mentioned above that Veeam uses the management TCP stack, but was it used:
1. By the proxy for writing data to a target datastore in HotAdd?
or
2. Between Veeam Data Movers when the data flow goes from the Data Mover running on a repository (VBR server, in your case) to a Data Mover on the proxy?
or
3. By the Source Data Mover on the VBR server, which reads data from NAS over iSCSI?

We had a quick chat with the support engineer, and the action plan now is to answer the questions above, understand what the 'bottleneck' is, determine what transport mode was used for the restore, and check the restore with the same proxy. I agree that all proxies seem the same at first glance, but perhaps there are some differences that are not obvious.

Thanks!

[NickKulkarni]

Hi PetrM,
Thanks for your input, much appreciated.

The logs indicate that Management stack was used. I am therefore removing the Standalone hosts from the VBR infrastructure and will be relying solely on vSphere. Will report back once I have done this and run another test restore.