Does Veeam use ESXi management network to carry traffic?

[HendersonD]

Our ESXi hosts all have two 1Gbe connections for the management network and two 10Gbe connections for VM, iSCSI, and vMotion traffic. We use Veeam for both backup and replication and have a physical proxy server to take advantage of SAN mode. Does Veeam use the ESXi management network to carry any of its backup/replication traffic? I am thinking for backup traffic it does not but for replication this might be different.

"Veeam Backup & Replication uses the Direct SAN access transport mode to read and write VM data only during the first session of the replication job. During subsequent replication job sessions, Veeam Backup & Replication will use the Virtual Appliance or Network transport mode on the target side. The source side proxy will keep reading VM data from the source datastore in the Direct SAN access transport mode."

For replication on the target side it appears that we are using network mode. What ESXi interface does this traffic pass through? The management interface? The reason I ask is we are purchasing new hosts and if Veeam replication or backup traffic passes through the management network I want to make sure the nic that the management network uses is 10Gbe

[PTide]

Hi, in case Network mode is used the target side proxy will connect to ESXi hosts on VMkernel interfaces.

Thanks

[HendersonD]

I should have done this before. I have SFlowTrend running and directed several different interfaces in my Production and Disaster Recovery sites to this product. I ran a replication and there were two interfaces that were heavily used. On the production side it was a 10Gbe interface that connects the proxy server. This means that SAN mode is doing the work on the source side.

The other heavily used interface was the management interface on my ESXi host that is the target of the replication. In other words, it is not the kernel interface for storage that is getting used during replication, it is the management interface. Moral of the story seems to be make sure that the ESXi host (or vCenter) that is the target for replication has a 10Gbe interface for management

[HendersonD]

I also ran a backup which should use SAN mode on the source and target side. I surprised to see the 1Gbe interface for the management network in production (source) was quite active during backup with utilization according to SFlow sitting at 50%. What traffic is carried by by the management interface during backup? Is it the traffic to tell ESXi to take the VM snapshots? This would seem to be a small amount of traffic. In SAN mode a physical proxy server should be generating all of its traffic on the storage interface.

I also have a second virtualize proxy server that is strictly used as the guest interaction proxy. Could the traffic on management interface be generated by this proxy server as it does the injection into each VM for application aware processing?

[PTide]

In other words, it is not the kernel interface for storage that is getting used during replication, it is the management interface.

Since management traffic is carried through VMkernel adapter, I'm really confused with what you've said. Please elaborate.

Moral of the story seems to be make sure that the ESXi host (or vCenter) that is the target for replication has a 10Gbe interface for management

That's not necessary if the target proxy utilizes hot-add mode. In this case all disk data will flow via the network which the proxy VMs belong to.

I also ran a backup which should use SAN mode on the source and target side. I surprised to see the 1Gbe interface for the management network in production (source) was quite active during backup with utilization according to SFlow sitting at 50%.

Would you confirm that it was indeed Direct SAN mode on the source side please? To do that you need to check the job session and see if there is "[san]" string in there. The management interface being utilized up to 50% during Direct SAN mode does not seems right.

Thanks

[HendersonD]

I just checked one of the VMs that was replicated and this is what I see:
Using source proxy vProxy1 for retrieving Hard disk 1 data from storage snapshot
Using target proxy vProxy1 for disk Hard disk 1 [nbd]

This verifies that it is using SAN mode on the source side and network mode on the target side during replication. My physical proxy server is directly accessing my production SAN to read the data and then writing it to storage on the target side via an ESXi host. In other words, the data on the target side goes through an ESXi host which is how replication works. The question becomes which interface the data coming into on the host?

My current host has two 10Gbe interfaces for VM, iSCSI, and vMotion traffic. It also has a 10Gbe interface for the management network. During replication, my sFlowTrend server is showing that the management network on this ESXi host is at 22% utilization. During replication, there is 220Mbps of data flowing through the management network. This is why I am thinking during replication the management network is where data flows through.

For my backup jobs this is what I am seeing for one of my VMs. I am fairly certain I am using SAN mode but do not see this. My guess is vProxy2 which is my guest interaction proxy is utilizing the 1Gbe management network during backup to do its injection. My physical proxy, vProxy1 should be using SAN mode for source and target during backup

Code: Select all

Using guest interaction proxy vProxy2 (Same subnet) 
Inventorying guest system 0:00:02
Preparing guest for hot backup 0:00:04
Releasing guest 
Creating VM snapshot 
Getting list of guest file system local users 
Collecting disk files location data 0:00:12
Removing VM snapshot 0:00:02
Queued for processing at 12/4/2017 12:26:00 AM 
Required backup infrastructure resources have been assigned 0:04:50
VM processing started at 12/4/2017 12:30:55 AM 
VM size: 60.0 GB (19.9 GB used) 
Saving [VMWare3] Licensing/Licensing.vmx 
Saving [VMWare3] Licensing/Licensing.vmxf 
Saving [VMWare3] Licensing/Licensing.nvram 
Using backup proxy vProxy1 for retrieving Hard disk 1 data from storage snapshot 
Hard disk 1 (60.0 GB) 1.1 GB read at 322 MB/s [CBT]0:00:21
Saving GuestMembers.xml 
Finalizing 0:00:01
Truncating transaction logs 0:00:01
Busy: Source 99% > Proxy 26% > Network 57% > Target 0% 
Primary bottleneck: Source 
Network traffic verification detected no corrupted blocks 
Processing finished at 12/4/2017 12:32:10 AM

[PTide]

The question becomes which interface the data coming into on the host?

Since it is NBD mode, the proxy connects to ESXi hosts on VMkernel interfaces by DNS name resolution and use this connection to transport data. If you want to change that behaviour you need to use Virtual appliance/Hot-add mode on the target proxy.

y guess is vProxy2 which is my guest interaction proxy is utilizing the 1Gbe management network during backup to do its injection.

The injection is small and should not cause spikes. Another question - does your vProxy reside on the same network with the VMs in the job?

Thanks

[HendersonD]

My understanding is VMKernel ports just provide access to the outside world. VMKernel ports can carry different types of traffic including

1. vMotion traffic
2. Management traffic
3. iSCSI traffic
4. NFS traffic

It appears that for replication that uses NBD on the target side, the traffic does go through the management network. This means to get decent throughput for replication, the management network should be backed by a 10Gbe nic and not a 1Gbe nic. Is that correct?

The physical proxy, vProxy1, has an IP address of 10.121.125.83
The virtual proxy, vProxy2, has an IP address of 10.121.125.85
Most but not all of the VMs being backed up have an IP address of 10.121.125.xxx
Using SAN mode, the proxy server accesses my SAN directly across my storage network which is 10.121.126.xxx
The management network on my ESXi hosts is on a separate vlan with IP addresses of 10.121.123.xxx

My core switch/router routes at wire speed so would it make any difference if I have several different vlans involved?

During a replication job I am seeing these two messages
Using guest interaction proxy vProxy2 (Same subnet)
No available proxies are running on ESX(i) management interface subnet. Using proxies from a different subnet, performance may be impacted.

[PTide]

Is that correct?

That's correct.

My core switch/router routes at wire speed so would it make any difference if I have several different vlans involved?

It does not make any difference for backup as long as the required infrastructure is accessible by the corresponding backup components.

I have some questions though. As far as I understand your setup:

Source Proxy1 (physical) has direct connection to SAN and gets VM data from storage snapshots which is what you were trying to achieve.

Where does you VBR server reside?
Where does Proxy2 (virtual) reside? Why don't you assign Guest Interaction proxy role to Proxy1 since it is close to the source and has access to VM subnet?

No available proxies are running on ESX(i) management interface subnet. Using proxies from a different subnet, performance may be impacted.

As far as I understand, neither Proxy1 nor Proxy2 has an access to ESXi management network, and that makes me wonder which proxy is being used. Which server has an access to ESXi management network?

Thanks

[HendersonD]

We are on a single 192 acre campus. My production data center is on the ground floor of the High School with our DR site across campus in the district office space. Underground fiber connects these locations at 10Gbe. The Veeam backup server, vProxy1 (physical), and vProxy2 (virtual) all reside in the DR site. So vProxy1 is reading data from our production Nimble array using direct SAN access across a 10Gbe link. It then writes this data to the ESXi host in our DR site using NBD with data flowing into the 10Gbe management network interface.

I decided to use vProxy2 as the guest interaction proxy just to split the load a bit. I could use vProxy1 to read/write the data and be the guest interaction proxy but then vProxy2 would sit idle.

Both proxy's have access to the ESXi management network. The vlan that the management network is part of is routeable. The ESXi host in our DR site has a hostname of vSphere15. If I ping vSphere15.vcs.local it returns 10.121.123.175 which is the IP address of the management interface on this host. The only vlan that we have that is non-routable is our storage vlan. Per best practice, storage (iSCSI) traffic should never be routed. This means that vProxy1 must have two nics:

1. A nic that is in the non-routable storage vlan (10.121.126.83)
2. A nic in our server vlan that is routable and can reach the ESXi management network (10.121.125.83)