HotAdd or Network mode

[aevans@ricoh.com.au]

Hi All,

We've been running Veeam happily for around 2 years now and been very happy with the speed and reliability.

Recently we just did a migration where our Production hardware became our DR hardware (and vice versa).
This meant production went from 6 Dell Blades on Compellent storage (Thick Provisioned) to 6 UCS Blades on NetApp storage (Thin Provisioned).

What I've found is that our replication jobs are taking hours longer than previously and often never finishing leaving VM's needing disk consolidation.
Pre-migration we could usually backup our entire 250VM environment in 3-4 hours. Now the jobs are still running after 10-16hours.

The main differences I can see are the following.
1. VM's are replicated from NFS to VMFS5 as Thin Provisioned (previously from VMFS5 to NFS as Thick)
2. Running 3 replication jobs with Parallel processing enabled (previously we had 15 jobs)

So I opened a support ticket and was told to use Network mode instead of HotAdd over the WAN connection which brings me to my question.
We have a ring network between Production and DR sites that is 20gb dark fibre, basically it will never ever be the bottleneck and it couldn't be more stable as if we lose one network link failover to second link is instant and non-disruptive.

In the past we've always used HotAdd as I thought this gave us best performance but I just read that Network mode can be useful in architectures that allow 10gb+ connectivity.

So should I be using Network mode or is HotAdd always going to be the faster transport?

FYI case ID is 00574113

Thanks.

[aevans@ricoh.com.au]

FYI I did try switching to Network mode but came across the 7 concurrent tasks issue. In order to workaround I'd have to reconfigure all my jobs to point to specific ESX hosts rather than the DR cluster. So this meant instead of 16 VM's backing up concurrently I only got 7 which was slower...

However I did notice than currently the bottleneck is reported as Target, under network mode the bottleneck was being reported as source.
A Network mode job ran for 2 1/2 hours with 151MB/s processing rate yet it only processed 600gb, read 200gb and transferred 4.6gb
A HotAdd job ran for 1hr with 51mb/s processing rate but processed 1TB, read 127gb and transferred 101gb.
I cancelled both in the end to try some other configurations, so not sure if any of that information is of use.

[Vitaliy S.]

We have a ring network between Production and DR sites that is 20gb dark fibre, basically it will never ever be the bottleneck and it couldn't be more stable as if we lose one network link failover to second link is instant and non-disruptive.

In the past we've always used HotAdd as I thought this gave us best performance but I just read that Network mode can be useful in architectures that allow 10gb+ connectivity.

So should I be using Network mode or is HotAdd always going to be the faster transport?

If you have 10gb network connection from source to proxy and then to target location, then it does make sense to make a switch to network mode.

However I did notice than currently the bottleneck is reported as Target, under network mode the bottleneck was being reported as source.
A Network mode job ran for 2 1/2 hours with 151MB/s processing rate yet it only processed 600gb, read 200gb and transferred 4.6gb
A HotAdd job ran for 1hr with 51mb/s processing rate but processed 1TB, read 127gb and transferred 101gb.
I cancelled both in the end to try some other configurations, so not sure if any of that information is of use.

Did you check with your support engineer what operation took the most time when you were using hotadd mode? Also not sure I fully understand the figures... are these for the same or different jobs?

[jbarrow.viracoribt]

If you have 10gb network connection from source to proxy and then to target location, then it does make sense to make a switch to network mode.
Did you check with your support engineer what operation took the most time when you were using hotadd mode? Also not sure I fully understand the figures... are these for the same or different jobs?

Trying this now, I have that exact setup and I had been using hotadd thinking it was the fastest method.

[jbarrow.viracoribt]

Q: How does it work?
A: Backup proxy server retrieves source VM data from production storage through ESX(i) network stack over management network.

Can you elaborate on the Management network portion of this statement?

[jbarrow.viracoribt]

Okay, I'm dumbfounded. My speeds just went through the roof with Network mode vs. Hotadd. Why is that?

[dellock6]

hotadd requires some pre and post job activities related to VMware operations, to mount and unmount the vmdk to the hotadd proxy. Network mode instead starts immediately to copy data after the snapshot is created, so sometimes you can experience faster operations, especially on small VMs (it takes more time for hotadd operations than the backup itself sometimes...)

[tsightler]

Okay, I'm dumbfounded. My speeds just went through the roof with Network mode vs. Hotadd. Why is that?

Hotadd is generally faster and reading and writing data than network mode, even over 10GbE (although the gap is narrowed significantly). However, there are several factors that can make network mode be a real win.

1. When you have a lot of servers that generally have a small change rate, and you have 10GbE, it's unlikely that hotadd is going to be a win for the incremental transfers. The reason why is simple to understand if you think about it. As Luca noted the hotadd process itself has a lot of steps, after the snapshot is taken Veeam has to tell vCenter to attach the disk to the proxy VM, the proxy OS has to scan and find the new disk, and at the end of the job all of this has to happen in reverse before the snapshot is taken. In a lab or very fast setup this can take 30 seconds, but in the "real world" this can easily be 60-90 seconds or more. That's 60-90 seconds where no data is being transferred at all. If I have 100 servers that 100-150 minutes of time spend doing nothing but setting up and tearing down hotadd. With replication you have to do all of this on both the source and target proxies (or twice on the same proxy if it's local replication).

If you have 10GbE for the ESXi mangement network, network mode can easily transfer 300-400MB/s per host, hotadd might be able to do faster, but how much? Even if it's twice as fast, and you have 6GB of data to transfer, that's 20 seconds to transfer data for Network mode, 10 seconds for Hotadd, but hotadd still needs 60-90 seconds for all of the setup/teardown, so that's 20 seconds vs 70 seconds best case, so network mode wins!!

2. With replication there is another special issue with using hotadd that can cause a huge amount of I/O on the target, especially for block devices. This is typically an issue when replication happens over a high speed link and you see "Target" at the bottleneck. In this case, you're going to be way better off using Network mode on the target side at least (you can still use hotadd on the source if it makes sense for you). The good thing I can say is that, at least so far in my limited testing, this specific issue is resolved in 8.0, but testing is ongoing and I'm waiting for some field results to confirm 100%.

3. Hotadd and NFS has a tendency to not get along well due to NFS locking issues. This doesn't really impact replication to the target since the VMs are powered off and powered off VMs don't have locks, but when using Hotadd with NFS at the source, it can be huge issue if you don't have a proxy on every host and you don't set a special registry key for Veeam to use only proxies on the same host as the VM in question. Or you can just use Network mode. You can read more about this here.

To optimize backup, and especially regular replication, you really have to look at the details of the logs and understand what the system is spending its time doing before you decide if hotadd is "faster". Hotadd will almost certainly be faster for full backups, but with a good solid Veeam setup you hopefully won't be running full backups very often, so that may not be the ideal focus of the design as, in most cases incremental data transfer won't be the number one contributor to the length of the job runs.