HotAdd mode for Replication destinations not recommended?

[evilaedmin]

We have proxies crashing due to: https://kb.vmware.com/s/article/56453

And leaving our Veeam Replica destinations in an invalid snapshot state.

Veeam support says (paraphrasing) that bug is an example of why HotAdd for replica destinations is not recommended.

I understand this recommendation with power of hindsight and experience with ESXi 6.5 U2 issue referenced above.

But is this a true recommendation in a general sense: Do not use hotadd for replication destinations and use san-direct or NBD instead?

[Rick.Vanover]

I'd never over-step a recommendation from support who has reviewed logs and details, but I'll share some high-level advice based on what I've seen over the years:

-NBD mode is very reliable and works well (especially if physical proxy is used) but isn't the fastest option
-HotAdd mode is good for high-amounts of change rate - however it has a lot of "API Overhead" ->> Meaning a lot of sequences need to happen
-SAN is good - if you can do it, is an option and can perform well

You can easily "step down" how the proxy operates by dis-allowing it to use HotAdd and explicitly tell it to do NBD; which is a property of the proxy itself. If the proxy is physical - then it's inherently only able to do SAN or NBD. If it's a virtual proxy it can determine such - so maybe explicitly configure that proxy to go NBD.

Maybe a recommendation is to go NBD for now - and upon your next vSphere upgrade (or Veeam upgrade), re-introduce HotAdd to benefit from it's throughput capabilities.

[evilaedmin]

Thanks Rick.

We will have to do a lot of juggling of physical networking infrastructure and ESXi reconfiguration to get the "FQDN" of the ESXi host onto a 10Gb port, "management" is all 1GbE now. Is there a way to set an alternate VM Kernel port address or hostname for NBD? That would be simplest for us.

I'll have to read and think about SAN-direct a bit more. Our proxies are all virtual but we've virtualized guest iscsi traffic before and I don't believe this would be any different.

[Rick.Vanover]

I believe alternative port and hostname manipulation is going down a bad path. I would think mass re-set of proxies to do NBD would be easiest - it's a property of the proxy.

[dellock6]

Be careful when choosing SAN mode for replication, it's totally fine at source but it's heavily limited by VMware VDDK at target: direct SAN is only supported for thick-provisioned disks, while our incremental data goes into a snapshot file, which is always thin-provisioned.
It's explained here: https://helpcenter.veeam.com/docs/backu ... tml?ver=95 (both restore and replica at target are write operations).

By deploying physical proxies, you risk to have those proxies only doing SAN mode on day 1 of each replica and then always go for network mode, which is where they will failover. This is why we usually suggest to have virtual proxies at target, so that you can use both hotadd (when applicable) or network mode.

[Rick.Vanover]

Very important points raised, Luca!

[evilaedmin]

By deploying physical proxies, you risk to have those proxies only doing SAN mode on day 1 of each replica and then always go for network mode, which is where they will failover. This is why we usually suggest to have virtual proxies at target, so that you can use both hotadd (when applicable) or network mode.

Oi, so there is no good work around until that ESXi 6.5 Update 2 bug is resolved.

Regarding NBD, Is there any way to have Veeam interact with a VMKernel IP of a Host on anything other than the host's FQDN?

[Rick.Vanover]

IP address only? But if vCenter is used - you'll be getting the hostname returned. You can add a host explicitly by IP (not the vCenter) of the vmkernel you are looking for.

[jsprinkleisg]

Regarding NBD, Is there any way to have Veeam interact with a VMKernel IP of a Host on anything other than the host's FQDN?

I believe alternative port and hostname manipulation is going down a bad path.

Yeah, for various reasons this is not something I'd normally recommend. But it's certainly possible to use hosts files to influence the proxies to resolve the FQDN to an alternate IP address, and thereby alter the data path.


From Veeam Backup & Replication Best Practices:

When ESXi hosts are added to vCenter it is recommended to use FQDN. When backing up through the network with the Network Block Device (NBD) transport mode, the FQDN is returned via VMware API for Data Protection (VADP) so the backup proxy server must be able to resolve the FQDN via DNS. Using the hosts file the data transport path can be altered for NBD transfers.

[evilaedmin]

But it's certainly possible to use hosts files to influence the proxies to resolve the FQDN to an alternate IP address, and thereby alter the data path.

Would I need to do this throughout my environment or just on the proxies? For example, /etc/hosts of Backup Server?

[jsprinkleisg]

Just the proxies

[ober72]

Hi Folks,

I had issues trying to use Hot-add with replication targets. The proxies were occasionally not removing the hot added disks and then would not boot after doing updates. To be honest on 10Gb network, network mode seems to be more than good enough and I believe is even faster than hotadd.

cheers