Discussions specific to the VMware vSphere hypervisor
Post Reply
evilaedmin
Expert
Posts: 106
Liked: 13 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

HotAdd mode for Replication destinations not recommended?

Post by evilaedmin » Oct 04, 2018 5:30 pm

We have proxies crashing due to: https://kb.vmware.com/s/article/56453

And leaving our Veeam Replica destinations in an invalid snapshot state.

Veeam support says (paraphrasing) that bug is an example of why HotAdd for replica destinations is not recommended.

I understand this recommendation with power of hindsight and experience with ESXi 6.5 U2 issue referenced above.

But is this a true recommendation in a general sense: Do not use hotadd for replication destinations and use san-direct or NBD instead?

Rick.Vanover
Veeam Software
Posts: 600
Liked: 127 times
Joined: Nov 30, 2010 3:19 pm
Full Name: Rick Vanover
Location: Columbus, Ohio USA
Contact:

Re: HotAdd mode for Replication destinations not recommended?

Post by Rick.Vanover » Oct 04, 2018 5:37 pm

I'd never over-step a recommendation from support who has reviewed logs and details, but I'll share some high-level advice based on what I've seen over the years:

-NBD mode is very reliable and works well (especially if physical proxy is used) but isn't the fastest option
-HotAdd mode is good for high-amounts of change rate - however it has a lot of "API Overhead" ->> Meaning a lot of sequences need to happen
-SAN is good - if you can do it, is an option and can perform well

You can easily "step down" how the proxy operates by dis-allowing it to use HotAdd and explicitly tell it to do NBD; which is a property of the proxy itself. If the proxy is physical - then it's inherently only able to do SAN or NBD. If it's a virtual proxy it can determine such - so maybe explicitly configure that proxy to go NBD.

Maybe a recommendation is to go NBD for now - and upon your next vSphere upgrade (or Veeam upgrade), re-introduce HotAdd to benefit from it's throughput capabilities.

evilaedmin
Expert
Posts: 106
Liked: 13 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

Re: HotAdd mode for Replication destinations not recommended?

Post by evilaedmin » Oct 04, 2018 11:10 pm

Thanks Rick.

We will have to do a lot of juggling of physical networking infrastructure and ESXi reconfiguration to get the "FQDN" of the ESXi host onto a 10Gb port, "management" is all 1GbE now. Is there a way to set an alternate VM Kernel port address or hostname for NBD? That would be simplest for us.

I'll have to read and think about SAN-direct a bit more. Our proxies are all virtual but we've virtualized guest iscsi traffic before and I don't believe this would be any different.

Rick.Vanover
Veeam Software
Posts: 600
Liked: 127 times
Joined: Nov 30, 2010 3:19 pm
Full Name: Rick Vanover
Location: Columbus, Ohio USA
Contact:

Re: HotAdd mode for Replication destinations not recommended?

Post by Rick.Vanover » Oct 04, 2018 11:36 pm

I believe alternative port and hostname manipulation is going down a bad path. I would think mass re-set of proxies to do NBD would be easiest - it's a property of the proxy.

dellock6
Veeam Software
Posts: 5734
Liked: 1625 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: HotAdd mode for Replication destinations not recommended?

Post by dellock6 » Oct 07, 2018 12:34 pm 2 people like this post

Be careful when choosing SAN mode for replication, it's totally fine at source but it's heavily limited by VMware VDDK at target: direct SAN is only supported for thick-provisioned disks, while our incremental data goes into a snapshot file, which is always thin-provisioned.
It's explained here: https://helpcenter.veeam.com/docs/backu ... tml?ver=95 (both restore and replica at target are write operations).

By deploying physical proxies, you risk to have those proxies only doing SAN mode on day 1 of each replica and then always go for network mode, which is where they will failover. This is why we usually suggest to have virtual proxies at target, so that you can use both hotadd (when applicable) or network mode.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

Rick.Vanover
Veeam Software
Posts: 600
Liked: 127 times
Joined: Nov 30, 2010 3:19 pm
Full Name: Rick Vanover
Location: Columbus, Ohio USA
Contact:

Re: HotAdd mode for Replication destinations not recommended?

Post by Rick.Vanover » Oct 07, 2018 3:45 pm

Very important points raised, Luca!

evilaedmin
Expert
Posts: 106
Liked: 13 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

Re: HotAdd mode for Replication destinations not recommended?

Post by evilaedmin » Oct 09, 2018 4:08 pm

dellock6 wrote:
Oct 07, 2018 12:34 pm
By deploying physical proxies, you risk to have those proxies only doing SAN mode on day 1 of each replica and then always go for network mode, which is where they will failover. This is why we usually suggest to have virtual proxies at target, so that you can use both hotadd (when applicable) or network mode.
Oi, so there is no good work around until that ESXi 6.5 Update 2 bug is resolved.

Regarding NBD, Is there any way to have Veeam interact with a VMKernel IP of a Host on anything other than the host's FQDN?

Rick.Vanover
Veeam Software
Posts: 600
Liked: 127 times
Joined: Nov 30, 2010 3:19 pm
Full Name: Rick Vanover
Location: Columbus, Ohio USA
Contact:

Re: HotAdd mode for Replication destinations not recommended?

Post by Rick.Vanover » Oct 09, 2018 4:18 pm

IP address only? But if vCenter is used - you'll be getting the hostname returned. You can add a host explicitly by IP (not the vCenter) of the vmkernel you are looking for.

jsprinkleisg
Service Provider
Posts: 22
Liked: 4 times
Joined: Dec 09, 2009 9:59 pm
Full Name: James Sprinkle
Contact:

Re: HotAdd mode for Replication destinations not recommended?

Post by jsprinkleisg » Oct 13, 2018 1:09 am 1 person likes this post

evilaedmin wrote:
Oct 09, 2018 4:08 pm
Regarding NBD, Is there any way to have Veeam interact with a VMKernel IP of a Host on anything other than the host's FQDN?
Rick.Vanover wrote:
Oct 04, 2018 11:36 pm
I believe alternative port and hostname manipulation is going down a bad path.

Yeah, for various reasons this is not something I'd normally recommend. But it's certainly possible to use hosts files to influence the proxies to resolve the FQDN to an alternate IP address, and thereby alter the data path.


From Veeam Backup & Replication Best Practices:
When ESXi hosts are added to vCenter it is recommended to use FQDN. When backing up through the network with the Network Block Device (NBD) transport mode, the FQDN is returned via VMware API for Data Protection (VADP) so the backup proxy server must be able to resolve the FQDN via DNS. Using the hosts file the data transport path can be altered for NBD transfers.

evilaedmin
Expert
Posts: 106
Liked: 13 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

Re: HotAdd mode for Replication destinations not recommended?

Post by evilaedmin » Oct 22, 2018 1:55 pm

But it's certainly possible to use hosts files to influence the proxies to resolve the FQDN to an alternate IP address, and thereby alter the data path.
Would I need to do this throughout my environment or just on the proxies? For example, /etc/hosts of Backup Server?

jsprinkleisg
Service Provider
Posts: 22
Liked: 4 times
Joined: Dec 09, 2009 9:59 pm
Full Name: James Sprinkle
Contact:

Re: HotAdd mode for Replication destinations not recommended?

Post by jsprinkleisg » Oct 29, 2018 4:15 pm

Just the proxies

ober72
Service Provider
Posts: 522
Liked: 74 times
Joined: Jan 24, 2014 4:10 pm
Full Name: Geoff Burke
Location: CANADA
Contact:

Re: HotAdd mode for Replication destinations not recommended?

Post by ober72 » Oct 30, 2018 11:52 am

Hi Folks,

I had issues trying to use Hot-add with replication targets. The proxies were occasionally not removing the hot added disks and then would not boot after doing updates. To be honest on 10Gb network, network mode seems to be more than good enough and I believe is even faster than hotadd.

cheers
Geoff Burke
Veeam Certified Architect

Post Reply

Who is online

Users browsing this forum: No registered users and 15 guests