Feature Request - Create Veeam Replica

[owen.wright]

Hi guys

Ive recently had issues with customer sites creating veeam replicas of the veeam VM. This has resulted in some issues where backups using hot-add work perfectly, and most replicas using hot-add complete perfectly, however the replica job for the Veeam server fails over to NBD. I get that this is a known issue (https://www.veeam.com/kb2008 , issue number after logging a ticket with support (02387488).

According to your help file (https://helpcenter.veeam.com/docs/backu ... tml?ver=95) its not recommended to backup / replicate the Veeam server.

Speaking logically however this is completely counter intuitive. The first thing I would want to do with both vCenter and Veeam server is complete backups AND replicas, as the process for resolving a vcenter rebuild is essentially building Veeam from the ground up again as well.

Yes Im aware of the migration tool, however this has let me down on a number of occasions to the point I have no confidence in it any more. That combined with the disclaimers applied to it doesnt inspire confidence. Maybe when this is a mature production class product with a GUI Ill revisit it. but I digress.

The issue with Veeam Replicas relates to BIOS UUID being identical across the Veeam server and replica. My point is that one of these VMs is powered on, the other isnt, so from a coding perspective, this shouldnt be a problem resolving, which would then resolve the "where to hot-add" dilemma, and the resulting "cant replicate Veeam" issue.

From a network manager/business perspective, I would want to minimise my effort to a couple of clicks (Veeam replica) in any scenario where Veeam requires a recovery for any reason. The current alternative of having to jump through the hoops of create a new VM, install OS, Install product, restore configuration backup, rescan everything, etc is about as unhelpful as it comes Im afraid.

Im currently reviewing the process using extract, however even this is far from streamlined, requiring adding another VM access to the Veeam repository in order to restore from veeam backups (which even then can only restore from the last full backup (bad news with recommended forward incrementals...) and once restored Im left with VM flat files which then need uploading to a vCenter datastore and registering before being able to stand up a Veeam server from backup. (But you recommend we dont back this up!)

As you can see, Im struggling with how this can be a recommended solution when a simple replica would resolve all of this.
Its just crazy that Veeam B&R isnt the recommended solution to backup & replicate the Veeam server. Even a crash consistent backup / replica would be perfectly acceptable!

Please note that this is not a Veeam bashing post, Im a HUGE Veeam advocate who recommends Veeam as the new saviour to anyone who will listen.

Cheers
Owen

[Andreas Neufert]

Interesting, I was not aware that we have this statement. We used Veeam Backup Jobs and Veeam Replication Jobs since years in the field.
And it is as well described in the Veeam best practices. https://bp.veeam.expert/resource_planni ... ethod.html

Will discuss internally and report back.

Regarding the failover to NBD, the correct KB is https://www.veeam.com/kb2159
It describes that VMware added for security reasons a step that do not allow you to process hotadd of 2 VMs with same Bios UUID in same vcenter cluster.
They had added it for a reason. Veeam automatically added this with the implementation of the VMware Software Development Kit VDDK at vSphere 6.0 times.

You can use NBD mode, which should be fine anyway as the B&R server (or Proxy Server) change rate is pretty low.
Or you can add the target ESXi host by IP a second time to Veeam and use it as replication target. And the last thing you can do is to use a second vcenter.

[owen.wright]

Thanks for the response, thats useful to know.

I can understand the logic in the change per se, however I think this could be completed with more finesse which would allow replicas to complete without the issues, such as an additional test for power on state (or some additional internal configuration flag which identifies this as a replica)? After all, pointless hot-adding a disk to a powered off Veeam Server (replica) as its not going to do much?

From your KB reference, this mentions the workaround is to make the BIOS UUID keys unique.
If Im replicating the Veeam server once a week for example, each replica run will result in resetting the BIOS UUID to identical which would reproduce this issue.

I Guess I could replicate Veeam once and ensure config backups are written out to the SAN backup location, then tweak the BIOS UUID, which should allow jobs to complete with hot-add and still allow replica to exist for when needed.
The problem with this is our customers also like to "learn on production" which makes for interesting conversations when investigating / resolving broken jobs / sites, but that is for me to deal with

Thanks again.
Owen

[Andreas Neufert]

I have an update. You will see some changes in the documentation soon.

The problem is that customers ran into bad situations when they didn´t performed the failover correctly.

We will do several things to make this step more convenient and reliable by documenting the correct way and we will discuss as well some change under the hood.

[owen.wright]

Thats great news.
Can you post an update when this goes live?
Soon is a very vague - moving target

Ill keep my eyes open tho.
Thanks Andreas

[Andreas Neufert]

I think a less restrict statement is already online. We are waiting until QA has free ressources to discuss after U3 release. Then we will work on an documentation update that reflect a recommended way. No planned release date, sorry.

[owen.wright]

Hi,

Is there any update on this guys?

Cheers

[csydas]

Yeah, some clarity would be nice. I was under the impression (and saw frequently first hand) tons of issues with backing up/replicating the Veeam server; the snapshots just wreaked havoc on backups.

Can we get a clear statement on this? The User Guide now says "it's not recommended", but I could have sworn it used to say "must not".

[Andreas Neufert]

Hi everybody.
Thanks for the patience. We have discussed the situation with our support and QA.

For the actual version you can backup and replicate the B&R ( and B&R Database Server) during times when no other backup is active processing data. The database system need enough RAM/CPU resources so that the VSS processing is fast and do not cause any issues.
If you can not process the B&R Server processing still fails under this conditions, please upload logs to a new support case and let me know the logs here together with the Job and B&R&DB VM name.

When you failover an B&R Server to other side with a database change (to older state), you manually need to scan for new backups on the repositories and perform an host scan. This is needed in case there were changes after the B&R Replication/Backups. If you do not perform this, you can run into some issues, which leaded to the previous change of the statement (discussed above).

We will implement some product changes that automates this for you. Basically detect that the B&R/Database was restored/failed over and take actions based on this.

In general everybody should use and focus on the official configuration backup and restore tool integrated with the product.