-
- Service Provider
- Posts: 15
- Liked: never
- Joined: Apr 13, 2017 10:28 am
- Full Name: Owen Wright
- Contact:
Feature Request - Create Veeam Replica
Hi guys
Ive recently had issues with customer sites creating veeam replicas of the veeam VM. This has resulted in some issues where backups using hot-add work perfectly, and most replicas using hot-add complete perfectly, however the replica job for the Veeam server fails over to NBD. I get that this is a known issue (https://www.veeam.com/kb2008 , issue number after logging a ticket with support (02387488).
According to your help file (https://helpcenter.veeam.com/docs/backu ... tml?ver=95) its not recommended to backup / replicate the Veeam server.
Speaking logically however this is completely counter intuitive. The first thing I would want to do with both vCenter and Veeam server is complete backups AND replicas, as the process for resolving a vcenter rebuild is essentially building Veeam from the ground up again as well.
Yes Im aware of the migration tool, however this has let me down on a number of occasions to the point I have no confidence in it any more. That combined with the disclaimers applied to it doesnt inspire confidence. Maybe when this is a mature production class product with a GUI Ill revisit it. but I digress.
The issue with Veeam Replicas relates to BIOS UUID being identical across the Veeam server and replica. My point is that one of these VMs is powered on, the other isnt, so from a coding perspective, this shouldnt be a problem resolving, which would then resolve the "where to hot-add" dilemma, and the resulting "cant replicate Veeam" issue.
From a network manager/business perspective, I would want to minimise my effort to a couple of clicks (Veeam replica) in any scenario where Veeam requires a recovery for any reason. The current alternative of having to jump through the hoops of create a new VM, install OS, Install product, restore configuration backup, rescan everything, etc is about as unhelpful as it comes Im afraid.
Im currently reviewing the process using extract, however even this is far from streamlined, requiring adding another VM access to the Veeam repository in order to restore from veeam backups (which even then can only restore from the last full backup (bad news with recommended forward incrementals...) and once restored Im left with VM flat files which then need uploading to a vCenter datastore and registering before being able to stand up a Veeam server from backup. (But you recommend we dont back this up!)
As you can see, Im struggling with how this can be a recommended solution when a simple replica would resolve all of this.
Its just crazy that Veeam B&R isnt the recommended solution to backup & replicate the Veeam server. Even a crash consistent backup / replica would be perfectly acceptable!
Please note that this is not a Veeam bashing post, Im a HUGE Veeam advocate who recommends Veeam as the new saviour to anyone who will listen.
Cheers
Owen
Ive recently had issues with customer sites creating veeam replicas of the veeam VM. This has resulted in some issues where backups using hot-add work perfectly, and most replicas using hot-add complete perfectly, however the replica job for the Veeam server fails over to NBD. I get that this is a known issue (https://www.veeam.com/kb2008 , issue number after logging a ticket with support (02387488).
According to your help file (https://helpcenter.veeam.com/docs/backu ... tml?ver=95) its not recommended to backup / replicate the Veeam server.
Speaking logically however this is completely counter intuitive. The first thing I would want to do with both vCenter and Veeam server is complete backups AND replicas, as the process for resolving a vcenter rebuild is essentially building Veeam from the ground up again as well.
Yes Im aware of the migration tool, however this has let me down on a number of occasions to the point I have no confidence in it any more. That combined with the disclaimers applied to it doesnt inspire confidence. Maybe when this is a mature production class product with a GUI Ill revisit it. but I digress.
The issue with Veeam Replicas relates to BIOS UUID being identical across the Veeam server and replica. My point is that one of these VMs is powered on, the other isnt, so from a coding perspective, this shouldnt be a problem resolving, which would then resolve the "where to hot-add" dilemma, and the resulting "cant replicate Veeam" issue.
From a network manager/business perspective, I would want to minimise my effort to a couple of clicks (Veeam replica) in any scenario where Veeam requires a recovery for any reason. The current alternative of having to jump through the hoops of create a new VM, install OS, Install product, restore configuration backup, rescan everything, etc is about as unhelpful as it comes Im afraid.
Im currently reviewing the process using extract, however even this is far from streamlined, requiring adding another VM access to the Veeam repository in order to restore from veeam backups (which even then can only restore from the last full backup (bad news with recommended forward incrementals...) and once restored Im left with VM flat files which then need uploading to a vCenter datastore and registering before being able to stand up a Veeam server from backup. (But you recommend we dont back this up!)
As you can see, Im struggling with how this can be a recommended solution when a simple replica would resolve all of this.
Its just crazy that Veeam B&R isnt the recommended solution to backup & replicate the Veeam server. Even a crash consistent backup / replica would be perfectly acceptable!
Please note that this is not a Veeam bashing post, Im a HUGE Veeam advocate who recommends Veeam as the new saviour to anyone who will listen.
Cheers
Owen
-
- VP, Product Management
- Posts: 7076
- Liked: 1510 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Feature Request - Create Veeam Replica
Interesting, I was not aware that we have this statement. We used Veeam Backup Jobs and Veeam Replication Jobs since years in the field.
And it is as well described in the Veeam best practices. https://bp.veeam.expert/resource_planni ... ethod.html
Will discuss internally and report back.
Regarding the failover to NBD, the correct KB is https://www.veeam.com/kb2159
It describes that VMware added for security reasons a step that do not allow you to process hotadd of 2 VMs with same Bios UUID in same vcenter cluster.
They had added it for a reason. Veeam automatically added this with the implementation of the VMware Software Development Kit VDDK at vSphere 6.0 times.
You can use NBD mode, which should be fine anyway as the B&R server (or Proxy Server) change rate is pretty low.
Or you can add the target ESXi host by IP a second time to Veeam and use it as replication target. And the last thing you can do is to use a second vcenter.
And it is as well described in the Veeam best practices. https://bp.veeam.expert/resource_planni ... ethod.html
Will discuss internally and report back.
Regarding the failover to NBD, the correct KB is https://www.veeam.com/kb2159
It describes that VMware added for security reasons a step that do not allow you to process hotadd of 2 VMs with same Bios UUID in same vcenter cluster.
They had added it for a reason. Veeam automatically added this with the implementation of the VMware Software Development Kit VDDK at vSphere 6.0 times.
You can use NBD mode, which should be fine anyway as the B&R server (or Proxy Server) change rate is pretty low.
Or you can add the target ESXi host by IP a second time to Veeam and use it as replication target. And the last thing you can do is to use a second vcenter.
-
- Service Provider
- Posts: 15
- Liked: never
- Joined: Apr 13, 2017 10:28 am
- Full Name: Owen Wright
- Contact:
Re: Feature Request - Create Veeam Replica
Thanks for the response, thats useful to know.
I can understand the logic in the change per se, however I think this could be completed with more finesse which would allow replicas to complete without the issues, such as an additional test for power on state (or some additional internal configuration flag which identifies this as a replica)? After all, pointless hot-adding a disk to a powered off Veeam Server (replica) as its not going to do much?
From your KB reference, this mentions the workaround is to make the BIOS UUID keys unique.
If Im replicating the Veeam server once a week for example, each replica run will result in resetting the BIOS UUID to identical which would reproduce this issue.
I Guess I could replicate Veeam once and ensure config backups are written out to the SAN backup location, then tweak the BIOS UUID, which should allow jobs to complete with hot-add and still allow replica to exist for when needed.
The problem with this is our customers also like to "learn on production" which makes for interesting conversations when investigating / resolving broken jobs / sites, but that is for me to deal with
Thanks again.
Owen
I can understand the logic in the change per se, however I think this could be completed with more finesse which would allow replicas to complete without the issues, such as an additional test for power on state (or some additional internal configuration flag which identifies this as a replica)? After all, pointless hot-adding a disk to a powered off Veeam Server (replica) as its not going to do much?
From your KB reference, this mentions the workaround is to make the BIOS UUID keys unique.
If Im replicating the Veeam server once a week for example, each replica run will result in resetting the BIOS UUID to identical which would reproduce this issue.
I Guess I could replicate Veeam once and ensure config backups are written out to the SAN backup location, then tweak the BIOS UUID, which should allow jobs to complete with hot-add and still allow replica to exist for when needed.
The problem with this is our customers also like to "learn on production" which makes for interesting conversations when investigating / resolving broken jobs / sites, but that is for me to deal with
Thanks again.
Owen
-
- VP, Product Management
- Posts: 7076
- Liked: 1510 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Feature Request - Create Veeam Replica
I have an update. You will see some changes in the documentation soon.
The problem is that customers ran into bad situations when they didn´t performed the failover correctly.
We will do several things to make this step more convenient and reliable by documenting the correct way and we will discuss as well some change under the hood.
The problem is that customers ran into bad situations when they didn´t performed the failover correctly.
We will do several things to make this step more convenient and reliable by documenting the correct way and we will discuss as well some change under the hood.
-
- Service Provider
- Posts: 15
- Liked: never
- Joined: Apr 13, 2017 10:28 am
- Full Name: Owen Wright
- Contact:
Re: Feature Request - Create Veeam Replica
Thats great news.
Can you post an update when this goes live?
Soon is a very vague - moving target
Ill keep my eyes open tho.
Thanks Andreas
Can you post an update when this goes live?
Soon is a very vague - moving target
Ill keep my eyes open tho.
Thanks Andreas
-
- VP, Product Management
- Posts: 7076
- Liked: 1510 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Feature Request - Create Veeam Replica
I think a less restrict statement is already online. We are waiting until QA has free ressources to discuss after U3 release. Then we will work on an documentation update that reflect a recommended way. No planned release date, sorry.
-
- Service Provider
- Posts: 15
- Liked: never
- Joined: Apr 13, 2017 10:28 am
- Full Name: Owen Wright
- Contact:
Re: Feature Request - Create Veeam Replica
Hi,
Is there any update on this guys?
Cheers
Is there any update on this guys?
Cheers
-
- Expert
- Posts: 193
- Liked: 47 times
- Joined: Jan 16, 2018 5:14 pm
- Full Name: Harvey Carel
- Contact:
Re: Feature Request - Create Veeam Replica
Yeah, some clarity would be nice. I was under the impression (and saw frequently first hand) tons of issues with backing up/replicating the Veeam server; the snapshots just wreaked havoc on backups.
Can we get a clear statement on this? The User Guide now says "it's not recommended", but I could have sworn it used to say "must not".
Can we get a clear statement on this? The User Guide now says "it's not recommended", but I could have sworn it used to say "must not".
-
- VP, Product Management
- Posts: 7076
- Liked: 1510 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Feature Request - Create Veeam Replica
Hi everybody.
Thanks for the patience. We have discussed the situation with our support and QA.
For the actual version you can backup and replicate the B&R ( and B&R Database Server) during times when no other backup is active processing data. The database system need enough RAM/CPU resources so that the VSS processing is fast and do not cause any issues.
If you can not process the B&R Server processing still fails under this conditions, please upload logs to a new support case and let me know the logs here together with the Job and B&R&DB VM name.
When you failover an B&R Server to other side with a database change (to older state), you manually need to scan for new backups on the repositories and perform an host scan. This is needed in case there were changes after the B&R Replication/Backups. If you do not perform this, you can run into some issues, which leaded to the previous change of the statement (discussed above).
We will implement some product changes that automates this for you. Basically detect that the B&R/Database was restored/failed over and take actions based on this.
In general everybody should use and focus on the official configuration backup and restore tool integrated with the product.
Thanks for the patience. We have discussed the situation with our support and QA.
For the actual version you can backup and replicate the B&R ( and B&R Database Server) during times when no other backup is active processing data. The database system need enough RAM/CPU resources so that the VSS processing is fast and do not cause any issues.
If you can not process the B&R Server processing still fails under this conditions, please upload logs to a new support case and let me know the logs here together with the Job and B&R&DB VM name.
When you failover an B&R Server to other side with a database change (to older state), you manually need to scan for new backups on the repositories and perform an host scan. This is needed in case there were changes after the B&R Replication/Backups. If you do not perform this, you can run into some issues, which leaded to the previous change of the statement (discussed above).
We will implement some product changes that automates this for you. Basically detect that the B&R/Database was restored/failed over and take actions based on this.
In general everybody should use and focus on the official configuration backup and restore tool integrated with the product.
Who is online
Users browsing this forum: Google [Bot] and 59 guests