-
- Influencer
- Posts: 18
- Liked: 1 time
- Joined: Aug 26, 2014 10:10 am
- Full Name: James Serjeant
- Contact:
vCenter recovery - using Veeam "NOT recommended"
I had issues recovering 2 vCenter VMs across two different sites and opened ticket # 01920319 to investigate why the Restores failed (one failure was an Instant Recovery during 'migrate to production' and the other was a Entire VM restore)
I now understand why IR would fail during 'migrate to production' (although others have apparently managed to restore vCenter using IR successfully). The second restore is still under investigation.
However, Veeam have now advised to follow this VMware KB instead of using Veeam !!
https://kb.vmware.com/selfservice/micro ... Id=1023985
My understanding was Veeam could be used to back up and restore vCenter directly via the Host and as far as I'm aware this was fully supported - so what changed and when did this change occur?
Any insight appreciated.
I now understand why IR would fail during 'migrate to production' (although others have apparently managed to restore vCenter using IR successfully). The second restore is still under investigation.
However, Veeam have now advised to follow this VMware KB instead of using Veeam !!
https://kb.vmware.com/selfservice/micro ... Id=1023985
My understanding was Veeam could be used to back up and restore vCenter directly via the Host and as far as I'm aware this was fully supported - so what changed and when did this change occur?
Any insight appreciated.
-
- Veeam Software
- Posts: 21073
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Some kind of misunderstanding could take place here. Generally, Veeam B&R can be used to back up/restore vCenter Server and it is fully supported. However, in case vCenter uses vPostgres database, the recommended way of getting its consistent backup and restore is described in this VMware article.
-
- Veteran
- Posts: 370
- Liked: 97 times
- Joined: Dec 13, 2015 11:33 pm
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
This is my primary reason for still having vSphere on Windows. To properly backup the appliance you have to shutdown vsphere to get a consistent DB backup, and if you script that to happen automatically when Veeam runs the job, well you're shutting down vsphere, which Veeam needs, while it's doing a backup.... I can only imagine that would fail for every job currently in progress
-
- Influencer
- Posts: 18
- Liked: 1 time
- Joined: Aug 26, 2014 10:10 am
- Full Name: James Serjeant
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
We use SQL, not vPostgres and the KB article provided by Veeam is for SQL and not foggy's vPostgres KB.
Now I'm even more confused.. should we use Veeam to protect vCenter or not?
Now I'm even more confused.. should we use Veeam to protect vCenter or not?
-
- Veteran
- Posts: 487
- Liked: 106 times
- Joined: Dec 08, 2014 2:58 pm
- Full Name: Steve Krause
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Interesting, I have been backing up my 5.5 VCSAs (postgres) for a couple years and have done a couple of restores in that time and did not have issues. I guess I will have to look into this more.
I think I would rather rebuild the VCSAs from scratch than go back to using Windows for my vCenter though.
I think I would rather rebuild the VCSAs from scratch than go back to using Windows for my vCenter though.
Steve Krause
Veeam Certified Architect
Veeam Certified Architect
-
- Veteran
- Posts: 354
- Liked: 72 times
- Joined: Jun 30, 2015 6:06 pm
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
So for v6 VCSA (w/ external PSC's) do we need to be using VMware's solution or Veeam B&R? What happens when we come in tomorrow morning and our vCenters are trashed?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
-
- Veeam Software
- Posts: 21073
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
The support case correspondence caused some sort of confusion regarding the ability to backup vCenter server using Veeam B&R. Basically, you can feel safe with backing up SQL-based vCenter, since it's database is automatically excluded from VSS processing (i.e. backed up crash-consistent, which is generally ok for a transactional database). In case of vCSA bundled with PostgreSQL database, stopping its services prior making the VM snapshot should guarantee you application-consistent backups and there's also a backup procedure provided by VMware that can be triggered as a pre-job script. I believe that James should also get some follow-up from his engineer soon.
-
- Veteran
- Posts: 371
- Liked: 32 times
- Joined: Oct 17, 2013 10:02 am
- Full Name: Mark
- Location: UK
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Hi, I would like an official statement and guide regarding the backup and restore of VCSA v6 with Postgres, as that's what many people will be using going forward.
We could all do testing in our labs, but in reality a production vcenter would have much more going on that we can produce in labs and we need to know for sure. Those of us using DVS will be particularly interested as having a solid vcenter recovery plan is essential.
According to v6 Documentation, Veeam should work without any scripts or stopping services on vcsa.
As an alternative to vSphere Data Protection, you can also use third-party products that are integrated with VMware vSphere Storage APIs - Data Protection to back up and restore a virtual machine that contains vCenter Server, a vCenter Server Appliance, or a Platform Services Controller.
http://pubs.vmware.com/vsphere-60/index ... 27ECA.html
Thanks
We could all do testing in our labs, but in reality a production vcenter would have much more going on that we can produce in labs and we need to know for sure. Those of us using DVS will be particularly interested as having a solid vcenter recovery plan is essential.
According to v6 Documentation, Veeam should work without any scripts or stopping services on vcsa.
As an alternative to vSphere Data Protection, you can also use third-party products that are integrated with VMware vSphere Storage APIs - Data Protection to back up and restore a virtual machine that contains vCenter Server, a vCenter Server Appliance, or a Platform Services Controller.
http://pubs.vmware.com/vsphere-60/index ... 27ECA.html
Thanks
-
- Veteran
- Posts: 354
- Liked: 72 times
- Joined: Jun 30, 2015 6:06 pm
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
FWIW, I did a test restore of our PSC and vCenter (to an isolated test network) and both came up working fine w/ only the exception of their IP configs were wiped. They kept DNS though. Re-entered their IP info and came up normally on their test network. Having said that, haven't done a real live test of downing our live PSC/vCenter and restoring one to see if it fully works, but will put that on list of restore testing. Doing straight forward Veeam backup job.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
-
- Veteran
- Posts: 354
- Liked: 72 times
- Joined: Jun 30, 2015 6:06 pm
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
I take that way back. They restored fine, each appears fine at their respective consoles, but my vCenter won't authenticate (not even local admin) and my PSC throws a 503 when I try to browse its web interface. So clearly, something's broken. ![Very Happy :D](https://forums.veeam.com/images/smilies/icon_biggrin.gif)
![Very Happy :D](https://forums.veeam.com/images/smilies/icon_biggrin.gif)
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
-
- Veeam Software
- Posts: 21073
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
This statement is too general and doesn't mention the database at all. Our recommendation for any application that doesn't support VSS is using pre-freeze/post-thaw scripts for consistent backups, otherwise you get them crash-consistent.lando_uk wrote:According to v6 Documentation, Veeam should work without any scripts or stopping services on vcsa.
As an alternative to vSphere Data Protection, you can also use third-party products that are integrated with VMware vSphere Storage APIs - Data Protection to back up and restore a virtual machine that contains vCenter Server, a vCenter Server Appliance, or a Platform Services Controller.
-
- Veteran
- Posts: 371
- Liked: 32 times
- Joined: Oct 17, 2013 10:02 am
- Full Name: Mark
- Location: UK
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Trying to interpret the docs, I get the impression that VCSA v6 does use VSS on backup and that no scripts are necessary.foggy wrote: This statement is too general and doesn't mention the database at all. Our recommendation for any application that doesn't support VSS is using pre-freeze/post-thaw scripts for consistent backups, otherwise you get them crash-consistent.
-
- Veteran
- Posts: 371
- Liked: 32 times
- Joined: Oct 17, 2013 10:02 am
- Full Name: Mark
- Location: UK
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
I was told previously by VMware support that you cant change the IP of VCSA v6. A restore/replication has to be on the same network as the original, no IP config change is supported.rreed wrote:I take that way back. They restored fine, each appears fine at their respective consoles, but my vCenter won't authenticate (not even local admin) and my PSC throws a 503 when I try to browse its web interface. So clearly, something's broken.
-
- Veteran
- Posts: 354
- Liked: 72 times
- Joined: Jun 30, 2015 6:06 pm
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
I didn't change anything about the VM. Same IP. A different vswitch but everything else unchanged. I haven't had time to go and fight w/ it any, I'll report back when I do.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
-
- Veeam Software
- Posts: 21073
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
That confuses me, since vCSA is a Linux appliance and doesn't support VSS.lando_uk wrote:Trying to interpret the docs, I get the impression that VCSA v6 does use VSS on backup and that no scripts are necessary.
-
- Veteran
- Posts: 880
- Liked: 47 times
- Joined: Nov 05, 2009 12:24 pm
- Location: Sydney, NSW
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Yes please, I'm interested to hear if the VCSA 6.0 is supported by Veeam 9.0 or not ?rreed wrote:I didn't change anything about the VM. Same IP. A different vswitch but everything else unchanged. I haven't had time to go and fight w/ it any, I'll report back when I do.
--
/* Veeam software enthusiast user & supporter ! */
/* Veeam software enthusiast user & supporter ! */
-
- Product Manager
- Posts: 6431
- Liked: 729 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Yes, it is supported. As foggy's already mentioned you can also use Veeam for VCSA 6.0 backups and restores, however one has to keep in mind that VCSA uses vPostgres database thus requires some additional steps to be performed in order to make a consistent backup.
Thank you.
Thank you.
-
- Veteran
- Posts: 371
- Liked: 32 times
- Joined: Oct 17, 2013 10:02 am
- Full Name: Mark
- Location: UK
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Hi Foggy,
I'm sorry to keep going on but your VMware KB link says "The backup of the vPostgres database is not required when performing a backup using a supported method. "
So is a standard Veeam v9 backup a `supported method` or not?
Also VMware says "This article is only supported for backup and restore of the vPostgres database to the same vCenter Server Appliance. Use of image-based backup and restore is the only solution supported for performing a full, secondary appliance restore"
So this is saying that if you're restoring for DR, image based backup is the way to go.
You understand that we don't want to and shouldn't need to (according to VMware) run any post backup scripts or stop vcsa for a constant backup.
Thanks
I'm sorry to keep going on but your VMware KB link says "The backup of the vPostgres database is not required when performing a backup using a supported method. "
So is a standard Veeam v9 backup a `supported method` or not?
Also VMware says "This article is only supported for backup and restore of the vPostgres database to the same vCenter Server Appliance. Use of image-based backup and restore is the only solution supported for performing a full, secondary appliance restore"
So this is saying that if you're restoring for DR, image based backup is the way to go.
You understand that we don't want to and shouldn't need to (according to VMware) run any post backup scripts or stop vcsa for a constant backup.
Thanks
-
- Expert
- Posts: 170
- Liked: 29 times
- Joined: Apr 28, 2015 7:18 am
- Full Name: Patrick
- Location: Germany
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
vSphere 6.5 has just been announced at VMworld Europe with new backup capabilities!
https://blogs.vmware.com/vsphere/2016/1 ... e-6-5.html
https://blogs.vmware.com/vsphere/2016/1 ... e-6-5.html
-
- Product Manager
- Posts: 6431
- Liked: 729 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
According to VMware documentation it is supported only if the following requirements are met:So is a standard Veeam v9 backup a `supported method` or not?
- it is full backup (no increments)
- VMware tools are installed
- VM uses FQDN with correct DNS resolution, or is with a static IP
- no snapshots
- no FT
All other cases are unsupported and might require extra steps to ensure backup consistency.
Also,
Thanks
-
- Veteran
- Posts: 354
- Liked: 72 times
- Joined: Jun 30, 2015 6:06 pm
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Veeam uses snapshots to do its backups, doesn't it?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
-
- VP, Product Management
- Posts: 27120
- Liked: 2721 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Yes, that's correct, but what Pavel tried to say "VMs that have existing snapshots", nothing to do with the way a backup job is done.
-
- Influencer
- Posts: 18
- Liked: 1 time
- Joined: Aug 26, 2014 10:10 am
- Full Name: James Serjeant
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Quick Update:
I ran into the same issue again this weekend when having to restore the same vCenter again (due to a failed in place upgrade). "Entire VM" restore kept failing and in order to recover the VM in the end I had to perform a "Restore VM files" job instead. New Case # 01969511 has been opened and support bundle has been Ftp'd.
Whilst troubleshooting the Restore failures I ruled out the following:
- Tried different transport modes on the Veeam Proxy
- Tried different credentials
- Tried different Veeam proxies, including a new Proxy
- Tried restoring to different hosts
- Made sure the Veeam proxy was on the same host that I was restoring vCenter to
- There were no orphaned HotAdd disks on the affected VM or any of the proxies.
- AutoMount was disabled in the Veeam VM Guest OS.
- Hardware versions were OK
- iSCSI controller had less than 15 disks attached
- The Veeam VM had been restarted in order to clear any possible locks
- The host was not a Standalone server
- Same cluster
- The VBKs were ‘full active’ and contained no snapshots
In order to have any confidence that vCenter is still fully protected I obviously need to understand why the latest Restores failed - so we'll see what Support come back with.
I ran into the same issue again this weekend when having to restore the same vCenter again (due to a failed in place upgrade). "Entire VM" restore kept failing and in order to recover the VM in the end I had to perform a "Restore VM files" job instead. New Case # 01969511 has been opened and support bundle has been Ftp'd.
Whilst troubleshooting the Restore failures I ruled out the following:
- Tried different transport modes on the Veeam Proxy
- Tried different credentials
- Tried different Veeam proxies, including a new Proxy
- Tried restoring to different hosts
- Made sure the Veeam proxy was on the same host that I was restoring vCenter to
- There were no orphaned HotAdd disks on the affected VM or any of the proxies.
- AutoMount was disabled in the Veeam VM Guest OS.
- Hardware versions were OK
- iSCSI controller had less than 15 disks attached
- The Veeam VM had been restarted in order to clear any possible locks
- The host was not a Standalone server
- Same cluster
- The VBKs were ‘full active’ and contained no snapshots
In order to have any confidence that vCenter is still fully protected I obviously need to understand why the latest Restores failed - so we'll see what Support come back with.
-
- Veteran
- Posts: 354
- Liked: 72 times
- Joined: Jun 30, 2015 6:06 pm
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
In further testing and closer scrutiny, ours (VC and PSC both) seem to restore fine to a test environment, albeit we have not yet tried to do a live restore to make sure it connects to hosts, talks on the network, etc. properly. We will find a later quiet time that we can power down our live VCSA's and do a live test.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
-
- VP, Product Management
- Posts: 6013
- Liked: 2843 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Just want to say that I've restored probably 100+ vCenter servers over the past 5 years (a lot of these were for a handful of customers testing this specific DR scenario), in both test and production customer environments, and I've never had an issue with not being able to recover the vCenter assuming the backup was properly made (i.e. included the database).
I did have one very complex issue with vSphere 5.5 because the inventory database itself was outside of the SQL database. We had to stop the inventory service and use file level recovery to restore some older backups, but we were finally able to get the inventory database to start and run properly. This was important because, while it's completely possible to create a fresh inventory database, in vSphere 5.5 all of the tag information was stored there, and this customer made heavy use of tags. I'm pretty sure in vSphere 6 that tags information is now in the main vCenter database.
Regardless, restoring vCenter is a "special" event. While you can backup vCenter with Veeam using the vCenter connection, it is obviously impossible to restore using this method since, to restore vCenter you must power vCenter off, but it is obviously impossible to communicate with a powered off vCenter, so it creates a significant chicken-and-the-egg problem.
The solution is to determine which specific host the vCenter VM is currently registered against, temporarily add that host to the Veeam console via IP address as a standalone ESXi host, and then chose and advanced restore and restore either the VM disks or entire VM, mapping the resources to the standlone hosts. The reason you have to add the host by IP address, rather than by name, is because the Veeam console will not allow you to have the same host added twice, once via vCenter and again by direct host, so adding the host by IP address is a little bit of a trick.
I did have one very complex issue with vSphere 5.5 because the inventory database itself was outside of the SQL database. We had to stop the inventory service and use file level recovery to restore some older backups, but we were finally able to get the inventory database to start and run properly. This was important because, while it's completely possible to create a fresh inventory database, in vSphere 5.5 all of the tag information was stored there, and this customer made heavy use of tags. I'm pretty sure in vSphere 6 that tags information is now in the main vCenter database.
Regardless, restoring vCenter is a "special" event. While you can backup vCenter with Veeam using the vCenter connection, it is obviously impossible to restore using this method since, to restore vCenter you must power vCenter off, but it is obviously impossible to communicate with a powered off vCenter, so it creates a significant chicken-and-the-egg problem.
The solution is to determine which specific host the vCenter VM is currently registered against, temporarily add that host to the Veeam console via IP address as a standalone ESXi host, and then chose and advanced restore and restore either the VM disks or entire VM, mapping the resources to the standlone hosts. The reason you have to add the host by IP address, rather than by name, is because the Veeam console will not allow you to have the same host added twice, once via vCenter and again by direct host, so adding the host by IP address is a little bit of a trick.
-
- Veteran
- Posts: 880
- Liked: 47 times
- Joined: Nov 05, 2009 12:24 pm
- Location: Sydney, NSW
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Does this means that the new VMware VCenter Standard Appliance version 6.0 and above backup is not supported by Veeam 9.0 yet ?foggy wrote: That confuses me, since vCSA is a Linux appliance and doesn't support VSS.
--
/* Veeam software enthusiast user & supporter ! */
/* Veeam software enthusiast user & supporter ! */
-
- Veeam Software
- Posts: 21073
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
No, this means that you need to use pre-freeze/post-thaw scripts to get a consistent backup of Linux VMs (or go crash-consistent otherwise).
-
- Veteran
- Posts: 880
- Liked: 47 times
- Joined: Nov 05, 2009 12:24 pm
- Location: Sydney, NSW
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Thanks Alex for the clarification, so where can I find the script for that pre-freeze/post-thaw scripts ?
Does this means that during the backup window, there might be some outage since the script kicks in![Rolling Eyes :roll:](https://forums.veeam.com/images/smilies/icon_rolleyes.gif)
Does this means that during the backup window, there might be some outage since the script kicks in
![Rolling Eyes :roll:](https://forums.veeam.com/images/smilies/icon_rolleyes.gif)
--
/* Veeam software enthusiast user & supporter ! */
/* Veeam software enthusiast user & supporter ! */
-
- Veeam Software
- Posts: 21073
- Liked: 2115 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Should be some generic script to prepare database for the backup.albertwt wrote:Thanks Alex for the clarification, so where can I find the script for that pre-freeze/post-thaw scripts ?
This implies a short period of time when the database is stopped.albertwt wrote:Does this means that during the backup window, there might be some outage since the script kicks in
-
- VeeaMVP
- Posts: 6139
- Liked: 1932 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: vCenter recovery - using Veeam "NOT recommended"
Actually, Postgres has continous log save, so any transaction is simply re-played upon restart if it is missing. This means that there is no real need to put the filesystem or the database in a consistent state. Per postgres documentation:
“We do not need a perfectly consistent file system backup as the starting point. Any internal inconsistency in the backup will be corrected by log replay (this is not significantly different from what happens during crash recovery). So we do not need a file system snapshot capability, just tar or a similar archiving tool.”
“We do not need a perfectly consistent file system backup as the starting point. Any internal inconsistency in the backup will be corrected by log replay (this is not significantly different from what happens during crash recovery). So we do not need a file system snapshot capability, just tar or a similar archiving tool.”
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Who is online
Users browsing this forum: Bing [Bot], msidnam, NightBird and 92 guests