Discussions specific to the VMware vSphere hypervisor
orangep7
Influencer
Posts: 18
Liked: 1 time
Joined: Aug 26, 2014 10:10 am
Full Name: James Serjeant
Contact:

vCenter recovery - using Veeam "NOT recommended"

Post by orangep7 » Oct 04, 2016 2:17 pm 1 person likes this post

I had issues recovering 2 vCenter VMs across two different sites and opened ticket # 01920319 to investigate why the Restores failed (one failure was an Instant Recovery during 'migrate to production' and the other was a Entire VM restore)

I now understand why IR would fail during 'migrate to production' (although others have apparently managed to restore vCenter using IR successfully). The second restore is still under investigation.

However, Veeam have now advised to follow this VMware KB instead of using Veeam !!

https://kb.vmware.com/selfservice/micro ... Id=1023985

My understanding was Veeam could be used to back up and restore vCenter directly via the Host and as far as I'm aware this was fully supported - so what changed and when did this change occur?

Any insight appreciated.

foggy
Veeam Software
Posts: 18295
Liked: 1569 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by foggy » Oct 04, 2016 4:09 pm

Some kind of misunderstanding could take place here. Generally, Veeam B&R can be used to back up/restore vCenter Server and it is fully supported. However, in case vCenter uses vPostgres database, the recommended way of getting its consistent backup and restore is described in this VMware article.

DaveWatkins
Expert
Posts: 349
Liked: 93 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by DaveWatkins » Oct 05, 2016 12:38 am 1 person likes this post

This is my primary reason for still having vSphere on Windows. To properly backup the appliance you have to shutdown vsphere to get a consistent DB backup, and if you script that to happen automatically when Veeam runs the job, well you're shutting down vsphere, which Veeam needs, while it's doing a backup.... I can only imagine that would fail for every job currently in progress

orangep7
Influencer
Posts: 18
Liked: 1 time
Joined: Aug 26, 2014 10:10 am
Full Name: James Serjeant
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by orangep7 » Oct 05, 2016 7:10 am

We use SQL, not vPostgres and the KB article provided by Veeam is for SQL and not foggy's vPostgres KB.

Now I'm even more confused.. should we use Veeam to protect vCenter or not?

skrause
Expert
Posts: 439
Liked: 91 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by skrause » Oct 05, 2016 1:21 pm

Interesting, I have been backing up my 5.5 VCSAs (postgres) for a couple years and have done a couple of restores in that time and did not have issues. I guess I will have to look into this more.

I think I would rather rebuild the VCSAs from scratch than go back to using Windows for my vCenter though.
Steve Krause
Veeam Certified Architect

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by rreed » Oct 07, 2016 8:01 pm

So for v6 VCSA (w/ external PSC's) do we need to be using VMware's solution or Veeam B&R? What happens when we come in tomorrow morning and our vCenters are trashed?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

foggy
Veeam Software
Posts: 18295
Liked: 1569 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by foggy » Oct 10, 2016 3:54 pm

The support case correspondence caused some sort of confusion regarding the ability to backup vCenter server using Veeam B&R. Basically, you can feel safe with backing up SQL-based vCenter, since it's database is automatically excluded from VSS processing (i.e. backed up crash-consistent, which is generally ok for a transactional database). In case of vCSA bundled with PostgreSQL database, stopping its services prior making the VM snapshot should guarantee you application-consistent backups and there's also a backup procedure provided by VMware that can be triggered as a pre-job script. I believe that James should also get some follow-up from his engineer soon.

lando_uk
Expert
Posts: 311
Liked: 26 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by lando_uk » Oct 12, 2016 11:43 am 2 people like this post

Hi, I would like an official statement and guide regarding the backup and restore of VCSA v6 with Postgres, as that's what many people will be using going forward.

We could all do testing in our labs, but in reality a production vcenter would have much more going on that we can produce in labs and we need to know for sure. Those of us using DVS will be particularly interested as having a solid vcenter recovery plan is essential.

According to v6 Documentation, Veeam should work without any scripts or stopping services on vcsa.

As an alternative to vSphere Data Protection, you can also use third-party products that are integrated with VMware vSphere Storage APIs - Data Protection to back up and restore a virtual machine that contains vCenter Server, a vCenter Server Appliance, or a Platform Services Controller.

http://pubs.vmware.com/vsphere-60/index ... 27ECA.html

Thanks

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by rreed » Oct 12, 2016 6:47 pm

FWIW, I did a test restore of our PSC and vCenter (to an isolated test network) and both came up working fine w/ only the exception of their IP configs were wiped. They kept DNS though. Re-entered their IP info and came up normally on their test network. Having said that, haven't done a real live test of downing our live PSC/vCenter and restoring one to see if it fully works, but will put that on list of restore testing. Doing straight forward Veeam backup job.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by rreed » Oct 12, 2016 9:03 pm

I take that way back. They restored fine, each appears fine at their respective consoles, but my vCenter won't authenticate (not even local admin) and my PSC throws a 503 when I try to browse its web interface. So clearly, something's broken. :D
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

foggy
Veeam Software
Posts: 18295
Liked: 1569 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by foggy » Oct 14, 2016 2:43 pm

lando_uk wrote:According to v6 Documentation, Veeam should work without any scripts or stopping services on vcsa.

As an alternative to vSphere Data Protection, you can also use third-party products that are integrated with VMware vSphere Storage APIs - Data Protection to back up and restore a virtual machine that contains vCenter Server, a vCenter Server Appliance, or a Platform Services Controller.
This statement is too general and doesn't mention the database at all. Our recommendation for any application that doesn't support VSS is using pre-freeze/post-thaw scripts for consistent backups, otherwise you get them crash-consistent.

lando_uk
Expert
Posts: 311
Liked: 26 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by lando_uk » Oct 17, 2016 11:26 am

foggy wrote: This statement is too general and doesn't mention the database at all. Our recommendation for any application that doesn't support VSS is using pre-freeze/post-thaw scripts for consistent backups, otherwise you get them crash-consistent.
Trying to interpret the docs, I get the impression that VCSA v6 does use VSS on backup and that no scripts are necessary.

lando_uk
Expert
Posts: 311
Liked: 26 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by lando_uk » Oct 17, 2016 11:29 am

rreed wrote:I take that way back. They restored fine, each appears fine at their respective consoles, but my vCenter won't authenticate (not even local admin) and my PSC throws a 503 when I try to browse its web interface. So clearly, something's broken. :D
I was told previously by VMware support that you cant change the IP of VCSA v6. A restore/replication has to be on the same network as the original, no IP config change is supported.

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by rreed » Oct 17, 2016 1:12 pm

I didn't change anything about the VM. Same IP. A different vswitch but everything else unchanged. I haven't had time to go and fight w/ it any, I'll report back when I do.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

foggy
Veeam Software
Posts: 18295
Liked: 1569 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by foggy » Oct 17, 2016 5:01 pm 1 person likes this post

lando_uk wrote:Trying to interpret the docs, I get the impression that VCSA v6 does use VSS on backup and that no scripts are necessary.
That confuses me, since vCSA is a Linux appliance and doesn't support VSS.

albertwt
Expert
Posts: 640
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by albertwt » Oct 18, 2016 5:16 am

rreed wrote:I didn't change anything about the VM. Same IP. A different vswitch but everything else unchanged. I haven't had time to go and fight w/ it any, I'll report back when I do.
Yes please, I'm interested to hear if the VCSA 6.0 is supported by Veeam 9.0 or not ?
--
/* Veeam software enthusiast user & supporter ! */

P.Tide
Product Manager
Posts: 5267
Liked: 461 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by P.Tide » Oct 18, 2016 7:04 am

Yes, it is supported. As foggy's already mentioned you can also use Veeam for VCSA 6.0 backups and restores, however one has to keep in mind that VCSA uses vPostgres database thus requires some additional steps to be performed in order to make a consistent backup.

Thank you.

lando_uk
Expert
Posts: 311
Liked: 26 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by lando_uk » Oct 18, 2016 11:53 am

Hi Foggy,
I'm sorry to keep going on but your VMware KB link says "The backup of the vPostgres database is not required when performing a backup using a supported method. "

So is a standard Veeam v9 backup a `supported method` or not?

Also VMware says "This article is only supported for backup and restore of the vPostgres database to the same vCenter Server Appliance. Use of image-based backup and restore is the only solution supported for performing a full, secondary appliance restore"

So this is saying that if you're restoring for DR, image based backup is the way to go.

You understand that we don't want to and shouldn't need to (according to VMware) run any post backup scripts or stop vcsa for a constant backup.

Thanks

Pat490
Expert
Posts: 160
Liked: 25 times
Joined: Apr 28, 2015 7:18 am
Full Name: Patrick
Location: Germany
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by Pat490 » Oct 18, 2016 11:57 am 1 person likes this post

vSphere 6.5 has just been announced at VMworld Europe with new backup capabilities!
https://blogs.vmware.com/vsphere/2016/1 ... e-6-5.html

P.Tide
Product Manager
Posts: 5267
Liked: 461 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by P.Tide » Oct 18, 2016 12:22 pm

So is a standard Veeam v9 backup a `supported method` or not?
According to VMware documentation it is supported only if the following requirements are met:

- it is full backup (no increments)
- VMware tools are installed
- VM uses FQDN with correct DNS resolution, or is with a static IP
- no snapshots
- no FT

All other cases are unsupported and might require extra steps to ensure backup consistency.

Also,
Thanks

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by rreed » Oct 18, 2016 1:22 pm

Veeam uses snapshots to do its backups, doesn't it?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

Vitaliy S.
Product Manager
Posts: 23016
Liked: 1561 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by Vitaliy S. » Oct 18, 2016 1:56 pm

Yes, that's correct, but what Pavel tried to say "VMs that have existing snapshots", nothing to do with the way a backup job is done.

orangep7
Influencer
Posts: 18
Liked: 1 time
Joined: Aug 26, 2014 10:10 am
Full Name: James Serjeant
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by orangep7 » Nov 07, 2016 9:24 am

Quick Update:

I ran into the same issue again this weekend when having to restore the same vCenter again (due to a failed in place upgrade). "Entire VM" restore kept failing and in order to recover the VM in the end I had to perform a "Restore VM files" job instead. New Case # 01969511 has been opened and support bundle has been Ftp'd.

Whilst troubleshooting the Restore failures I ruled out the following:

- Tried different transport modes on the Veeam Proxy
- Tried different credentials
- Tried different Veeam proxies, including a new Proxy
- Tried restoring to different hosts
- Made sure the Veeam proxy was on the same host that I was restoring vCenter to
- There were no orphaned HotAdd disks on the affected VM or any of the proxies.
- AutoMount was disabled in the Veeam VM Guest OS.
- Hardware versions were OK
- iSCSI controller had less than 15 disks attached
- The Veeam VM had been restarted in order to clear any possible locks
- The host was not a Standalone server
- Same cluster
- The VBKs were ‘full active’ and contained no snapshots

In order to have any confidence that vCenter is still fully protected I obviously need to understand why the latest Restores failed - so we'll see what Support come back with.

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by rreed » Nov 07, 2016 5:52 pm

In further testing and closer scrutiny, ours (VC and PSC both) seem to restore fine to a test environment, albeit we have not yet tried to do a live restore to make sure it connects to hosts, talks on the network, etc. properly. We will find a later quiet time that we can power down our live VCSA's and do a live test.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

tsightler
VP, Product Management
Posts: 5425
Liked: 2246 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by tsightler » Nov 07, 2016 9:32 pm 3 people like this post

Just want to say that I've restored probably 100+ vCenter servers over the past 5 years (a lot of these were for a handful of customers testing this specific DR scenario), in both test and production customer environments, and I've never had an issue with not being able to recover the vCenter assuming the backup was properly made (i.e. included the database).

I did have one very complex issue with vSphere 5.5 because the inventory database itself was outside of the SQL database. We had to stop the inventory service and use file level recovery to restore some older backups, but we were finally able to get the inventory database to start and run properly. This was important because, while it's completely possible to create a fresh inventory database, in vSphere 5.5 all of the tag information was stored there, and this customer made heavy use of tags. I'm pretty sure in vSphere 6 that tags information is now in the main vCenter database.

Regardless, restoring vCenter is a "special" event. While you can backup vCenter with Veeam using the vCenter connection, it is obviously impossible to restore using this method since, to restore vCenter you must power vCenter off, but it is obviously impossible to communicate with a powered off vCenter, so it creates a significant chicken-and-the-egg problem.

The solution is to determine which specific host the vCenter VM is currently registered against, temporarily add that host to the Veeam console via IP address as a standalone ESXi host, and then chose and advanced restore and restore either the VM disks or entire VM, mapping the resources to the standlone hosts. The reason you have to add the host by IP address, rather than by name, is because the Veeam console will not allow you to have the same host added twice, once via vCenter and again by direct host, so adding the host by IP address is a little bit of a trick.

albertwt
Expert
Posts: 640
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by albertwt » Nov 10, 2016 10:12 am

foggy wrote: That confuses me, since vCSA is a Linux appliance and doesn't support VSS.
Does this means that the new VMware VCenter Standard Appliance version 6.0 and above backup is not supported by Veeam 9.0 yet ?
--
/* Veeam software enthusiast user & supporter ! */

foggy
Veeam Software
Posts: 18295
Liked: 1569 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by foggy » Nov 10, 2016 10:19 am 1 person likes this post

No, this means that you need to use pre-freeze/post-thaw scripts to get a consistent backup of Linux VMs (or go crash-consistent otherwise).

albertwt
Expert
Posts: 640
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by albertwt » Nov 10, 2016 10:25 am

Thanks Alex for the clarification, so where can I find the script for that pre-freeze/post-thaw scripts ?
Does this means that during the backup window, there might be some outage since the script kicks in :roll:
--
/* Veeam software enthusiast user & supporter ! */

foggy
Veeam Software
Posts: 18295
Liked: 1569 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by foggy » Nov 10, 2016 12:43 pm 1 person likes this post

albertwt wrote:Thanks Alex for the clarification, so where can I find the script for that pre-freeze/post-thaw scripts ?
Should be some generic script to prepare database for the backup.
albertwt wrote:Does this means that during the backup window, there might be some outage since the script kicks in :roll:
This implies a short period of time when the database is stopped.

dellock6
Veeam Software
Posts: 5734
Liked: 1627 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: vCenter recovery - using Veeam "NOT recommended"

Post by dellock6 » Nov 11, 2016 10:05 pm

Actually, Postgres has continous log save, so any transaction is simply re-played upon restart if it is missing. This means that there is no real need to put the filesystem or the database in a consistent state. Per postgres documentation:

“We do not need a perfectly consistent file system backup as the starting point. Any internal inconsistency in the backup will be corrected by log replay (this is not significantly different from what happens during crash recovery). So we do not need a file system snapshot capability, just tar or a similar archiving tool.”
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 17 guests