vCenter recovery - using Veeam "NOT recommended"

[orangep7]

I had issues recovering 2 vCenter VMs across two different sites and opened ticket # 01920319 to investigate why the Restores failed (one failure was an Instant Recovery during 'migrate to production' and the other was a Entire VM restore)

I now understand why IR would fail during 'migrate to production' (although others have apparently managed to restore vCenter using IR successfully). The second restore is still under investigation.

However, Veeam have now advised to follow this VMware KB instead of using Veeam !!

https://kb.vmware.com/selfservice/micro ... Id=1023985

My understanding was Veeam could be used to back up and restore vCenter directly via the Host and as far as I'm aware this was fully supported - so what changed and when did this change occur?

Any insight appreciated.

[foggy]

Some kind of misunderstanding could take place here. Generally, Veeam B&R can be used to back up/restore vCenter Server and it is fully supported. However, in case vCenter uses vPostgres database, the recommended way of getting its consistent backup and restore is described in this VMware article.

[DaveWatkins]

This is my primary reason for still having vSphere on Windows. To properly backup the appliance you have to shutdown vsphere to get a consistent DB backup, and if you script that to happen automatically when Veeam runs the job, well you're shutting down vsphere, which Veeam needs, while it's doing a backup.... I can only imagine that would fail for every job currently in progress

[orangep7]

We use SQL, not vPostgres and the KB article provided by Veeam is for SQL and not foggy's vPostgres KB.

Now I'm even more confused.. should we use Veeam to protect vCenter or not?

[skrause]

Interesting, I have been backing up my 5.5 VCSAs (postgres) for a couple years and have done a couple of restores in that time and did not have issues. I guess I will have to look into this more.

I think I would rather rebuild the VCSAs from scratch than go back to using Windows for my vCenter though.

[rreed]

So for v6 VCSA (w/ external PSC's) do we need to be using VMware's solution or Veeam B&R? What happens when we come in tomorrow morning and our vCenters are trashed?

[foggy]

The support case correspondence caused some sort of confusion regarding the ability to backup vCenter server using Veeam B&R. Basically, you can feel safe with backing up SQL-based vCenter, since it's database is automatically excluded from VSS processing (i.e. backed up crash-consistent, which is generally ok for a transactional database). In case of vCSA bundled with PostgreSQL database, stopping its services prior making the VM snapshot should guarantee you application-consistent backups and there's also a backup procedure provided by VMware that can be triggered as a pre-job script. I believe that James should also get some follow-up from his engineer soon.

[lando_uk]

Hi, I would like an official statement and guide regarding the backup and restore of VCSA v6 with Postgres, as that's what many people will be using going forward.

We could all do testing in our labs, but in reality a production vcenter would have much more going on that we can produce in labs and we need to know for sure. Those of us using DVS will be particularly interested as having a solid vcenter recovery plan is essential.

According to v6 Documentation, Veeam should work without any scripts or stopping services on vcsa.

As an alternative to vSphere Data Protection, you can also use third-party products that are integrated with VMware vSphere Storage APIs - Data Protection to back up and restore a virtual machine that contains vCenter Server, a vCenter Server Appliance, or a Platform Services Controller.

http://pubs.vmware.com/vsphere-60/index ... 27ECA.html

Thanks

[rreed]

FWIW, I did a test restore of our PSC and vCenter (to an isolated test network) and both came up working fine w/ only the exception of their IP configs were wiped. They kept DNS though. Re-entered their IP info and came up normally on their test network. Having said that, haven't done a real live test of downing our live PSC/vCenter and restoring one to see if it fully works, but will put that on list of restore testing. Doing straight forward Veeam backup job.

[rreed]

I take that way back. They restored fine, each appears fine at their respective consoles, but my vCenter won't authenticate (not even local admin) and my PSC throws a 503 when I try to browse its web interface. So clearly, something's broken.

[foggy]

According to v6 Documentation, Veeam should work without any scripts or stopping services on vcsa.

As an alternative to vSphere Data Protection, you can also use third-party products that are integrated with VMware vSphere Storage APIs - Data Protection to back up and restore a virtual machine that contains vCenter Server, a vCenter Server Appliance, or a Platform Services Controller.

This statement is too general and doesn't mention the database at all. Our recommendation for any application that doesn't support VSS is using pre-freeze/post-thaw scripts for consistent backups, otherwise you get them crash-consistent.

[lando_uk]

This statement is too general and doesn't mention the database at all. Our recommendation for any application that doesn't support VSS is using pre-freeze/post-thaw scripts for consistent backups, otherwise you get them crash-consistent.

Trying to interpret the docs, I get the impression that VCSA v6 does use VSS on backup and that no scripts are necessary.

[lando_uk]

I take that way back. They restored fine, each appears fine at their respective consoles, but my vCenter won't authenticate (not even local admin) and my PSC throws a 503 when I try to browse its web interface. So clearly, something's broken.

I was told previously by VMware support that you cant change the IP of VCSA v6. A restore/replication has to be on the same network as the original, no IP config change is supported.

[rreed]

I didn't change anything about the VM. Same IP. A different vswitch but everything else unchanged. I haven't had time to go and fight w/ it any, I'll report back when I do.

[foggy]

Trying to interpret the docs, I get the impression that VCSA v6 does use VSS on backup and that no scripts are necessary.

That confuses me, since vCSA is a Linux appliance and doesn't support VSS.

[albertwt]

I didn't change anything about the VM. Same IP. A different vswitch but everything else unchanged. I haven't had time to go and fight w/ it any, I'll report back when I do.

Yes please, I'm interested to hear if the VCSA 6.0 is supported by Veeam 9.0 or not ?

[PTide]

Yes, it is supported. As foggy's already mentioned you can also use Veeam for VCSA 6.0 backups and restores, however one has to keep in mind that VCSA uses vPostgres database thus requires some additional steps to be performed in order to make a consistent backup.

Thank you.

[lando_uk]

Hi Foggy,
I'm sorry to keep going on but your VMware KB link says "The backup of the vPostgres database is not required when performing a backup using a supported method. "

So is a standard Veeam v9 backup a `supported method` or not?

Also VMware says "This article is only supported for backup and restore of the vPostgres database to the same vCenter Server Appliance. Use of image-based backup and restore is the only solution supported for performing a full, secondary appliance restore"

So this is saying that if you're restoring for DR, image based backup is the way to go.

You understand that we don't want to and shouldn't need to (according to VMware) run any post backup scripts or stop vcsa for a constant backup.

Thanks

[Pat490]

vSphere 6.5 has just been announced at VMworld Europe with new backup capabilities!
https://blogs.vmware.com/vsphere/2016/1 ... e-6-5.html

[PTide]

So is a standard Veeam v9 backup a `supported method` or not?

According to VMware documentation it is supported only if the following requirements are met:

- it is full backup (no increments)
- VMware tools are installed
- VM uses FQDN with correct DNS resolution, or is with a static IP
- no snapshots
- no FT

All other cases are unsupported and might require extra steps to ensure backup consistency.

Also,
Thanks

[rreed]

Veeam uses snapshots to do its backups, doesn't it?

[Vitaliy S.]

Yes, that's correct, but what Pavel tried to say "VMs that have existing snapshots", nothing to do with the way a backup job is done.

[orangep7]

Quick Update:

I ran into the same issue again this weekend when having to restore the same vCenter again (due to a failed in place upgrade). "Entire VM" restore kept failing and in order to recover the VM in the end I had to perform a "Restore VM files" job instead. New Case # 01969511 has been opened and support bundle has been Ftp'd.

Whilst troubleshooting the Restore failures I ruled out the following:

- Tried different transport modes on the Veeam Proxy
- Tried different credentials
- Tried different Veeam proxies, including a new Proxy
- Tried restoring to different hosts
- Made sure the Veeam proxy was on the same host that I was restoring vCenter to
- There were no orphaned HotAdd disks on the affected VM or any of the proxies.
- AutoMount was disabled in the Veeam VM Guest OS.
- Hardware versions were OK
- iSCSI controller had less than 15 disks attached
- The Veeam VM had been restarted in order to clear any possible locks
- The host was not a Standalone server
- Same cluster
- The VBKs were ‘full active’ and contained no snapshots

In order to have any confidence that vCenter is still fully protected I obviously need to understand why the latest Restores failed - so we'll see what Support come back with.

[rreed]

In further testing and closer scrutiny, ours (VC and PSC both) seem to restore fine to a test environment, albeit we have not yet tried to do a live restore to make sure it connects to hosts, talks on the network, etc. properly. We will find a later quiet time that we can power down our live VCSA's and do a live test.

[tsightler]

Just want to say that I've restored probably 100+ vCenter servers over the past 5 years (a lot of these were for a handful of customers testing this specific DR scenario), in both test and production customer environments, and I've never had an issue with not being able to recover the vCenter assuming the backup was properly made (i.e. included the database).

I did have one very complex issue with vSphere 5.5 because the inventory database itself was outside of the SQL database. We had to stop the inventory service and use file level recovery to restore some older backups, but we were finally able to get the inventory database to start and run properly. This was important because, while it's completely possible to create a fresh inventory database, in vSphere 5.5 all of the tag information was stored there, and this customer made heavy use of tags. I'm pretty sure in vSphere 6 that tags information is now in the main vCenter database.

Regardless, restoring vCenter is a "special" event. While you can backup vCenter with Veeam using the vCenter connection, it is obviously impossible to restore using this method since, to restore vCenter you must power vCenter off, but it is obviously impossible to communicate with a powered off vCenter, so it creates a significant chicken-and-the-egg problem.

The solution is to determine which specific host the vCenter VM is currently registered against, temporarily add that host to the Veeam console via IP address as a standalone ESXi host, and then chose and advanced restore and restore either the VM disks or entire VM, mapping the resources to the standlone hosts. The reason you have to add the host by IP address, rather than by name, is because the Veeam console will not allow you to have the same host added twice, once via vCenter and again by direct host, so adding the host by IP address is a little bit of a trick.

[albertwt]

That confuses me, since vCSA is a Linux appliance and doesn't support VSS.

Does this means that the new VMware VCenter Standard Appliance version 6.0 and above backup is not supported by Veeam 9.0 yet ?

[foggy]

No, this means that you need to use pre-freeze/post-thaw scripts to get a consistent backup of Linux VMs (or go crash-consistent otherwise).

[albertwt]

Thanks Alex for the clarification, so where can I find the script for that pre-freeze/post-thaw scripts ?
Does this means that during the backup window, there might be some outage since the script kicks in

[foggy]

Thanks Alex for the clarification, so where can I find the script for that pre-freeze/post-thaw scripts ?

Should be some generic script to prepare database for the backup.

Does this means that during the backup window, there might be some outage since the script kicks in

This implies a short period of time when the database is stopped.

[dellock6]

Actually, Postgres has continous log save, so any transaction is simply re-played upon restart if it is missing. This means that there is no real need to put the filesystem or the database in a consistent state. Per postgres documentation:

“We do not need a perfectly consistent file system backup as the starting point. Any internal inconsistency in the backup will be corrected by log replay (this is not significantly different from what happens during crash recovery). So we do not need a file system snapshot capability, just tar or a similar archiving tool.”