backup vcenter (Photon OS) with application aware processing

[mcz]

Hi everybody,

now veeam has implemented aap for photon OS in the latest version, so I wounder if I should use that to backup a vcenter? Is it recommended to do so or only for very specific situations? I've tried to enable it but veeam isn't able to connect to the vcenter even if I enable SSH. Eror messages 'mlocate was not found'. Now I've googled a bit and read that you shouldn't install mlocate on the vcenter because you could run into upgrade issues or whatever later.

So I've got the suspicion that it's maybe not needed/built for that? Please shed some light on this - thanks!

[nokogerra]

Hello there.
Well, you don't want to install something in your vCenter appliance, cause it is not supported by VMware. Even the installation of vROps agent into VCSA is not supported, despite the fact, that vROps is a VMware product. Do not modify the package set of your vCenter appliance if you don't want to get in trouble, when you will raise the SR in VMware GSS.

About the backup: I've heard a lot of things about the image-level backup of VCSA. Someone says that image-level backup is not supported for vSphere 7, here is the weird statement from Nakivo: https://www.nakivo.com/vmware-backup/vm ... whats-new/, "Image-based backups of vCenter are not supported in vSphere 7. Use NAKIVO Backup & Replication to perform image-based backup of VMware VMs including VMs running vCenter Server 7." Someone says it is still supported, but not for long: https://veducate.co.uk/vcenter-7-0-imag ... -versions/.
I've been told by VMware employee that image-based backup of VCSA is not supported for vCenter 7, and it was during the vSphere 7 webinar this spring.

If you are running more than one PSC or more than one vCenter with embedded PSC in a vSphere SSO-domain, you will get a USN roll-back after the image-level restore. There is no option to launch the reconcile procedure even via API (I couldnt do it in vSphere 6.7u1) https://docs.vmware.com/en/VMware-vSphe ... F320B.html, https://vdc-download.vmware.com/vmwb-re ... 7427A.html. GSS engineer wished me a good luck with that lol. If you are running only a single PSC or vCenter with embedded PSC within the SSO-domain, then USN-rollback is not an issue.

In any cases, it seems the VCSA file-level backup (via VAMI for example) is the best choice now, cause it is fully supported by a vendor (VMware I mean) and will be supported in the near future. Furthermore, it is the only way in vSphere 7 (where you can't have external PSC) in case you are running 2 or more vCenters with embedded PSC. During the file-based restore the procedure of domain rejoin is launched (vmdird restore, USN is dropped to 100 afaik), which protects you from USN-rollback. So, I guess, it's better to plan file-level backup for you vCenter appliances.

[PetrM]

Hello,

This KB article describes the best practices for backing up VCSA with embedded or external database, I'd suggest to take a look.

Also, I couldn't find so far an official statement from VMware that image-based backup is not supported for vCenter 7.0.
At least there are no similar statements on this page which clearly says that backup and restore of virtual machines containing vCenter via third-party products is possible.

For example, different restore scenarios are described here.

Thanks!

[nokogerra]

PetrM, let's imagine you have 3 vCenters 7.0 in vSphere SSO-domain. How would you restore any of them via Veeam B&R in case of failure without getting a USN-rollback and without breaking vmdird replication?

[PetrM]

Hello,

This is a definitely tricky case and I guess USN-rollback issue must be addressed separately once the restore is done. However, the best practices provided above are still applicable for deployments having a single VC.

Thanks!

[nokogerra]

"Tricky" is the correct word. Just to summarize, there are 2 ways to restore VCSA with an embedded PSC in case of multiple vCenters in a single SSO-domain:
1. Restore from the file-based native backup. You will not get the USN-rollback, casue the reconcile operation will be initiated during the restore process (I've mentioned it before).
2. Resotre from the image-based backup (it doesn't matter if it was made by Veeam B&R or by some other tool). You get USN-rollback after restore, break the replication between PSCs in your domain and should raise an SR in GSS.

[mcz]

Thanks guys for this very valuable input! We are running a single vcenter, so we don't have the issue with the USN. At the moment, I still wonder if you should do application aware processing for a single vcenter or not. I could also ask the question from the "other" direction: Why did veeam ever include application aware processing for Photon OS when it's (maybe) not needed/applicable for a vcenter?

[nokogerra]

Hello again.

> At the moment, I still wonder if you should do application aware processing for a single vcenter or not.
Well, since the Postgres SQL processing is available only with Veeam agent for Linux (afaik), and since you can't install this agent into VCSA (well, you can, but you shouldn't), then no, I guess you shouldn't use an application aware processing in case of vCenter appliance.

> Why did veeam ever include application aware processing for Photon OS when it's (maybe) not needed/applicable for a vcenter?
Well, ususally "appliance" (regardless of manufacturer) is the blackbox, which includes general-purpose OS with pre-installed specific application and it's dependencies. This is a complete solution. Just imagine how would you support your appliance, if people will modify it as they like. You won't. Same thing with VCSA. You should differentiate vCenter server appliance (Phtoton OS based) and Photon OS. Application aware processing is not supported for VCSA, since the appliance vendor (VMware) can't support the installation of various 3d-party agents/tools/utilities and their behaviour inside the appliance OS. But Photon OS is just a Linux, which is maintained by VMware. So you can use application aware processing with Photon OS guests, you just can't do it with particular appliances like VCSA.

You can use the mentioned KB as a workflow to backup and restore the VCSA, but for me it's too many steps and caveats in this approach. It's just too tricky, so you should do it manually or automate these teps. Furthermore, you should stop the vpxd. For me it's not an option (we should order a maintenance window 2 days before the procedure haha).
Image-based backup is great as well as application processing, but not in this particular case. I would say it's just much easier to use native file-level backups for vCenters. At least the restore process is straightforward and backup doesn't require to stop the vpxd. Take a read the doc https://docs.vmware.com/en/VMware-vSphe ... D7EA4.html.

Also I would say, you should use backup best practices of the solution vendor. And the solution vendor in your case is VMware. If something went wrong during the restore, and after all, your vPostgres engine won't start, or vpxd, or vmdird or someting else won't start, you will raise an SR in VMware GSS, not in Veeam tech support.

In my opinion it's better to do both: image-level backups (without app aware processing, but could use quiesce) and file-level backups, so you have more options in case of crash. For example, image-based backup can be used to restore the appliance into iolated area (there can be situations, when it could be useful).

[PetrM]

Hello,

@nokogerra thanks for the feedback! Image-level backup by Veeam as per the KB above + file-level backup by VAMI seems to be a reliable way to fully protect vCenter.

Apparently, vpxd stop is not an option for large-scale environments where maintenance window must be scheduled in advance and depends on various administrative tasks.
Nevertheless, a separate backup of vCenter database would also work if maintenance was performed on a regular basis. Automation of this procedure does not seem to be a sophisticated task at first sight: as far as I see PG_DUMP command is triggered from the scripts attached to this KB and the script itself can be called on schedule.

By the way, test restore in the isolated environment is a way to go to prevent issues during restore in case of real disaster. In theory, SureBackup functionality might be helpful if you want to validate availability of services after restore or if you want to test a restored VM manually.

Thanks!