Best practices for backing up the B&R and ESXi special purpose servers?

[cosmik]

I've been backing up my ESXi-based infrastructure with 11a, with the exception of certain VMs. This is a simple, 2-host ESXi cluster under a VMware essentials plus license.

My questions are about backing up the Veeam B&R VM itself, as well as the vCenter VM.

With regard to the former first: let me add that it's running the server, the main backup proxy, the console and it will probably host a WAN acceleator (150-200Gb storage). I think I saw somewhere that backup jobs that include backup backup proxy hosts should not mingle with the other backups taken, or performance drops will take place.

As for the latter, it's a 7.0 vCenter appliance.

Any hints/advice and especially pitfalls to avoid?

[HannesK]

Hello,
for the backup server, configuration backup is recommended. That prevents from any chicken-egg issues. Technically it works, but I would avoid it, because the snapshot affects the database, which is running on the VM. As alternative, a standalone Veeam Agent for Windows backup could also be used.

For VCenter: please use forum / internet search. It's still supported, but deprecated by VMware. So I would not start it today. File-based backup of the configuration is the recommended way from VMware side.

In any case, I would not back up VCenter via VCenter, because it again can affect the database running inside the VCSA.

In both cases, it will likely work. I would just not do it.

Best regards,
Hannes

[barlid]

Hi Michael,

For VBR and vCenter backup strategy, I follow this method:

• For backup server I totally agree with Hannes, use Veeam configuration backup approach for backup server when your backup server failed, you can easy install new VBR and then restore it to have your VBR back.

• For vCenter which you mention version 7.0, you can perform file-based backup in VAMI, you can schedule backup through VAMI and put backup to remote system(Typically I use Linux VM) through NFS, ftp or sftp, In case your vCenter failure you can just deploy new vcenter and perform restoration process on stage 2 deployment

• I also perform image-level backup of Linux VM which use as target vCenter backup to VBR to protect vCenter backup files

hope this answers your question
Thanks
Barli

[cosmik]

Thank you both for the information. For the backup server, I'm already taking configuration backups to the hardened/immutable backup repo. I'm also taking backups daily via VAMI for vCenter to a Synology password protected SMB share (snapshotted, just in case). However setting up B&R is quite easy. I can not say the same thing for ESXi. Never had done the part. I understand that I'm going a bit off-topic here, but assuming some catastrophic event that would leave me with me with my ESXi hosts wiped, how would I go about recreating things? I mean, does a vcenter config contain things like NIC allocations/FC mappings to the storage etc?

TBH, setting ESXi from scratch to the way it was scares me shitless. If you have any (extensive) link on how that is done, feel free to throw it my way

[HannesK]

TBH, setting ESXi from scratch to the way it was scares me shitless

setting up VMware / ESXi hosts should not be scary My suggestion would be to set it up as nested environment to learn about it (it even works in VMware workstation. I did that many years). There are blogs and videos all over the internet how to do it. Just pick your preferred platform / search engine and there should be good results (I use Google & YouTube)

[Regnor]

The vCenter backup contains only the vCenter, but no ESXi configuration. So, if you're not using vDS, you'll have to setup networking manually for every ESXi host.
But the setup and configuration of an ESXi host is much simpler than for example installing a Window Server.
If your configuration isn't complex, everything can be done in like 30-60 minutes. Just document all settings and perhaps do it one time from scratch, then you're good to go.
In addition you can export/backup the configuration of each host via ESXCLI, PowerCLI and so one: https://kb.vmware.com/s/article/2042141
->Just make sure that you recreate those exports after every update, because the build number from source and target needs to be identical.

[cosmik]

Just document all settings and perhaps do it one time from scratch, then you're good to go.
In addition you can export/backup the configuration of each host via ESXCLI, PowerCLI and so one: https://kb.vmware.com/s/article/2042141

Awesome! I've got a somewhat complex setup regarding the 8 GbE links of each host, and I don't have them documented clearly. I'll make sure to backup the host settings.

Thanks again!

[barlid]

Thank you both for the information. For the backup server, I'm already taking configuration backups to the hardened/immutable backup repo. I'm also taking backups daily via VAMI for vCenter to a Synology password protected SMB share (snapshotted, just in case). However setting up B&R is quite easy. I can not say the same thing for ESXi. Never had done the part. I understand that I'm going a bit off-topic here, but assuming some catastrophic event that would leave me with me with my ESXi hosts wiped, how would I go about recreating things? I mean, does a vcenter config contain things like NIC allocations/FC mappings to the storage etc?

TBH, setting ESXi from scratch to the way it was scares me shitless. If you have any (extensive) link on how that is done, feel free to throw it my way

Okay, your first thread said about VBR and vCenter backup anyway and then now ESXi . You dont have to worry about ESXI, I encounter many times to reinstall ESXI cause PSOD, it wont be hard stuff to do i think.

Thanks,

[cosmik]

I assume you are referring to this part (note the part in bold ):

I can not say the same thing for ESXi. Never had done the part. I understand that I'm going a bit off-topic here, but assuming some catastrophic event that would leave me with me with my ESXi hosts wiped, how would I go about recreating things? I mean, does a vcenter config contain things like NIC allocations/FC mappings to the storage etc?

As for the initial reference to ESXi: it was not clear to me whether the vcenter configuration backup includes low level information from the ESXi hosts, like NIC mappings etc. Hence for my reference. regnor clarified that vcenter does not include that. Started taking backups for each host, as per VMWare's KB

All in all, thanks for strengthening my resolve here! I am quite stressed about having a huge malfunction on my ESXi hosts or something

[Regnor]

Awesome! I've got a somewhat complex setup regarding the 8 GbE links of each host, and I don't have them documented clearly. I'll make sure to backup the host settings.

Thanks again!

But please also create a documentation as a fallback when the ESXi configuration backup/restore doesn't work
Also, before being able to restore the backup, you'll initially have to install ESXi and at least assign an IP.

It's unlikely that you'll use all ESXi hosts at the same time, unless there's some really catastrophic event or all hosts use SD cards which die at the same time (or have HPE SSDs with some old firmware ).

[cosmik]

I had tried a zillion times to document networks, IP configs, vlans, special settings etc. The average day work load does not allow that.
I'm running around fixing cabling, providing end user support (read: showing again and again the same excel simple function), recovering deleted files from net shares, replacing failed capacitors on motherboards. At the same time I'm trying to push the architecture one step forward, have some security around that's better than the average Joe's etc.

Time: never enough of that...

[MatzeB]

For documentation I can also recommend RVTools. Either once or even better periodically, e.g. every week.
It can be executed by script and saved as a file or sent by mail.

Regards
Matze

[abdullahmukadam]

I've been backing up my ESXi-based infrastructure with 11a, with the exception of certain VMs. This is a simple, 2-host ESXi cluster under a VMware essentials plus license.

My questions are about backing up the Veeam B&R VM itself, as well as the vCenter VM.

With regard to the former first: let me add that it's running the server, the main backup proxy, the console and it will probably host a WAN acceleator (150-200Gb storage). I think I saw somewhere that backup jobs that include backup backup proxy hosts should not mingle with the other backups taken, or performance drops will take place.

As for the latter, it's a 7.0 vCenter appliance.

Any hints/advice and especially pitfalls to avoid?

We had this customer who wanted to backup the VBR VM and we had to involve a personnel from Veeam and he had advised that backup is possible, however not recommended, and if they still want the VBR backup then they would need to disable application aware image processing and take a crash consistent backup.

With regards to vCenter you can take both via the Veeam VBR server and also the database backup via vCenter Server. You cant test a recovery if possible, by both and see the RTO, whichever is better you can use that during recovery as and when needed.

I have never heard anyone take backup of ESXi, so cant comment.

[YouGotServered]

I've done the ESXi host config backup before and it was easy enough, but never had to restore. Part of Our DR plan is making sure that we have backups of all critical network devices - firewalls, switches, ESXi hosts, WAPs, etc - anything that isn't covered by Veeam, essentially!

I will note that I do have Veeam configuration backups, but I also do a Veeam replication job of our backup server to another host as a secondary measure. I've actually used it a few times, and while it may not be 100% application consistent (due to the chicken and the egg issue), it has worked every time for me. It is valuable if you have an issue with your B&R server, but simply do not have time to rebuild it and the networking over again - just failover the VM and it picks up where things left off (given that your other infrastructure components are accounted for as well).

As far as vCenter - it's flaky as heck (IMO, all VMware products are inherently a little janky) - take every possible avenue for backing it up - config backup, image level backup, replication. Inevitably at least one of those will fail you when you need it for some silly VMware reason.

[cosmik]

Thanks for the advice everybody. Seems like a documentation project will be coming up...

[JC_Solvos]

For the vCenter, we back up and replicate the VCSA machine with Veeam through the host itself (obviously not the vCenter through the vCenter, in order to avoid circular dependencies). All our VMs are added through the vCenter, except that one, so we add both the source and the destination hosts of the VCSA replica as standalone hosts in Veeam.

As we don't have vMotion, whenever we need to do maintenance in the server hosting the vCenter machine, we put it in failover following this method. We've done it like 10 times and it worked flawlessly. I suppose that it should also work with a backup job.

[craigcurtin]

The other thing i would add for the OP is to document the config of the switch that the ESXi box is connected to and understand how to modify it.

In particular if you have 8 nics they will be in some form of bond/channel - possibly with VLANs etc - in particular when you install ESxi from scratch and have the configure the initial management NIC you can run into problems if the switch will not accept it due to bonding/VPN/Trunk issues.

On a weekly basis we replicate both the VCSA and the B&R server to external storage (Local NAS) and then to USB drives - very worse case we can stand up VMWare Workstation and attach to get infrastructure up and running whilst waiting for new/fixed hardware from vendors (in the case of single server sites)

Craig

[cosmik]

Indeed documenting the switch setups is a must. Both a config backup is taken as well as a printed description that is affixed to the front of a rack as a quick reminder.