Feature Request: Backup ESXi Host Configurations

[telston]

We recently had a USB drive with ESXi die in our blade server setup. All of our blades run on USB boot devices rather than iSCSI. I realized its about half a day to a day of work to rebuild and troubleshoot reconfiguring the host. We have a fairly complex setup with 10+ NIC's per host, 60+ servers, 12 blades, a 280 desktop VDI setup, NFS and iSCSI storage, and a few stand alone hosts for DMZ and DR type stuff which got me thinking of ways to automate configuring hosts. Then I started thinking, I dont need to automate the setup, I can just backup the setup and deploy a backup. Once the hosts are setup, they dont change much unless VM"s or new storage is added.

So I thought maybe I can backup a USB drive from another host, reset the SSL configurations, and IP with the host that died's IP configuration. Unfortunately, its not possible to restore a configuration from one host to a different host per VMWare documentation. This requires that each host has its own backup file.

Currently the only 2 ways I can find to backup configurations of ESXi hosts are either manually using PowerCLI/ESXcli or Acronis Backup. Acronis has the capability to setup an OVF on a hypervisor and backup host configurations to data stores, local file servers, or the cloud by connecting to the vCenter server.

Why doesnt Veeam have this feature? Is it already in the pipeline? If your data is important enough to backup with a high end backup product like Veeam, I would think its important enough to make sure your hosts are also backed up to minimize down time.

[Mike Resseler]

Hi Tyler,

First: Welcome to our forums!
Second: We believe that host backup is not that important as the data backup. In the (always) shifting IT world, hosts should be considered as a set of scripts/ configs that can be rapidly deployed again whenever something happens. Kind of the "threat hosts as cuddle, not pets" kind off approach.
I don't believe we have something in the pipeline for this type of backup. But obviously, as you say yourself, you can always use PowerCli to take that backup to a fileserver or something and then backup that fileserver. Item-Level recovery would give you the possibility to recover it afterwards.

Cheers
Mike

[telston]

There are a couple issues with PowerCLI. First and foremost, its not automated. In large organizations, usually ones using Veeam that means someone has to devote time to make backups manually. Now I dont know about HyperV's best practice, but for VMWare its best practice to have SSH disabled on hosts which makes using PowerCLI even more cumbersome. Now you have to find the time to back up all of your hosts, go to every host, enable SSH, putty into them, make a backup, save that backup somewhere, go back to each host and disable SSH, and if you deploy an update to any host, you have to remember to back them up again.

Using Acronis, its able to setup access to vCenter which is already authenticated to each host via its cert so there isnt a need to enable SSH and introduce that security risk. It also removes the need to manually do the backup as it can be configured as a standard job to any storage you have whether it be a file server, SAN, or NFS share.

Anyways, just thought I would toss this idea out there and see how many others like the idea of this.

[nmdange]

I would agree with Mike Resseler, It is probably more important to have a good host deployment procedure to quickly provision new hosts rather than worrying about backing up your hosts. If a host dies, you should be able to deploy a replacement host relatively quickly. In my environment, we don't backup VMWare or Hyper-V hosts. For Hyper-V host deployment, I have power shell scripts that automate the configuration so even installing Windows from base media does not take that long to deploy.

[telston]

I would agree with Mike Resseler, It is probably more important to have a good host deployment procedure to quickly provision new hosts rather than worrying about backing up your hosts. If a host dies, you should be able to deploy a replacement host relatively quickly. In my environment, we don't backup VMWare or Hyper-V hosts. For Hyper-V host deployment, I have power shell scripts that automate the configuration so even installing Windows from base media does not take that long to deploy.

Scripts only get you so far. How do you document vNIC's and vSwitch IP scopes, VLAN's, and configurations for each host? They cant all be the same.

[nmdange]

Aside from the actual IPs assigned to each host, the config should be pretty much identical. Even if you have a few different clusters which have different configs, each cluster should have same config on every node. You might be better off trying to simplify your environment. 10+ NICs per host? Why can't you use 2 10GB+ NICs for storage, and 2 10GB+ NICs for the vswitch? The physical wiring alone has to be a nightmare!

Also curious why you have a single USB stick for booting ESX. It's not like getting mirrored SD cards or even using two cheap HDDs in RAID 1 with the server's onboard raid controller is all that expensive compared to the rest of the server.

[telston]

Aside from the actual IPs assigned to each host, the config should be pretty much identical. Even if you have a few different clusters which have different configs, each cluster should have same config on every node. You might be better off trying to simplify your environment. 10+ NICs per host? Why can't you use 2 10GB+ NICs for storage, and 2 10GB+ NICs for the vswitch? The physical wiring alone has to be a nightmare!

Also curious why you have a single USB stick for booting ESX. It's not like getting mirrored SD cards or even using two cheap HDDs in RAID 1 with the server's onboard raid controller is all that expensive compared to the rest of the server.

We use Cisco UCS 5108 with 12 blades. So we have 10 virtual NIC's across 2 fiber fabrics, not 10 physical NIC's. They are required for different VLAN's. Our storage VLAN, Server VLAN, Management VLAN, vMotion VLAN, and Workstation VLAN. 2 vNIC's each for redundancy. Each VLAN has a vNIC on each fabric on the UCS and each fabric goes to different storage switches. From a physical standpoint there is only 4 fiber connections and 2 copper connections across both fabrics so wiring isnt bad.

I inherited this system as it stands now with the exception of adding additional blades as resources were needed. I was also curious why we werent using mirrored SD cards.

[nmdange]

Ok yeah I haven't worked with those specialized blade systems that let you split physical NICs but I am familiar with them. Obviously not easy to change after the fact, but you might consider just tagging your VLANs to a single vNIC pair rather than having to create all these separate vNICs for every VLAN that have to be managed in ESX. Just for comparison, my environment has dozens and dozens of VLANs for different networks, but we just tag them all on a pair of trunk ports to each ESX host, aside from storage traffic which for us is a fiber channel HBA, but even with iSCSI that should be separated.

[mthiele]

Please bring back an agent to backup ESXi host configuration.

[foggy]

Hi Mark, I'm not sure what do you mean by 'bringing back', since we never had this sort of a backup. You can review some considerations regarding this in the thread above. Thanks!

[smartsys]

I think that for smaller VMware sites an option to backup the host configuration can simplify life for administrators a lot!
We install VMware at many customers and use Veeam to secure the data.
These sites typically use 2 to 3 ESXi hosts, maybe 10 at the max. So, no Enterprise Plus licenses with Host-Profiles options and so on...
Since these are all different companies it is very difficult to use scripts.
I have to change scripts for all these different locations and it is faster to just configure the ESXi host manually.

But. If, and this does happen from time to time, a SD card of USB drive fails. We need to reconfigure the host exact like it was. Think of IQN's when using iSCSI or certificates.
If I can restore the configuration from Veeam it will take me a few minutes compared to a few hours to get the host perfectly running again.
So I think, host backups, can be a handy feature for smaller VMware installations.

[icetrain]

Totally agree. If you manage various smaller sites, setting up the hosts from scratch can be a nightmare.
Isn't there any easy solution to accomplish this? I remember Acrons for VMWare was able to backup the config for hostst.

[GreenAlpha55]

I agree. This is a needed feature. Maybe not so much for large enterprises, but for SMB's yes.

In our DC our hosts use mirrored SD cards. We do not back them up. If one of these hosts goes boom, the workloads will be transferred to other hosts, but we will need to manually rebuild the bad host. This will probably take a few days to build, whereas a backup would bring it back in 10 minutes. I believe there's a need for this.

Out DR location uses a USB stick, but there's nothing critical here and the host can be rebuilt quickly because it's a very simple setup.

[FrancWest]

Agreed, would be a nice feature. For now, our management server uses a scheduled task with a power-cli script to backup the config of all hosts to this management server which is included in the nightly Veeam backup.

[samroy]

Same here. I found this thread while questionning myself if there would be a better way of doing it other than power-cli. I'm in a smaller environment and I wish to reduce complexity and documentation fails/holes at a maximum. It would be a nice to have!

[PetrM]

Hello,

Currently, we don't have plans to deliver this functionality but your request is noted anyway. Also, you can find more tips for backing up and restoring ESXi config in this KB article.

Thanks!

[DonZoomik]

+1 to this.
I'd consider this even more important nowadays with ConfigStore being encrypted by TPM. If you don't know to back up recovery key, a motherboard swap will render your ESXi unbootable. So in an ideal world, Veeam could back up both ESXi configuration and TPM recovery key (needed to recover configuration).
Not something that enterprises with dedicated teams would really care about but clueless SME-s could really use this as a safety net.

[Klarmann]

Hi,

i use a Powershell Script for backing up our complete ESXi environment. It connects to the vCenter, makes a backup of the complete ESX config and write it as an archive to the file system. -> https://www.virtualhome.blog/2019/12/24 ... -powercli/
Restore is pretty easy. Setup an ESX server with no config except ssh enabled ... could have IP via DHCP. Copy the archive back to the ESX (eg. to /tmp) ... ssh to the host and type: /bin/firmwareConfig.py --restore /tmp/configBundle.tgz
1sec later ESX is rebooting and comes up with your complete config you have done and backed up before!

BTW - I have backed up my 42 ESX hosts config (worldwide) within 2 min!

[Kazz]

If you have Ansible running in your environment, you can also back up host configs with a playbook

Code: Select all

---
- hosts: localhost

  tasks:
  - name: Get Date
    setup:
      filter: "ansible_date_time"
      gather_subset: "!all"

  - name: Store Date
    set_fact:
      DTG: "{{ ansible_date_time.date }}"

  - name: Create Directory with Current Date
    file:
      path: /home/backups/esxi_hosts/{{ hostvars.localhost.DTG }}
      state: directory
  run_once: true

- name: Backup ESXi Host Config
  hosts: esxi_hosts
  gather_facts: no

  tasks:
    - name: Obtain and Save ESXi Host Config
      community.vmware.vmware_cfg_backup:
        hostname: "{{ esxi_server }}"
        username: "{{ esxi_username }}"
        password: "{{ esxi_password }}"
        validate_certs: no
        state: saved
        dest: "/home/backups/esxi_hosts/{{ hostvars.localhost.DTG }}/"
      delegate_to: localhost