Recover Entire Cluster - Starting with vCenter

[penguinpages]

Case: 05029640

Their are various posts around about recovery from total site rebuild using Veeam, but documentation and experience are not seeing to match up.

Site Design:
3 x Intel vsphere 7 hosts (each host has a 1TB SSD for local vmfs / cache / logs / non-replication vms)
1 x VM vCentner
1 x vSAN shared

########

vSAN crashed and after a week, I am giving up on data repair of volumes. Disk are fine, Network is fine, but something happened when power lost and UPS agent did not gracefully shut down... or something else <shurgs>

I have veeam and kasten backups (OCP and Tanzu Test environment also) but need to start with basics of restoring:
1) vCenter
2) Domain Controllers
3) Ansible / Git Repo systems
4) OCP / Tanzu VMs

But the issue is recovery of first VM... vCenter. I built a standalone Windows recovery VM called "veeam" and attached to backup, imported data, and now wish to select to restore vCenter .. but direct to vSphere host vs via vCenter connection. (Maybe in future I will setup seperate backup job direct to vSphere host for vCenter to avoid this sillyness)

I select VM -> Restore to different location / host -> manual input hostname but the button to continue never allows it to continue.

I have logged in to all three hosts via that "veeam" VM via Putty and Web so IP communication is not an issue. This is something within Veeam recovery process, or something that I am missing.

[foggy]

Hi Jeremey, have you added the ESXi host to Veeam B&R prior to that?

[penguinpages]

Only in context of I bound vCenter to the backup routine and schedule.

That I think is my mistake... as it has no inventory / library of connection / credentials for direct ESXI host.

I can change that once I get things back.. but.. how do I get around this

Oh.. and maybe a wizard can say 'your backing up a vCenter host... you should not backup this host via itselft as connection but setup a seperate backup path for this particular vm"

[foggy]

No, adding vCenter and pointing backup jobs to it is fine. I mean you should add the ESXi host directly in the new Veeam B&R instance you're using for restore - in this case, you will be able to select it as a restore target.

[penguinpages]

This errors out. To add the host directly errors because the Veeam system knows the host is present as an object under vCenter. (error: VMWare ESXi with name "esxi.penguinpages.local" already exists"

But.. you did give me an idea.. I just added the host by IP address directly and that worked around Veeam not understanding it as duplicate target. (error: Failed to login to "vcenter01.penguinpages.local, port 443....." -> It errored as it tried to pull inventory from its vcenter hosting system.. but this time it did add the connection direct via IP.

With that I was able to run restore-> Select full VM -> restore custom location settings -> <it listed original vcenter+esxi1 as backup source but you can select "host" and select IP of desired esxi host -> new restore path -> restore

VM failed to restore.. but that is because dvswitches did not have ephemeral ports enabled such that the host can allocate a port to the restored vcenter01 VM. So I have now break out a NIC into a standard switch and move it over to that to power it on.. Working on that 20 step process now....

PS: their was an entire sub thread about getting valid backup of vCenter due to lock files and corruption. My backup Sunday was that issue.. Trying to role back to Saturday now.

[Origin 2000]

Veeam cant manage the same host twice and it will always remember that esxi.penguinpages.local comes from vcenter01.penguinpages.local.

Solution is quite simple... add the ESXi manually trough its IP Address a 2nd. time and not using FQHN.

YES... if you use a VDS you needs a portgroup from type ephemeral for all (vSphere) management related stuff and for sure the vCenter VCSA needs to place in such a portgroup. There is a VMware KB about.

Regards,
Joerg

[penguinpages]

All six backup copies I had of vCenter .. Though log says successful .. restores fail.

############

Power On VM
Key
haTask-506-vim.VirtualMachine.powerOn-1599121178

Description
Power On this virtual machine

Virtual machine
vcenter01
State
Failed - File system specific implementation of Ioctl[file] failed

Errors
File system specific implementation of Ioctl[file] failed
Failed to lock the file
Cannot open the disk '/vmfs/volumes/611bec6e-48e0ca22-11e8-a0423f377a7e/vcenter01/vcenter01_2-000001.vmdk' or one of the snapshot disks it depends on.
Module 'Disk' power on failed.
File system specific implementation of Ioctl[file] failed
Failed to lock the file
Cannot open the disk '/vmfs/volumes/611bec6e-48e0ca22-11e8-a0423f377a7e/vcenter01/vcenter01_1-000001.vmdk' or one of the snapshot disks it depends on.
File system specific implementation of Ioctl[file] failed
Failed to lock the file
Cannot open the disk '/vmfs/volumes/611bec6e-48e0ca22-11e8-a0423f377a7e/vcenter01/vcenter01-000001.vmdk' or one of the snapshot disks it depends on.
Failed to start the virtual machine.

#####

Veeam Restore log:
9/17/2021 2:54:40 PM Error Restore job failed Error: An unexpected network error occurred.
Asynchronous request operation has failed. [readsize = 1052672] [offset = 43848802304]
Unable to retrieve next block transmission command. Number of already processed blocks: [2042].
Failed to download disk 'Device '\\.\PhysicalDrive2''.
Agent failed to process method {DataTransfer.SyncDisk}.



Seems backup via snapshots is not going to be productive means. I guess we go back to rebuild and restore from ftp copy out of VMWare database

[NightBird]

C

vSAN crashed and after a week, I am giving up on data repair of volumes. Disk are fine, Network is fine, but something happened when power lost and UPS agent did not gracefully shut down... or something else <shurgs>

Ouarf your vSAN do not survive a power outage ??? seriously ???

[penguinpages]

Ya.. Dedupe and compression I think was bad idea.. until I get my base vSAN stable and tested out more. https://communities.vmware.com/t5/VMwar ... 402#M13461 What I am struggling with now is Veeam restoring VMs. vCenter backup was fubar... but trying to recover the rest. Realizing I missed several VMs such as templates which kind of was bad on my part..... Need to test recovery . Basic steps of a good admin that I neglected and just A$$umed worked.

[jamcool]

Something that we did was to replicate our vcenter's to another datacenter. I know that would not work here, but maybe if one of your ESXi servers had some local storage, you can replicate to it so have an offline copy of your vcenter on different storage. We also do this with our Veeam VBR (not repository servers) and Active Directory domain controllers servers. Just tossing an idea out there going forward.

[ChuckS42]

If you're on VMware v7 you can set up file based backups of vCenter data; store those someplace safe and you can restore vCenter in a VMware-supported fashion. (Deploy new VC appliance, then run a restore from the VC appliance management page.)