vCenter I/O filters upgrade

[mohkhalifa]

Dear Guys,
I have a small lab with 2 hosts in a Cluster. Just trying to uninstall the filters and re-install as I added a new host to my cluster. The host which needed to be in maintenance mood is the host which includes the VCSA and Veaam B&R. So, If I put the host in a maintenance mood, all connections will be lost. Also, I can't migrate the VMs to another host for a hardware compatibility issue.
So, How can I upgrade the CDP I/O filters with this scenario ? as I'm getting an error Host can't entered Maintenance mood

Thanks

[Gostev]

Hi, Mohammed. Unfortunately, Veeam has no control over I/O filter management operations, as everything is handled by vSphere exclusively. And if vSphere determines that the ESXi host needs to be put into a maintenance mode, then there's no way around this.

I assume you have maintenance windows periodically, these would be a good time to perform Veeam upgrades too. Just treat it the same way as you treat ESXi host upgrades.

Thanks!

[mohkhalifa]

Dear Gostev,

Thanks for your reply. But How will I perform the upgrade as the Veeam B&R is a part of the ESXi which means If I put the ESXi in a maintenance mood, then I won't be able to reach the Veeam B&R VM as it's turned-off

Please Advise!

Thanks

[Gostev]

Well, I have a much better question for you then: how are you going to restore your backups in case your ESXi host dies? This should be a much bigger concern for you, no?

Basically, the point I am trying to make here is that you've just ran into one of many reasons why deploying Veeam as a virtual machine has always been considered a bad practice. As a rule of thumb, a backup system must never rely in any way on the production system it is designed to protect. While in your case, this production system is literally THE ONLY place backup server can run on!

So, I would highly recommend that you start by moving your backup server to a standalone physical server or workstation. Really, for your environment size even any desktop should do well.

[mohkhalifa]

Dear Gostev,

Thanks for your reply and thanks for your recommendations but you must understand from my point of view that everything now is virtual and working with VM is much easier and more controllable than physical machine problems. So, I prefer to use Veeam B&R as a VM which makes me comfortable.

After working-out for my issue, I want your recommendation for this below solution

1- Switching-off all the ESXi VMs
2- Put the desired host into maintenance mode
3- Using ESXi shell commands for:
- esxcli software vib list (to select Veeam CDP filter name)
- esxcli software vib remove -n veecdp (to remove the filters)
4- Putting the host out of maintenance mode
5- Re-check with Veaam B&R filters status and re-install.

What do you think ?

awaiting your kind feedback.

Thanks

[wishr]

Hi Mohamed,

Please refer to the KB4151 for the correct VAIO filter removal procedure.

Thanks

[mohkhalifa]

Dear All,

Problem solved by putting the hosts into maintenance mood then installing the VIB veecdp manually to the ESXi with esxcli commands. then putting the host out of the maintenance mood then update the cluster I/O filters from Veeam B&R CDP wizard.

Thanks

[hexadecimal]

After some serious research I came to the same conclusion regarding the VBR server being a VM. This is how my predecessor to my position deployed this, and done to the production site. Challenge then becomes if we lose the production site, our DR site has these replicas which now have to be manually turned on, as opposed to the remote VBR server handling the orchestration and such. The production just needs a proxy and repo.

Regardless, I'm in the same boat as mohkhalifa till I'm able to get a new physical server to our DR site and migrate VBR there. Temporarily, will be setting up a VM in our DR site to handle these jobs and at least get VBR away from the production.

@mohkhalifa, would you be able to shed some light on the commands you were able to use to manually install the io filter and where you got the io filter files from? I have 2 hosts with HA available and shared storage. As such my plan is to update the IO filter on each host one at a time, and vcenter last.

[veremin]

After some serious research I came to the same conclusion regarding the VBR server being a VM.

Correct, this is the preferable way to manage replication activity - have backup server deployed at DR site.

if we lose the production site, our DR site has these replicas which now have to be manually turned on, as opposed to the remote VBR server handling the orchestration and such.

Be aware that to fail over to CDP replica you need to have a backup server to replay the journal or handle .delta-disks.

Thanks!

[hexadecimal]

Scratch that. I figurd it out. For anyone wishing to attempt this, the veecdp vib bundle is part of the VBR iso. Once your host is in maintenance mode, proceed with the above commands to remove. Once removed, issue the "esxcli software vib install -d <path to veecdp_vib_zip_file>".

After installation, take the host out of maintenance and return to your VBR console. Under the backup infrastructure, locate the two hosts under vcenter. You'll want to rescan. If you were previously in an errored state, the refresh can take a couple of minutes. If it stays alarmed, go to Manage IO Filters and attempt to reinstall the filters (do not remove, just install.) You will see in the console, and vsphere, an attempt to reinstall. This will ultimately fail however it will clear the errored state within the VBR console.

I have 3 hosts, 2 of which are in a single cluster with HA. I disabled my CDP job, moved my VMs to one host while I removed veecdp and reinstalled, performed the necessary VBR console rescans. I then moved the VMs to the upgraded host (you will get an error message if you move the CDP VM without installing the new veecdp filter and the VM will not move.)

Maintenance mode to do the filter work is a vmware restriction and is necessary. Hope this helps the next person, or myself if I need to do this again in the future.

Thanks, @veremin for the confirmation.