VMs crashing

[dreeves]

We have 2 tickets open on this. 07819627 and 07820223. Both tiskets have escaleted but I thought I cast a wider net here. We had Veeam CDP I/O filter 12.2 installed on two vSphere 7.03 clusters running 115 jobs all of the time and things were fine.
We upgraded two of our clusters to vsphere 8.0 U3g. All the VMs in those clusters have the veeam cdp storage policy attached. We use vsphere lifecycle manager. After all of the hosts were upgraded we then start ran the vbr i/o filter wizard with drs off . Then we started using the LCM to push out the new i/o filter. During this random VMs would get to 68% in the vmotion process for entering maintenance mode and then lock up. The VM became unresponsive and the process for the VM had to be killed via ssh. We had an idea about removing the cdp storage policy. When several VMs were selected and the policy set to datastore default, some of the VMs took a long time to change that settings and then they locked up. Not at 68% but other lower percentages. We found this kb at Broadcom wihich sounds very much like what we encountered. https://knowledge.broadcom.com/external ... veeam.html There are three of us here working on getting all of this worked out and each of us have a feeling this issue is somehow related to veeam cdp. We're unable to upgrade the i/o filter using normal methods because we have a large likelihood of VMs becoming unresponsive. We're looking for help understand how/if veeam could be involved in this issue and what other methods to use to update the i/o filter on esxi hosts. Or even to remove the filter completely without moving VMs or putting the host into maintenance mode.
We have 3 other vcenters with 5 clusters and 12 hosts total. All of these were upgraded using the same software and methods and all of them are running fine. The difference is that none of those have the Veeam CDP filter.

[Kwa-GJ]

There is also post on reddit with sounds a bit like your issue (Veeam CDP VMs hanging when backed up or vMotioned)

This CDP issue make me nervous we are running the same versions now with CDP

[dreeves]

What we got back from support:

This is specific to upgrading from ESXi 7.0.3 build-24585291 with VAIO bundle 12.3.19-1OEM.700.1.0.15843807 installed to
ESXi 8.0.3 build-24674464 while retaining the same VAIO bundle.

VMware changed compatibility between vSphere 7.0 and 8.0, and the VAIO bundle from VAIODK 7.0 is not fully compatible with vSphere 8.0.

There are 2 methods to avoid the issue when upgrading from Esxi 7 > ESXi 8:

• Update the VAIO bundle on ESXi hosts to 12.3.20-1OEM.800.1.0.20613240. This can be done using the I/O Filter wizard. Prior to attempting migration.
• Remove the VMs from the CDP policy. This will remove the I/O Storage Policy from the VM and unload the I/O filter from the VM process.


There is no such a problem with vSphere 8.0 and 9.0. Bundles for ESXi 8.0 will work with ESXi 9.0.

[BoKTC]

I had this exact same issue last night (and this morning) whilst upgrading from 7.0U3 to 8.0U3 - two virtual machines protected by CDP stuck at 68% during vMotion from a v7 host to a v8 host (and also between v8 hosts) during my upgrade and the source host had to be rebooted to free them up to allow HA to take over and relocate them to a new host. At the time, I'd not pushed the I/O filter upgrade yet as not all of the hosts in the cluster had been updated to 8.0U3.

I ended up manually applying the new I/O filters because I couldn't risk Veeam trying to automate the process and stacking a bunch of critical VMs (i.e. the type of VMs you protect with CDP)

Is it the case then that Veeam are confident this issue is resolved by updating the I/O filters to the version 8 bundle?

[Gostev]

Veeam would need to investigate and understand your specific issue before we could confidently say it is resolved.

However, if your issue happens to be specifically the one explained by our Support to the previous poster, then yes upgrading the Veeam I/O filter driver to a version that is compatible with vSphere 8.0 address that specific issue.

[BoKTC]

Hi Gostev,

Thanks for the response - it would be useful to confirm if the issue would affect upgrades from any version of vSphere 7.0U3 with VAIO bundle 12.3.19-1OEM.700.1.0.15843807 installed to vSphere 8.0U3 prior to upgrading the I/O filters to 12.3.20-1OEM.800.1.0.20613240.

It may also be useful to note that due to the issues caused by attempting to vmotion CDP-protected virtual machines through DRS via the I/O filter wizard, I found it necessary to manually remove the I/O filters using CLI to the affected hosts, then install the update via the ZIP file bundle (again using CLI) and then re-running the wizard to confirm the update was complete.