VEEAM V12 do not support HotAdd and quiescence with vVOLs

[adurelli]

Hello everyone.

I'm using vVol´s on my VM´s (Pure Storage with ESXi 7 U3). After upgrading to V12 my backup jobs do not work anymore. They fail with "No backup proxy is able to process this VM due to proxy processing mode restrictions".

I opened two support cases with VEEAM (#05880708 and #05894905). After some testing, the official answer is that V12 do not support Virtual Appliance transport mode (HotAdd) with VMware Tools Quiescence on vVOL's "by design". This is not reported anywhere, though. The solution offered is to use NBD transport mode.

I'd like to know if anyone elese is using the same config as mine. HotAdd gives me better performance and less load on network side. And since it works on V11, I'd like to ask VEEAM to continue supporting this on V12.

Thanks.

Anderson Stano

[Andreas Neufert]

Hello Anderson,

please allow me to review the cases and I will come back to you.

[Andreas Neufert]

Update, I escalated the support case and will come back to you a bit later with additional information.

[adurelli]

Update, I escalated the support case and will come back to you a bit later with additional information.

Thank you @Andreas Neufert!

[Andreas Neufert]

I found out that we disabled this in our code (hence the statement from our support to you) because of some limitations of VMware. I am researching now if this is a bug or newly introduced limitation on their side. I am sorry for the situation, but it will take some time to nail down the final result.

[adurelli]

Ok, Andreas. I´ll wait for your investigation.

[GabesVirtualWorld]

Love to see the end conclusion on this since we're considering move to vvols and this could be a show stopper for us.

[SergeyGCHR]

If your windows proxy is inside the vApp then move it to the root of the VC and hotadd will work, another workaround is to use a linux proxy (hotadd works with vvol). And if you want to restore vm in hotadd mode, migrate proxy to vvol destination datastore beforehand

[adurelli]

Hello Sergey!

Thanks for your reply! But I'm using linux proxies and can confirm that is not possible to use hotadd and vVOLs on V12 with them too.

[Andreas Neufert]

Just wanted to send a short update. We were able to reproduce the issue without Veeam with manual snapshots and disk mounts and have created a partner case at VMware to get a statement. It looks like that it might be a limitation on the VVOL storage vendor side potentially, but we first need to get some general guidance from VMware.

[DonZoomik]

This thread caught my attention so I did some tests in my environment.
vCenter is 8.0U1, hosts are a mix of 8.0U1 and 7.0U3. Veeam v12. Pure X20 with VVOLs. Linux HotAdd Proxy (Oracle Linux 8 ), not on VVOL. Test VM is some random Windows VM.
Indeed, when I enable quiescing on the job, transport mode degrades from HotAdd to NBD. Looks like a strange limitation, as quiescing is basically only guest OS VSS snap integration.

[Andreas Neufert]

Hi Mihkel,
yes, v12 will automatically fallback (without trying) to NBD-Network mode because of the number of issues we faced and in our lab it did not worked at all.

@All
VMware confirmed that this is not a known product limitation on their side and have requested additional logs for their analyze. So for now it is not clear if it is an issue on the VMware or VVOL storage vendor side. If you want to test if you are affected, you need to test with Veeam Backup & Replication v11 (all versions). Maybe we can nail it down together on some of the vendors or can confirm that more than one vendor is affected. We will do some testing on our side as well and reach out to the storage vendors.

[adurelli]

Hi Andreas!

The test should be done on V12, correct? V11 is working as expected.

[Andreas Neufert]

Hi Team, an update from our side. We reached out to VMware and they confirmed that it is not a limitation on their side. We checked various cases with vendors and whenever it did not worked there was a message in the debug logs of VMware that stated from the VVOL vendor that a operation was not supported. We could nail down a vendor and are working with this vendor currently to get some statements from their side.

What we saw in our lab was that ONTAP (Lenovo/NetApp) is working, as well as Pure Storage.

@adurelli test with v11 as v12 has the function deactivated by design. We will correct this soon, but for now for testing use v11 (latest patch).

[adurelli]

Hi Andreas!

Great news. I'm using vVOL with Pure Storage, so I'll be waiting for the update on V12.

I can confirm that Pure storage is working as expected on V11.

[Andreas Neufert]

Hi everyone, we have create a bugfix on top of v12 12.0.0.1420 P20230412 (April 14, 2023) and will add it as well to the next cumulative update.
The bugfix is in QA testing and I saw the first successful runs already. So in the next days you can reach out to support to get the bugfix. I will inform you when available.

[adurelli]

I applied the patch on my environment and jobs are running as expect with vVOL + hot-add + VMware quiescence.

Thanks for the support @Andreas Neufert!

[Andreas Neufert]

And the update from support/QA. For those that are affected, please update to latest v12 12.0.0.1420 P20230412 (April 14, 2023) then please open a support ticket and request access to bugfix for bug 513102. We apologize for the inconvenience.

For those that implement the bugfix and the job errors out while processing, please mention here your ticket number and share as well your storage model and firmware number. We can check on our existing list if you storage vendor did not implemented the specific needed VVOL operation. For those where it worked in v11, should not face any issue after applying the patch in v12.

[wilhelmsch26]

upgrading to VEEAM 12 has been an unmitigated disaster. We upgraded from the latest version of 11 to 12 build 1420.

Once upgraded none of our backups can be processed through hotadd using an ESXI proxy if the storage is hosted on a vvol: Case #06022337 support had no solution for us and I ended up having to downgrade back to 11..

[adurelli]

Hello @wilhelmsch26!

Same case here. But VEEAM release a patch for this. As Andreas reported, you should open a ticket and request patch for bug 513102

[bhrvi]

Hi. Does anyone know if this bug still exists in 12.1.2.172? We are experiencing the exact same problem and support just told me to stop using vmtools quiescence, which is really not a good solution. Case #03183605.