-
- Novice
- Posts: 9
- Liked: never
- Joined: Dec 17, 2019 1:58 pm
- Full Name: Anderson Stano Durelli
- Contact:
VEEAM V12 do not support HotAdd and quiescence with vVOLs
Hello everyone.
I'm using vVol´s on my VM´s (Pure Storage with ESXi 7 U3). After upgrading to V12 my backup jobs do not work anymore. They fail with "No backup proxy is able to process this VM due to proxy processing mode restrictions".
I opened two support cases with VEEAM (#05880708 and #05894905). After some testing, the official answer is that V12 do not support Virtual Appliance transport mode (HotAdd) with VMware Tools Quiescence on vVOL's "by design". This is not reported anywhere, though. The solution offered is to use NBD transport mode.
I'd like to know if anyone elese is using the same config as mine. HotAdd gives me better performance and less load on network side. And since it works on V11, I'd like to ask VEEAM to continue supporting this on V12.
Thanks.
Anderson Stano
I'm using vVol´s on my VM´s (Pure Storage with ESXi 7 U3). After upgrading to V12 my backup jobs do not work anymore. They fail with "No backup proxy is able to process this VM due to proxy processing mode restrictions".
I opened two support cases with VEEAM (#05880708 and #05894905). After some testing, the official answer is that V12 do not support Virtual Appliance transport mode (HotAdd) with VMware Tools Quiescence on vVOL's "by design". This is not reported anywhere, though. The solution offered is to use NBD transport mode.
I'd like to know if anyone elese is using the same config as mine. HotAdd gives me better performance and less load on network side. And since it works on V11, I'd like to ask VEEAM to continue supporting this on V12.
Thanks.
Anderson Stano
-
- VP, Product Management
- Posts: 7074
- Liked: 1507 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
Hello Anderson,
please allow me to review the cases and I will come back to you.
please allow me to review the cases and I will come back to you.
-
- VP, Product Management
- Posts: 7074
- Liked: 1507 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
Update, I escalated the support case and will come back to you a bit later with additional information.
-
- Novice
- Posts: 9
- Liked: never
- Joined: Dec 17, 2019 1:58 pm
- Full Name: Anderson Stano Durelli
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
Thank you @Andreas Neufert!Update, I escalated the support case and will come back to you a bit later with additional information.
-
- VP, Product Management
- Posts: 7074
- Liked: 1507 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
I found out that we disabled this in our code (hence the statement from our support to you) because of some limitations of VMware. I am researching now if this is a bug or newly introduced limitation on their side. I am sorry for the situation, but it will take some time to nail down the final result.
-
- Novice
- Posts: 9
- Liked: never
- Joined: Dec 17, 2019 1:58 pm
- Full Name: Anderson Stano Durelli
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
Ok, Andreas. I´ll wait for your investigation.
-
- Expert
- Posts: 248
- Liked: 38 times
- Joined: Jun 15, 2009 10:49 am
- Full Name: Gabrie van Zanten
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
Love to see the end conclusion on this since we're considering move to vvols and this could be a show stopper for us.
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Dec 13, 2020 8:28 pm
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
If your windows proxy is inside the vApp then move it to the root of the VC and hotadd will work, another workaround is to use a linux proxy (hotadd works with vvol). And if you want to restore vm in hotadd mode, migrate proxy to vvol destination datastore beforehand
-
- Novice
- Posts: 9
- Liked: never
- Joined: Dec 17, 2019 1:58 pm
- Full Name: Anderson Stano Durelli
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
Hello Sergey!
Thanks for your reply! But I'm using linux proxies and can confirm that is not possible to use hotadd and vVOLs on V12 with them too.
Thanks for your reply! But I'm using linux proxies and can confirm that is not possible to use hotadd and vVOLs on V12 with them too.
-
- VP, Product Management
- Posts: 7074
- Liked: 1507 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
Just wanted to send a short update. We were able to reproduce the issue without Veeam with manual snapshots and disk mounts and have created a partner case at VMware to get a statement. It looks like that it might be a limitation on the VVOL storage vendor side potentially, but we first need to get some general guidance from VMware.
-
- Service Provider
- Posts: 372
- Liked: 120 times
- Joined: Nov 25, 2016 1:56 pm
- Full Name: Mihkel Soomere
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
This thread caught my attention so I did some tests in my environment.
vCenter is 8.0U1, hosts are a mix of 8.0U1 and 7.0U3. Veeam v12. Pure X20 with VVOLs. Linux HotAdd Proxy (Oracle Linux 8 ), not on VVOL. Test VM is some random Windows VM.
Indeed, when I enable quiescing on the job, transport mode degrades from HotAdd to NBD. Looks like a strange limitation, as quiescing is basically only guest OS VSS snap integration.
vCenter is 8.0U1, hosts are a mix of 8.0U1 and 7.0U3. Veeam v12. Pure X20 with VVOLs. Linux HotAdd Proxy (Oracle Linux 8 ), not on VVOL. Test VM is some random Windows VM.
Indeed, when I enable quiescing on the job, transport mode degrades from HotAdd to NBD. Looks like a strange limitation, as quiescing is basically only guest OS VSS snap integration.
-
- VP, Product Management
- Posts: 7074
- Liked: 1507 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
Hi Mihkel,
yes, v12 will automatically fallback (without trying) to NBD-Network mode because of the number of issues we faced and in our lab it did not worked at all.
@All
VMware confirmed that this is not a known product limitation on their side and have requested additional logs for their analyze. So for now it is not clear if it is an issue on the VMware or VVOL storage vendor side. If you want to test if you are affected, you need to test with Veeam Backup & Replication v11 (all versions). Maybe we can nail it down together on some of the vendors or can confirm that more than one vendor is affected. We will do some testing on our side as well and reach out to the storage vendors.
yes, v12 will automatically fallback (without trying) to NBD-Network mode because of the number of issues we faced and in our lab it did not worked at all.
@All
VMware confirmed that this is not a known product limitation on their side and have requested additional logs for their analyze. So for now it is not clear if it is an issue on the VMware or VVOL storage vendor side. If you want to test if you are affected, you need to test with Veeam Backup & Replication v11 (all versions). Maybe we can nail it down together on some of the vendors or can confirm that more than one vendor is affected. We will do some testing on our side as well and reach out to the storage vendors.
-
- Novice
- Posts: 9
- Liked: never
- Joined: Dec 17, 2019 1:58 pm
- Full Name: Anderson Stano Durelli
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
Hi Andreas!
The test should be done on V12, correct? V11 is working as expected.
The test should be done on V12, correct? V11 is working as expected.
-
- VP, Product Management
- Posts: 7074
- Liked: 1507 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
Hi Team, an update from our side. We reached out to VMware and they confirmed that it is not a limitation on their side. We checked various cases with vendors and whenever it did not worked there was a message in the debug logs of VMware that stated from the VVOL vendor that a operation was not supported. We could nail down a vendor and are working with this vendor currently to get some statements from their side.
What we saw in our lab was that ONTAP (Lenovo/NetApp) is working, as well as Pure Storage.
@adurelli test with v11 as v12 has the function deactivated by design. We will correct this soon, but for now for testing use v11 (latest patch).
What we saw in our lab was that ONTAP (Lenovo/NetApp) is working, as well as Pure Storage.
@adurelli test with v11 as v12 has the function deactivated by design. We will correct this soon, but for now for testing use v11 (latest patch).
-
- Novice
- Posts: 9
- Liked: never
- Joined: Dec 17, 2019 1:58 pm
- Full Name: Anderson Stano Durelli
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
Hi Andreas!
Great news. I'm using vVOL with Pure Storage, so I'll be waiting for the update on V12.
I can confirm that Pure storage is working as expected on V11.
Great news. I'm using vVOL with Pure Storage, so I'll be waiting for the update on V12.
I can confirm that Pure storage is working as expected on V11.
-
- VP, Product Management
- Posts: 7074
- Liked: 1507 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
Hi everyone, we have create a bugfix on top of v12 12.0.0.1420 P20230412 (April 14, 2023) and will add it as well to the next cumulative update.
The bugfix is in QA testing and I saw the first successful runs already. So in the next days you can reach out to support to get the bugfix. I will inform you when available.
The bugfix is in QA testing and I saw the first successful runs already. So in the next days you can reach out to support to get the bugfix. I will inform you when available.
-
- Novice
- Posts: 9
- Liked: never
- Joined: Dec 17, 2019 1:58 pm
- Full Name: Anderson Stano Durelli
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
I applied the patch on my environment and jobs are running as expect with vVOL + hot-add + VMware quiescence.
Thanks for the support @Andreas Neufert!
Thanks for the support @Andreas Neufert!
-
- VP, Product Management
- Posts: 7074
- Liked: 1507 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
And the update from support/QA. For those that are affected, please update to latest v12 12.0.0.1420 P20230412 (April 14, 2023) then please open a support ticket and request access to bugfix for bug 513102. We apologize for the inconvenience.
For those that implement the bugfix and the job errors out while processing, please mention here your ticket number and share as well your storage model and firmware number. We can check on our existing list if you storage vendor did not implemented the specific needed VVOL operation. For those where it worked in v11, should not face any issue after applying the patch in v12.
For those that implement the bugfix and the job errors out while processing, please mention here your ticket number and share as well your storage model and firmware number. We can check on our existing list if you storage vendor did not implemented the specific needed VVOL operation. For those where it worked in v11, should not face any issue after applying the patch in v12.
-
- Novice
- Posts: 3
- Liked: never
- Joined: Apr 22, 2023 8:49 am
- Full Name: Wilhelm Schonfeldt
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
upgrading to VEEAM 12 has been an unmitigated disaster. We upgraded from the latest version of 11 to 12 build 1420.
Once upgraded none of our backups can be processed through hotadd using an ESXI proxy if the storage is hosted on a vvol: Case #06022337 support had no solution for us and I ended up having to downgrade back to 11..
Once upgraded none of our backups can be processed through hotadd using an ESXI proxy if the storage is hosted on a vvol: Case #06022337 support had no solution for us and I ended up having to downgrade back to 11..
-
- Novice
- Posts: 9
- Liked: never
- Joined: Dec 17, 2019 1:58 pm
- Full Name: Anderson Stano Durelli
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
Hello @wilhelmsch26!
Same case here. But VEEAM release a patch for this. As Andreas reported, you should open a ticket and request patch for bug 513102
Same case here. But VEEAM release a patch for this. As Andreas reported, you should open a ticket and request patch for bug 513102
-
- Novice
- Posts: 3
- Liked: never
- Joined: Jun 25, 2020 12:05 am
- Contact:
Re: VEEAM V12 do not support HotAdd and quiescence with vVOLs
Hi. Does anyone know if this bug still exists in 12.1.2.172? We are experiencing the exact same problem and support just told me to stop using vmtools quiescence, which is really not a good solution. Case #03183605.
Who is online
Users browsing this forum: No registered users and 24 guests