Discussions specific to the Nutanix AHV hypervisor
Post Reply
wa15
Expert
Posts: 312
Liked: 24 times
Joined: Jan 02, 2014 4:45 pm
Contact:

Veeam Nutanix AHV backup woes

Post by wa15 »

We have been Veeam users since B&R v6, mostly in a vSphere environment. Last year we started backing up some AHV clusters with Veeam's AHV backup, and quiet frankly, this has been quiet a terrible administration experience.

I have opened up a few tickets on this, but so far the issues are:

1. The proxy stops talking to a cluster. Worked with both Nutanix and Veeam support, and so far no solution. Here is hoping the upgrade from V2.0 to V2.1 fixes this.

2. Backup jobs randomly hang. Just today I saw a job that had been running for 7 days and stuck at 36%.

3. I couldn't stop said job from the B&R console, proxy console, or even shutting down the AHV proxy. It kept running.

4. Trying to upgrade from 2.0 to 2.1 while on V10 didn't work. So I thought we might as well upgrade to upgrade to V11 so see if I can upgrade that way. That didn't help either as we are now seeing backup files can't be found from upgrading from the proxy (there is a thread on this here)

5. Veeam AHV support response is slow compared to vSphere support. I opened a priority 2 ticket on Thursday. Support agent asked when I will be available for a remote session, I said Friday. Didn't hear back so I called on Friday asking to speak with support. I was told they will be in touch. I didn't hear back until I received an email on Sunday (which I responded to on Sunday). I escalated the case on Monday, and am supposed to chat with support on Tuesday. I was also told that Veeam AHV support team doesn't have a phone queue I can get transferred to. My experience is that Veeam support for vSphere is just faster.

6. Looking at the new features list for major versions, it's quiet clear that Veeam AHV is considered "second class" as the number/depth of features is miniscule compared to vSphere versions. I realize that AHV adoption is nowhere near as common as vSphere, but I feel that if Veeam is advertising that they support AHV and "is committed to delivering the best availability for applications and workloads running on AHV" (taken from the product page), then the software should deliver accordingly and it is not. What doesn't help is the ever increasing cost of support.

There is also lack of feature parity between AHV and vSphere (e.g. GFS) but I am discounting that as I know the product is in it's early stages. HOWEVER, I do experience the advertised features to work properly. All in all, I feel that we have a product deployed that feels like it's in alpha stages. From the seemingly small bugs (backup progress showing zero in main B&R console while the job is running) to the more critical issues, the product just doesn't seem ready for prime time.

Is anyone else having similar experiences?

HannesK
Veeam Software
Posts: 7895
Liked: 1374 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Veeam Nutanix AHV backup woes

Post by HannesK »

Hello,
can you please post the case numbers you are talking about? Support not answering sounds strange.

Sure, there are differences in AHV and vSphere functionality. Some of it is because AHV has many limitations that VMware doesn't have (for example external datastores with NFS).

And sure, we also focus on the vendor with the highest marked share (VMware). I didn't find current numbers on market share, but globally AHV is a niche player. AHV has a good marked share for hyper-converged systems. But hyper-converged systems only have a small market share.

Best regards,
Hannes

tedsteenvoorden
Enthusiast
Posts: 38
Liked: never
Joined: Apr 21, 2011 4:53 pm
Full Name: Ted
Contact:

Re: Veeam Nutanix AHV backup woes

Post by tedsteenvoorden »

Hello wa15,

I am currently in the preparation of migrating an environment to Nutanix AHV and Veeam for Nutanix AHV. Having years of experience with Veeam for vSphere, I must say I am a little disappointed by the Veeam for Nutanix AHV product. It has quite a few limitations:

General
-The weird design choice of multiple management interfaces (Veeam MMC and AHV proxy Web Interface). In the Veeam Backup & Replication console, you cannot create or edit jobs of AHV Backup Proxy.

Backup
-Synthetic full backups are not supported for AHV VM backups.
-SureBackup is not supported for backups created by AHV Backup Proxy.
-You can add only deploy one AHV Backup Proxy per Nutanix AHV cluster.
-Instant VM Recovery is not supported for AHV hosts/clusters.
-Only support for per-VM backup chains.
-"After this job" schedule is not available for AHV Backup Proxy jobs.
-You cannot disable deduplication and compression of backups.
-Health check of backup files is not supported for backups created by AHV Backup Proxy
-Veeam Cloud Connect repositories are not supported for backups created by AHV Backup Proxy.

Backup Copy
-The Immediate Copy mode is not supported for backups created with AHV Backup Proxy.

Guest OS integration
-There is no Guest OS integration for hypervisor backups: No guest OS file indexing, no application aware processing from Veeam.
-Application aware processing is depended on Nutanix application consistent snapshots. This is limited in functionality compared to Veeam application processing (e.g. no log file truncation).

I have no experience with the stability of the product yet, the migrating has not yet been started. I will share my experiences as soon as we start using the product.

HannesK
Veeam Software
Posts: 7895
Liked: 1374 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Veeam Nutanix AHV backup woes

Post by HannesK » 1 person likes this post

Hello,
sure, there are open feature requests compared to VMware. VMware has by far the largest market share and most features. Hyper-V already has much less. And Nutanix is a newcomer from a global point of view. There must still be room for improvement in a version 2 product :-)

A few things to note:
-SureBackup is not supported for backups created by AHV Backup Proxy.
Instant VM Recovery is not supported for AHV hosts/clusters.
Nutanix doesn't allow external datastores. That means: no instant VM recovery. That means no Sure Backup. There are rumors that Nutanix might allow it for backup purposes, but I have no timeline for that.
You can add only deploy one AHV Backup Proxy per Nutanix AHV cluster.
do you have more details how large your cluster is? Amount of data, backup window? Most Nutanix installations are so small, that one proxy is enough. If you have performance issues, please open a support case to get that fixed.
Only support for per-VM backup chains.
That's good I think. Per job-chains only have disadvantages. (no, the little space saving is irrelevant with todays storage prices)
You cannot disable deduplication and compression of backups.
uhm, what is the idea of doing that? You can always decompress on the repository level if you have a not-recommended primary backup storage (I'm talking about inline dedupe appliances).https://helpcenter.veeam.com/docs/backu ... ml?ver=110 "Decompress backup data blocks before storing "

"After this job" schedule is not available for AHV Backup Proxy jobs.
it's a bad practice. what is your use case? starting all your jobs for example at 10:00pm, 10:01pm, 10:02pm does almost same without creating ugly dependency chains.
-Veeam Cloud Connect repositories are not supported for backups created by AHV Backup Proxy.
reading your other questions I'm surprised that you want to back up directly to cloud connect. Because cloud connect for example also doesn't support instant VM recovery. What do you plan in your environment?
The Immediate Copy mode is not supported for backups created with AHV Backup Proxy.
Agree that immediate copy mode is easier... but which scenario are you trying to cover where the periodic mode is impossible (nothing comes into my mind right now)

Best regards,
Hannes

Mildur
Veeam Legend
Posts: 800
Liked: 387 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian
Location: Switzerland
Contact:

Re: Veeam Nutanix AHV backup woes

Post by Mildur »

I am missing mostly the synthetic Full Backups for Nutanix. And the Application Aware Processing.
And of course Powershell integration for Nutanix Restore Points in the VBR Console. :lol:

We are planning to migrate some workload to Nutanix in the next 12 months. That will be around 200-250 VMs.
For Applications and File System Indexing, I have todo Agent Backups now :) It would be nice that the Application Aware Agent can be copied to Nutanix VMs too prior the Snapshot process.

Besides that, Backups and restores of Nutanix are working fine. I‘m a little worried with the performance about a entire cluster restore. Are there any whitepapers about the restore performance when you have todo restores for 20-30 vms at the same time? I see the Proxy as a bottleneck here. I need to test that as soon I have my new backup environment.
VMCE 9 | Veeam Legends 2021
Working with Veeam since 2017 for a VCSP in Switzerland
—————————————————————
VeeamOn 2021: https://www.veeam.com/veeamon

HannesK
Veeam Software
Posts: 7895
Liked: 1374 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Veeam Nutanix AHV backup woes

Post by HannesK » 1 person likes this post

with restore of 20-30 VMs in parallel, I assume the storage to become a bottleneck.

I'm not aware of performance whitepapers for AHV and Veeam. If you see less than 1 GByte/s speed in your tests, please open a case to check out the bottleneck :-)

Mildur
Veeam Legend
Posts: 800
Liked: 387 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian
Location: Switzerland
Contact:

Re: Veeam Nutanix AHV backup woes

Post by Mildur »

Ok, thanks. :) Our new backup environment will be available to me in two months.
We will test Max Throughput's then.
VMCE 9 | Veeam Legends 2021
Working with Veeam since 2017 for a VCSP in Switzerland
—————————————————————
VeeamOn 2021: https://www.veeam.com/veeamon

Gostev
SVP, Product Management
Posts: 28291
Liked: 5029 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Nutanix AHV backup woes

Post by Gostev »

All I can say is, our V2 AHV support is far ahead of what our V2 vSphere support was in terms of features and functionality :D

But of course, there's no point in comparing 14 years old vSphere support with 2 years old AHV support in terms of depth and breadths of functionality. And it is very hard to not be disappointed coming from many years of Veeam+vSphere to Veeam+AHV :D so I just hope you all have a very good reason to do this!

wesprather
Novice
Posts: 7
Liked: never
Joined: Jul 06, 2018 2:53 am
Full Name: Wesley Prather
Contact:

Re: Veeam Nutanix AHV backup woes

Post by wesprather »

I just want to *bump* this thread a little...
I'm working on a project to move from HyperV to Nutanix AHV, but hate to lose advanced functionality of Veeam Virtual Labs & SureBackup.
Even in v2.1 we stand to lose it.
It'd be great to see this open up. I've got Veeam V11 & Nutanix CE (AHV) running in my lab in case there happens to be early access to evaluate... ;-)
Thanks.
-wes

Gostev
SVP, Product Management
Posts: 28291
Liked: 5029 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Nutanix AHV backup woes

Post by Gostev »

As it was already explained above by Hannes, SureBackup is technically impossible on AHV due to a hypervisor limitation.

wa15
Expert
Posts: 312
Liked: 24 times
Joined: Jan 02, 2014 4:45 pm
Contact:

Re: Veeam Nutanix AHV backup woes

Post by wa15 »

Thanks for the discussion all.

@HannesK, the first case I opened was 04768475. This is where I called Veeam support to speak to an engineer and was told AHV support team does not have a phone queue. Part of the issue was also that the case was assigned to the EMEA region though I am US based.

I encountered another issue after the V11 upgrade and opened case 04783623. I ended up escalating this ticket. The support tech I worked with was very good here and resolved the issues on the Veeam side. At this point, waiting for Nutanix support.

HannesK
Veeam Software
Posts: 7895
Liked: 1374 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Veeam Nutanix AHV backup woes

Post by HannesK »

Hello,
support can make remote sessions if necessary (I see webex invitations in the case communication, so that should be fine). Calling support makes little sense in general, as it's complicated to submit logfiles via phone :-)

The EMEA region might have happened because your company headquarter seems to be in EMEA (at least that's where the licenses are registered as far as I see). You can always ask support to transfer the case to your geo. When you open a case, you should also be able to select the geo, but with production support it's "follow the sun" to meet SLAs... so that could also be the reason.

Best regards,
Hannes

tedsteenvoorden
Enthusiast
Posts: 38
Liked: never
Joined: Apr 21, 2011 4:53 pm
Full Name: Ted
Contact:

Re: Veeam Nutanix AHV backup woes

Post by tedsteenvoorden »

HannesK wrote: May 07, 2021 6:31 am Hello,
sure, there are open feature requests compared to VMware. VMware has by far the largest market share and most features. Hyper-V already has much less. And Nutanix is a newcomer from a global point of view. There must still be room for improvement in a version 2 product :-)

A few things to note:

Nutanix doesn't allow external datastores. That means: no instant VM recovery. That means no Sure Backup. There are rumors that Nutanix might allow it for backup purposes, but I have no timeline for that.

do you have more details how large your cluster is? Amount of data, backup window? Most Nutanix installations are so small, that one proxy is enough. If you have performance issues, please open a support case to get that fixed.

That's good I think. Per job-chains only have disadvantages. (no, the little space saving is irrelevant with todays storage prices)

uhm, what is the idea of doing that? You can always decompress on the repository level if you have a not-recommended primary backup storage (I'm talking about inline dedupe appliances).https://helpcenter.veeam.com/docs/backu ... ml?ver=110 "Decompress backup data blocks before storing "

it's a bad practice. what is your use case? starting all your jobs for example at 10:00pm, 10:01pm, 10:02pm does almost same without creating ugly dependency chains.

reading your other questions I'm surprised that you want to back up directly to cloud connect. Because cloud connect for example also doesn't support instant VM recovery. What do you plan in your environment?

Agree that immediate copy mode is easier... but which scenario are you trying to cover where the periodic mode is impossible (nothing comes into my mind right now)

Best regards,
Hannes
Hello Hannes,

Thanks for the feedback. Coming from the Veeam vSphere version, the Nutanix version takes some getting used to in terms of feature set. I agree, some point are minor things, the most missing things are Synthetic Full Backups, SureBackup (I know, it's an AHV thing) and Application Aware Processing. And I don't like the split in management interfaces.

Feedback to your questions:

- Immediate Copy mode: We like to start the Backup Copy to the other site, as soon as new backup files are being made.
- One AHV Backup Proxy per Nutanix AHV cluster: It's more that with multiple proxies the processing load (e.g. network) can be distributed over multiple hosts.

Best regards,
Ted

HannesK
Veeam Software
Posts: 7895
Liked: 1374 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Veeam Nutanix AHV backup woes

Post by HannesK »

We like to start the Backup Copy to the other site, as soon as new backup files are being made.
if you start the backup copy job for example 5min before your backups jobs start, then this will be accomplished (assuming that you have a "normal" schedule of your backup jobs)

For the network load: Agree that load distribution would be good in general. Just to be sure: you are hitting a 10G / 20G / 40G limit of your Nutanix hosts regularly?

wa15
Expert
Posts: 312
Liked: 24 times
Joined: Jan 02, 2014 4:45 pm
Contact:

Re: Veeam Nutanix AHV backup woes

Post by wa15 »

One more situation we just ran into: we needed to rebuild one of the Veeam AHV proxies, and we couldn't export the proxy config from the old proxy(and we don't have a backup/snapshot of the proxy). So we deployed a new Veeam AHV proxy and set up the job, pointing the job to the same repository as the old proxy with backup files. But how do we go about mapping the backup job to existing backup files? This is something you can do in vSphere. Does Veeam AHV pick up the existing backups automatically and continue the backup chain? Or does it start a new chain? In our case it seems like it started a new chain.

ronnmartin61
Veeam Software
Posts: 186
Liked: 57 times
Joined: Mar 07, 2016 3:55 pm
Full Name: Ronn Martin
Contact:

Re: Veeam Nutanix AHV backup woes

Post by ronnmartin61 »

There is an import (essentially rescan) capability that can be run from the proxy console see https://helpcenter.veeam.com/docs/van/u ... tml?ver=21 however if memory serves it will only turn up visibility to the existing backups and not "map" them...

wa15
Expert
Posts: 312
Liked: 24 times
Joined: Jan 02, 2014 4:45 pm
Contact:

Re: Veeam Nutanix AHV backup woes

Post by wa15 »

Thanks, I noticed that. And correct, it seems it only turns up existing backups but doesn't map to an existing backup file chain. Can we get another feature request please? :)

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 2 guests