Veeam Nutanix AHV backup woes

[wa15]

We have been Veeam users since B&R v6, mostly in a vSphere environment. Last year we started backing up some AHV clusters with Veeam's AHV backup, and quiet frankly, this has been quiet a terrible administration experience.

I have opened up a few tickets on this, but so far the issues are:

1. The proxy stops talking to a cluster. Worked with both Nutanix and Veeam support, and so far no solution. Here is hoping the upgrade from V2.0 to V2.1 fixes this.

2. Backup jobs randomly hang. Just today I saw a job that had been running for 7 days and stuck at 36%.

3. I couldn't stop said job from the B&R console, proxy console, or even shutting down the AHV proxy. It kept running.

4. Trying to upgrade from 2.0 to 2.1 while on V10 didn't work. So I thought we might as well upgrade to upgrade to V11 so see if I can upgrade that way. That didn't help either as we are now seeing backup files can't be found from upgrading from the proxy (there is a thread on this here)

5. Veeam AHV support response is slow compared to vSphere support. I opened a priority 2 ticket on Thursday. Support agent asked when I will be available for a remote session, I said Friday. Didn't hear back so I called on Friday asking to speak with support. I was told they will be in touch. I didn't hear back until I received an email on Sunday (which I responded to on Sunday). I escalated the case on Monday, and am supposed to chat with support on Tuesday. I was also told that Veeam AHV support team doesn't have a phone queue I can get transferred to. My experience is that Veeam support for vSphere is just faster.

6. Looking at the new features list for major versions, it's quiet clear that Veeam AHV is considered "second class" as the number/depth of features is miniscule compared to vSphere versions. I realize that AHV adoption is nowhere near as common as vSphere, but I feel that if Veeam is advertising that they support AHV and "is committed to delivering the best availability for applications and workloads running on AHV" (taken from the product page), then the software should deliver accordingly and it is not. What doesn't help is the ever increasing cost of support.

There is also lack of feature parity between AHV and vSphere (e.g. GFS) but I am discounting that as I know the product is in it's early stages. HOWEVER, I do experience the advertised features to work properly. All in all, I feel that we have a product deployed that feels like it's in alpha stages. From the seemingly small bugs (backup progress showing zero in main B&R console while the job is running) to the more critical issues, the product just doesn't seem ready for prime time.

Is anyone else having similar experiences?

[HannesK]

Hello,
can you please post the case numbers you are talking about? Support not answering sounds strange.

Sure, there are differences in AHV and vSphere functionality. Some of it is because AHV has many limitations that VMware doesn't have (for example external datastores with NFS).

And sure, we also focus on the vendor with the highest marked share (VMware). I didn't find current numbers on market share, but globally AHV is a niche player. AHV has a good marked share for hyper-converged systems. But hyper-converged systems only have a small market share.

Best regards,
Hannes

[tedsteenvoorden]

Hello wa15,

I am currently in the preparation of migrating an environment to Nutanix AHV and Veeam for Nutanix AHV. Having years of experience with Veeam for vSphere, I must say I am a little disappointed by the Veeam for Nutanix AHV product. It has quite a few limitations:

General
-The weird design choice of multiple management interfaces (Veeam MMC and AHV proxy Web Interface). In the Veeam Backup & Replication console, you cannot create or edit jobs of AHV Backup Proxy.

Backup
-Synthetic full backups are not supported for AHV VM backups.
-SureBackup is not supported for backups created by AHV Backup Proxy.
-You can add only deploy one AHV Backup Proxy per Nutanix AHV cluster.
-Instant VM Recovery is not supported for AHV hosts/clusters.
-Only support for per-VM backup chains.
-"After this job" schedule is not available for AHV Backup Proxy jobs.
-You cannot disable deduplication and compression of backups.
-Health check of backup files is not supported for backups created by AHV Backup Proxy
-Veeam Cloud Connect repositories are not supported for backups created by AHV Backup Proxy.

Backup Copy
-The Immediate Copy mode is not supported for backups created with AHV Backup Proxy.

Guest OS integration
-There is no Guest OS integration for hypervisor backups: No guest OS file indexing, no application aware processing from Veeam.
-Application aware processing is depended on Nutanix application consistent snapshots. This is limited in functionality compared to Veeam application processing (e.g. no log file truncation).

I have no experience with the stability of the product yet, the migrating has not yet been started. I will share my experiences as soon as we start using the product.

[HannesK]

Hello,
sure, there are open feature requests compared to VMware. VMware has by far the largest market share and most features. Hyper-V already has much less. And Nutanix is a newcomer from a global point of view. There must still be room for improvement in a version 2 product

A few things to note:

-SureBackup is not supported for backups created by AHV Backup Proxy.
Instant VM Recovery is not supported for AHV hosts/clusters.

Nutanix doesn't allow external datastores. That means: no instant VM recovery. That means no Sure Backup. There are rumors that Nutanix might allow it for backup purposes, but I have no timeline for that.

You can add only deploy one AHV Backup Proxy per Nutanix AHV cluster.

do you have more details how large your cluster is? Amount of data, backup window? Most Nutanix installations are so small, that one proxy is enough. If you have performance issues, please open a support case to get that fixed.

Only support for per-VM backup chains.

That's good I think. Per job-chains only have disadvantages. (no, the little space saving is irrelevant with todays storage prices)

You cannot disable deduplication and compression of backups.

uhm, what is the idea of doing that? You can always decompress on the repository level if you have a not-recommended primary backup storage (I'm talking about inline dedupe appliances).https://helpcenter.veeam.com/docs/backu ... ml?ver=110 "Decompress backup data blocks before storing "

"After this job" schedule is not available for AHV Backup Proxy jobs.

it's a bad practice. what is your use case? starting all your jobs for example at 10:00pm, 10:01pm, 10:02pm does almost same without creating ugly dependency chains.

-Veeam Cloud Connect repositories are not supported for backups created by AHV Backup Proxy.

reading your other questions I'm surprised that you want to back up directly to cloud connect. Because cloud connect for example also doesn't support instant VM recovery. What do you plan in your environment?

The Immediate Copy mode is not supported for backups created with AHV Backup Proxy.

Agree that immediate copy mode is easier... but which scenario are you trying to cover where the periodic mode is impossible (nothing comes into my mind right now)

Best regards,
Hannes

[Mildur]

I am missing mostly the synthetic Full Backups for Nutanix. And the Application Aware Processing.
And of course Powershell integration for Nutanix Restore Points in the VBR Console.

We are planning to migrate some workload to Nutanix in the next 12 months. That will be around 200-250 VMs.
For Applications and File System Indexing, I have todo Agent Backups now It would be nice that the Application Aware Agent can be copied to Nutanix VMs too prior the Snapshot process.

Besides that, Backups and restores of Nutanix are working fine. I‘m a little worried with the performance about a entire cluster restore. Are there any whitepapers about the restore performance when you have todo restores for 20-30 vms at the same time? I see the Proxy as a bottleneck here. I need to test that as soon I have my new backup environment.

[HannesK]

with restore of 20-30 VMs in parallel, I assume the storage to become a bottleneck.

I'm not aware of performance whitepapers for AHV and Veeam. If you see less than 1 GByte/s speed in your tests, please open a case to check out the bottleneck

[Mildur]

Ok, thanks. Our new backup environment will be available to me in two months.
We will test Max Throughput's then.

[Gostev]

All I can say is, our V2 AHV support is far ahead of what our V2 vSphere support was in terms of features and functionality

But of course, there's no point in comparing 14 years old vSphere support with 2 years old AHV support in terms of depth and breadths of functionality. And it is very hard to not be disappointed coming from many years of Veeam+vSphere to Veeam+AHV so I just hope you all have a very good reason to do this!

[wesprather]

I just want to *bump* this thread a little...
I'm working on a project to move from HyperV to Nutanix AHV, but hate to lose advanced functionality of Veeam Virtual Labs & SureBackup.
Even in v2.1 we stand to lose it.
It'd be great to see this open up. I've got Veeam V11 & Nutanix CE (AHV) running in my lab in case there happens to be early access to evaluate...
Thanks.
-wes

[Gostev]

As it was already explained above by Hannes, SureBackup is technically impossible on AHV due to a hypervisor limitation.

[wa15]

Thanks for the discussion all.

@HannesK, the first case I opened was 04768475. This is where I called Veeam support to speak to an engineer and was told AHV support team does not have a phone queue. Part of the issue was also that the case was assigned to the EMEA region though I am US based.

I encountered another issue after the V11 upgrade and opened case 04783623. I ended up escalating this ticket. The support tech I worked with was very good here and resolved the issues on the Veeam side. At this point, waiting for Nutanix support.

[HannesK]

Hello,
support can make remote sessions if necessary (I see webex invitations in the case communication, so that should be fine). Calling support makes little sense in general, as it's complicated to submit logfiles via phone

The EMEA region might have happened because your company headquarter seems to be in EMEA (at least that's where the licenses are registered as far as I see). You can always ask support to transfer the case to your geo. When you open a case, you should also be able to select the geo, but with production support it's "follow the sun" to meet SLAs... so that could also be the reason.

Best regards,
Hannes

[tedsteenvoorden]

Hello,
sure, there are open feature requests compared to VMware. VMware has by far the largest market share and most features. Hyper-V already has much less. And Nutanix is a newcomer from a global point of view. There must still be room for improvement in a version 2 product

A few things to note:

Nutanix doesn't allow external datastores. That means: no instant VM recovery. That means no Sure Backup. There are rumors that Nutanix might allow it for backup purposes, but I have no timeline for that.

do you have more details how large your cluster is? Amount of data, backup window? Most Nutanix installations are so small, that one proxy is enough. If you have performance issues, please open a support case to get that fixed.

That's good I think. Per job-chains only have disadvantages. (no, the little space saving is irrelevant with todays storage prices)

uhm, what is the idea of doing that? You can always decompress on the repository level if you have a not-recommended primary backup storage (I'm talking about inline dedupe appliances).https://helpcenter.veeam.com/docs/backu ... ml?ver=110 "Decompress backup data blocks before storing "

it's a bad practice. what is your use case? starting all your jobs for example at 10:00pm, 10:01pm, 10:02pm does almost same without creating ugly dependency chains.

reading your other questions I'm surprised that you want to back up directly to cloud connect. Because cloud connect for example also doesn't support instant VM recovery. What do you plan in your environment?

Agree that immediate copy mode is easier... but which scenario are you trying to cover where the periodic mode is impossible (nothing comes into my mind right now)

Best regards,
Hannes

Hello Hannes,

Thanks for the feedback. Coming from the Veeam vSphere version, the Nutanix version takes some getting used to in terms of feature set. I agree, some point are minor things, the most missing things are Synthetic Full Backups, SureBackup (I know, it's an AHV thing) and Application Aware Processing. And I don't like the split in management interfaces.

Feedback to your questions:

- Immediate Copy mode: We like to start the Backup Copy to the other site, as soon as new backup files are being made.
- One AHV Backup Proxy per Nutanix AHV cluster: It's more that with multiple proxies the processing load (e.g. network) can be distributed over multiple hosts.

Best regards,
Ted

[HannesK]

We like to start the Backup Copy to the other site, as soon as new backup files are being made.

if you start the backup copy job for example 5min before your backups jobs start, then this will be accomplished (assuming that you have a "normal" schedule of your backup jobs)

For the network load: Agree that load distribution would be good in general. Just to be sure: you are hitting a 10G / 20G / 40G limit of your Nutanix hosts regularly?

[wa15]

One more situation we just ran into: we needed to rebuild one of the Veeam AHV proxies, and we couldn't export the proxy config from the old proxy(and we don't have a backup/snapshot of the proxy). So we deployed a new Veeam AHV proxy and set up the job, pointing the job to the same repository as the old proxy with backup files. But how do we go about mapping the backup job to existing backup files? This is something you can do in vSphere. Does Veeam AHV pick up the existing backups automatically and continue the backup chain? Or does it start a new chain? In our case it seems like it started a new chain.

[ronnmartin61]

There is an import (essentially rescan) capability that can be run from the proxy console see https://helpcenter.veeam.com/docs/van/u ... tml?ver=21 however if memory serves it will only turn up visibility to the existing backups and not "map" them...

[wa15]

Thanks, I noticed that. And correct, it seems it only turns up existing backups but doesn't map to an existing backup file chain. Can we get another feature request please?

[garethcayman]

All I can say is, our V2 AHV support is far ahead of what our V2 vSphere support was in terms of features and functionality

But of course, there's no point in comparing 14 years old vSphere support with 2 years old AHV support in terms of depth and breadths of functionality. And it is very hard to not be disappointed coming from many years of Veeam+vSphere to Veeam+AHV so I just hope you all have a very good reason to do this!

Honestly I'm at my wits end with Veeam. Are these unrealistic Expectations? I don't think so. We've been using Veeam for Hyper-V and ESX forever and these are expected minimum feature set. Unless the vendor can't support it like Instant Restore...

My experience in the last year with Azure Backup appliance and now Nutanix AHV has been so disappointing and I've wasted so much time. If I'd known before how immature the products were I would have recommended a different backup product. These other products should come with warning labels of all the features you've come to expect from Veeam, but are not included.

All of these features like log truncation, synthetic fulls, etc should be default out the door. Azure Backup appliance was terrible. I haven't tried the latest version because I had to delete the install and switch to Veeam Agent and I haven't had a chance to try again.

Nutanix AHV is Alpha release as mentioned. It's poor. I don't care if its v2 or not, come on Veeam, pull your socks up.

AWS Appliance v3 was decent, thankfully that just got released before I deployed it for the first time but you're cheapening your brand when you release these incomplete products to customers and we have to wait 3 versions before we get a decent product? Feels like Microsoft, rolling out new incomplete products so their clients can test and improve it for them.

Excuse me while I now go figure out how to enable log truncation for 26 SQL servers that Veeam was managing fine on Hyper-V but now doesn't work on Nutanix AHV.

[Mildur]

The warning is in the guide:

https://helpcenter.veeam.com/docs/van/u ... tml?ver=21

My company was aware of all the woes before we decided to buy Veeam Backup for Nutanix.

Excuse me while I now go figure out how to enable log truncation for 26 SQL servers that Veeam was managing fine on Hyper-V but now doesn't work on Nutanix AHV.

Use Veeam Agent.
You can directly restore an Agent backup back as a vm to Nutanix. Without the recovery Iso

[Gostev]

AWS Appliance v3 was decent, thankfully that just got released before I deployed it for the first time but you're cheapening your brand when you release these incomplete products to customers and we have to wait 3 versions before we get a decent product?

I can give you some theory behind our Product Management approaches. People write books about this, but I will try to fit it in a few lines.

The choice we face is really simple:

1. Release V1-class minimum viable product immediately, and mature it for the next 3 years (both features and engine).

2. No product at all for 3 years, then release V3-class aka "decent" product right away (with lots of features but immature engine).

The first approach has numerous benefits for us, for example:

a) For us it's not only about features, but even more so about the engine (its reliability and scalability). With our approach, in 3 years we will have an engine that is polished in the real-world installations AND we will have features. While with the other approach, we will have lots of features but on V1 engine that still has 3 more years to go through all the teething issues in real-world environments, like data losses and critical security vulnerabilities.

b) We would miss 3 years of feedback with customers telling us what features are really important, instead building something WE think is important (which is usually incorrect).

c) We won't potentially waste 3 years of R&D on something that gets no tractions (we've discontinued products before).

Now may be you can tell me, how you as a customer would benefit if Veeam practiced the 2nd approach instead?

[garethcayman]

Understood that makes sense, I need to be more careful of new products but we really don't want to step outside the Veeam ecosystem.

Please keep the releases coming; hopefully some of these newer product lines get a more aggressive release cadence.

When is the next version of Veeam Nutanix AHV due to release? Can you share anything on the roadmap?

Looking forward to v3 already.

[dpeach01]

My experience is the same. Currently, I have a PPRI 1 case open for about 3 weeks because a patch they provided for my AHV Proxy to address a UTC time zone issue broke my proxy. Now it freezes at various random times during the day and night. There is no failure, the GUI still responds, and jobs do not fail. They just freeze. there is no way to monitor for these failures, I just have to log into B&R periodically through the day and night to check and reboot the proxy from Prism if necessary, then rerun all failed jobs. Support wont engage me to look at he issue or offer workarounds. I have been unable to have a single session with support sincde this case was escalated 3 weeks ago. Management simply tells me they are aware of a global issue and are working on it. The AHV product is not ready for production workloads. yes, Veeam for VSphere is a solid product and I have been using it for 12 years, but VAN has really become an embarrassment for this company.

[Mildur]

Mine too. Same error with timezone. New build freezed every morning. We were working 3-4 weeks on this issue with veeam support. Daily Reboots at 8am has helped with the issue, but that should not be.
Case #04882207

We are now using a new vbr environment with the old proxy build. This was already planned for this summer.

[tedsteenvoorden]

Since I am implementing Veeam for AHV within a couple of weeks, I am very curious which build version of the AHV proxy has this problem?

[Mildur]

The issue with the freezing was build nr 2.1.451.
Today, If you have Veeam V11 and the most current patch, you will get a lower build number (2.1.396) when you deploy or upgrade the proxy. This build do not have the freezing issue.

[tedsteenvoorden]

Thanx Mildur for the info. I hope Veeam support gets the problem fixed soon.

[Mildur]

There is a privat build (2.1.467) now for this issue, if you are on 2.1.451 and have the freezing issue. It should be resolved in this build. I cannot test it because we have downgraded the proxy already.

You can get it, if you open a Support Case for the freezing issue.

[tedsteenvoorden]

Mine too. Same error with timezone. New build freezed every morning. We were working 3-4 weeks on this issue with veeam support. Daily Reboots at 8am has helped with the issue, but that should not be.
Case #04882207

We are now using a new vbr environment with the old proxy build. This was already planned for this summer.

For my understanding, does this time zone issue relate to backup scheduling times not consistent between the AHV Proxy and the Veeam Management Console (e.g. it is not corrected for the timezone)? I am currently experiencing this behavior.

[Mildur]

With our old implementation, we had a timezone issue, that the backups job wasn't starting at the correct time.
They always started 1 hour to late because timezone was not saved correctly and always keep resetting to the default.

Now with the new appliance, the timezone is correct in the settings.
The backup are starting at the correct time.

The only thing, which doesn't work, is that in the job summary in the vbr console, the nutanix ahv backup show the time in the UTC+0 format.

[PTide]

the nutanix ahv backup show the time in the UTC+0 format.

You mean "Next run" time and "Last run" time?

Thanks!