AOS 6.8 Veeam AHV Support/Interoperability/Validation

[HYF_JE]

Figured I'd ask this question while I'm already logged in:

I've heard through our Nutanix account team that the next LTS of AOS ( 6.8 ) is close to being released. I wanted to ask/get a feel for if the latest version of the AHV plugin/appliance is all set and ready to go for this next LTS, if Veeam engineering/R&D has access to private builds of the latest AOS from Nutanix for testing, etc etc.

I'm a relatively new Nutanix admin and haven't been through an upgrade like this before and while I suspect it will be pretty mild going from one LTS to the next, backups in our enterprise are tablestakes so if Veeam and Nutanix aren't 100% in sync with this release, that will influence how fast we pursue this upgrade.

[ronnmartin61]

Yes that is our understanding as well. Generally speaking there should be no issues and we've only very rarely seen any of the Nutanix API's we require break in upgrade situations. We do not have 6.8 testing scheduled at the moment but of course will run validation as time allows. We'd recommend that you monitor our helpcenter documentation https://helpcenter.veeam.com/docs/vbahv ... tml?ver=50 which will be updated once we confirm compatibility.

[HYF_JE]

I see AOS 6.8 shows a product release as of today. I haven't delved further into it, but I recommend Veeam do so.

[ronnmartin61]

Yes we're aware and will work in validation testing as our scheduling permits

[ronnmartin61]

@HYF_JE just FYI we have performed some preliminary testing with AOS 6.8 and it appears it introduces a bug affecting restore operations. For the time being we recommend delaying the upgrade but will keep you posted on our progress to resolve this with Nutanix.

[JaySt]

interested in this as well. I'm seeing the restore issues here with 6.8 as well. any ETA for a fix?

Failed to check VM exist: Invalid characters (CR/LF) in header Exception

[ronnmartin61]

@JaySt yes that's precisely the issue. They changed the response body we receive and parse for that particular API call which has caused the issue. We know Nutanix is treating this as a priority one issue but do not have an estimate on a fix from them as yet. I'll update this thread as soon as we know. In the meantime we've updated our AHV helpcenter documentation to explicitly state we support up to AOS 6.7

[JaySt]

ok thanks for the update. so the fix/patch needs to come from Nutanix? or will Veeam provide a patch?

[ronnmartin61]

@JaySt yes in fact just off the phone with the Nutanix product management team and their engineers are actively working on the fix. Unfortunately they're not able to give us an estimate on delivery timing just yet however...

[JaySt]

any news on this one perhaps?

[ronnmartin61]

@JaySt not yet but hope to get some sort of status from Nutanix by the end of the week

[ronnmartin61]

@JaySt we did speak to Nutanix yesterday however nothing new to report as yet...

[JaySt]

got a case running (07314435) because things were failing big time when the certificate got updated on nutanix clusters and the appliances were not updating/accepting the cert.
i'm not that sure yet, but the appliances not updating the cert might have some relation to the 6.8 communication problems as i see some log messages that might suggest that. Relayed that to support.
More of a FYI

[JulienBubrecht]

@JaySt we did speak to Nutanix yesterday however nothing new to report as yet...

Any news on this topic ?

[ronnmartin61]

@JulienBubrecht yes I did receive word from Nutanix earlier this week that they had identified the root cause of the issue and were working on a fix

[JaySt]

have to say, they do seem to take their time for a priority one issue...
doesn't seem so hard to fix.

[flow90]

I have opened a ticket about this on Nutanix Support 2 weeks ago. Issue has been identified and would be fixed in a 6.8.x release but no timeline yet...

I hope they put a high priority on this issue...

[AbsoluteBS]

Hello - I am foaming at the mouth to finally get 6.8 up and running. I as well, have a support case open with Nutanix. Has anyone heard a status update on this?

Ive bugged our rep, but havent heard anything from them yet on this topic.

Best

[ronnmartin61]

Hey all, just wanted to drop an update (though an unsatisfying one) on this. The good news is that we're pre-engineering support for a more rational HTTP response from the broken API per Nutanix engineering's advice that we will include with our next AHV plugin release. The less-than-good news is that we still don't have a committed timeframe from Nutanix on the fix availability.

[JaySt]

I dont understand.
"we" in this case being Veeam & Nutanix ? Or just Veeam? Who is fixing what now actually? Or is Veeam just providing a workaround within the next update to not break on the still broken Nutanix API?

[ronnmartin61]

@JaySt to be clear we (Veeam) are waiting on a fix from Nutanix. They *may* be sending us a different and more appropriate HTTP status code in the fixed version so we need to be able to handle the legacy or updated scenario and since we will likely ship our v6.0 plugin prior to the Nutanix fix are preparing this release to handle either case. We did of course scope a workaround effort but it was impractical.

[JaySt]

Thanks for clarifying.

[ybarrap2003]

@JaySt here is the official Nutanix KB - https://portal.nutanix.com/page/documen ... 0003djB0AQ (login required)

[JaySt]

For those who dont have access to the link above:
===
This issue occurs due to newline characters (\n) being included in the exception message which causes a "CR/LF error" in the HTTP header. Since HTTP headers do not support newline characters, the request fails. Nutanix is currently working on a fix to sanitize such messages before setting them in the headers.


Customers using Veeam backups for Nutanix AHV are advised not to upgrade to AOS version 6.8 or later. This KB article will be updated once Nutanix fixes this issue and Veeam backups for Nutanix AHV are supported on the AOS 6.8.x version. This issue is being tracked via ENG-663089.

===
also:
i think i saw Nutanix mentioning Target fix version: 6.8.1.5

[ronnmartin61]

@JaySt, @AbsoluteBS, @flow90, @JulienBubrecht, @HYF_JE I wanted to let everyone know that I did receive word from my Nutanix counterpart late last week that the updated API should ship shortly in the 6.8.1.5 AOS release. Once it's available our QA team will need a week or two to put it through its paces and make a final determination on supportability. I'll continue to update this thread as things progress

[ybarrap2003]

Good update @ronnmartin61

[flow90]

Hello guys,

Just for information, AOS 6.8.1.5 is out today !

[ronnmartin61]

@flow90 just saw that too! Thanks for the heads-up. Our QA team will be testing as they're able and we'll notify here of the results as well as update system requirements in helpcenter

[ronnmartin61]

We'd also be interested in any testing results anyone here is willing to share. Thanks in advance!

[kurtvbmsve76]

Just tested with Veeam 12.2.0.334 ans AOS 6.8.1.5 and same issue when trying to restore a VM to Nutanix Cluster.