After Veeam 12.2 upgrade backups are failing

[arngrimurm]

Case #07428369 — Backup of file servers running on Nutanix fails after upgrade to 12.2
Backups are failing randomly, but it seems mostly be VM's with large disks.

Failed to perform backup: [RemoteDiskChangesAccessorProxy] GetNextArea exception: External GetNextAreaCallback failed, return code = 1
Failed to upload disk '19b91d1f-3a04-4f30-ac4e-97940c274343'
Agent failed to process method {DataTransfer.SyncDisk}

The silence from Veeam support is frighting, no fix or workaround is offered.
Anyone having the similar problems after upgrading to Veeam 12.2 ?

[Mildur]

Hi Arngrimur

I can see from the case notes that the investigation is still ongoing.
Have you tried to contact support management to ask for a case update?
https://www.veeam.com/kb2320

Best,
Fabian

[arngrimurm]

Yes several times, I also filled out the form "Talk to a manager" on the Veeam support site, but it's just "yes we are investigating it".
That is all good and fine, but it is not helping me at all at the moment were all my Nutanix backups are failing randomly.
Tried to have the case updated to severity one, but it was denied.
I have opened a Nutanix support case as well and they of course just point to Veeam and say they need to check for any timeout values.
This level of support is really disappointing to say the least.

[anbj01]

We have the same issue , Veeam support has find this in our logs:
"From the error message we can see that the AHV Proxy sends the request to the Cluster but gets the error in return"

and asked us to open a support ticket by Nutanix how has collected logs but i have got any news after that.

[arngrimurm]

Something like this?

Code: Select all

2024-09-18 23:25:52.7162 00072 [59229] ERROR | [NutanixClientV3][cluster1]:  <== Request "Post" "https://cluster1:9440/api/nutanix/v3/data/changed_regions", body: "{"snapshot_file_path":"/VM_Container/.snapshot/20/5804727506788731936-1640089859473891-1126520/.acropolis/vmdisk/0ad33b9a-34d0-4108-820f-51c54ba51210","reference_snapshot_file_path":null,"start_offset":1891159261184,"end_offset":1892233003008}"
2024-09-18 23:25:52.7496 00072 [59229] ERROR | [NutanixClientV3][cluster1]:  ==> Response "Post" "https://cluster1:9440/api/nutanix/v3/data/changed_regions", "code: 500", duration: "4 sec 770 msec", body: "Request to Aplos failed with response code 503"
2024-09-18 23:25:52.8338 00072 [59229] ERROR | [AhvBackupDataSource]: System.Net.Http.HttpRequestException: Request failed with status code InternalServerError

My Nutanix cluster is running AOS 6.5.5.7 LTS and CPU load is below 40% average.

[Gostev]

I doubt this has anything to deal with the cluster load because there are dedicated HTTP error codes for overload, while error code 500 is used for generic/unspecified error. As a quick fix I suppose we can try and add retries on the Veeam side, and see if that helps.

[anbj01]

This is info i got i my case. (We are running AOS 6.7.1.8.)

AHV Proxy sends the request to get the changed regions and received the error in response

Code: Select all

2024-09-11 06:38:41.1621 00100 [310189] ERROR | [NutanixClientV3][xxx.xxx.local]:  <== Request "Post" "https://xxx.xxx.local:9440/api/nutanix/v3/data/changed_regions", body: "{"snapshot_file_path":"/Default/.snapshot/45/818718483217355342-1721913420961586-247745/.acropolis/vmdisk/802ab180-1299-47b8-8576-0897da2799ad","reference_snapshot_file_path":null,"start_offset":981400027136,"end_offset":982473768960}"
2024-09-11 06:38:41.1985 00100 [310189] ERROR | [NutanixClientV3][xxx.xxx.local]:  ==> Response "Post" "https://xxx.xxx.local:9440/api/nutanix/v3/data/changed_regions", "code: 401", duration: "5 sec 728 msec", body: "{"state":"ERROR","code":401,"message_list":[{"reason":"AUTHENTICATION_REQUIRED","message":"Authentication required.","details":"Authentication tokens in API request are invalid"}],"api_version":"3.1"}"

as a consequence, since we have not received no information about the changed regions, we are unable to get the next block data

Code: Select all

2024-09-11 07:14:38.7422 00004 [310189] ERROR | [VbrDiskBackup]: Failed to get next data area: System.Net.Http.HttpRequestException: Request failed with status code Unauthorized

And we cannot to proceed further

Code: Select all

2024-09-11 07:14:39.2518 00004 [310189] ERROR | [VbrVmBackupSession]: Unable to retrieve next block transmission command. Number of already processed blocks: [931524].
2024-09-11 07:14:39.2518 00004 [310189] ERROR | [VbrVmBackupSession]: Failed to download disk '802ab180-1299-47b8-8576-0897da2799ad'.

[ronnmartin61]

@arngrimurm as @Gostev mentioned at this point we believe adding retries will be the proper path toward issue resolution and we're diligently working toward this end. Will update this thread as soon as we have some sort of ETA for the patch

[anbj01]

@ronnmartin61 good to know , can you add our case to your list so we will get a fix when its done. (#07415487 )

[ronnmartin61]

@anbj01 the auth error you're seeing is a bit different but seems to be the same API so perhaps related. Will track along with yours also...

[anbj01]

How long can we aspect for a workaround for this ? Days / weeks ?

[ronnmartin61]

@anbj01 definitely not weeks. We're QA testing the patch now and barring any unexpected interruptions are hoping to have it out in a couple more days

[anbj01]

That sounds good

[anbj01]

@ronnmartin61 Any updates on how QA testing is going ,is release comming up

[ronnmartin61]

@anbj01 good news! v6.1 is now available for download from your Veeam "my account" or public pages (https://www.veeam.com/availability-nuta ... nload.html)

[arngrimurm]

I installed this update yesterday (26-SEP-20204), backups are still failing on the same VM's as before, this VM size is 2.4TB. The error came after processing 166GB.

27.9.2024 02:42:27 :: SERVERNAME : Failed to perform backup: [RemoteDiskChangesAccessorProxy] GetNextArea exception: External GetNextAreaCallback failed, return code = 1
Failed to upload disk '19b91d1f-3a04-4f30-ac4e-97940c274343'
Agent failed to process method {DataTransfer.SyncDisk}.

[arngrimurm]

@anbj01 Have you installed v6.1 ?
@ronnmartin61 I have already notified Veeam support about that this update is not solving my problem.

[geetansh]

We are facing the same issue with Veeam Nutanix Backup jobs and the patch has not resolved it

[ronnmartin61]

@arngrimurm thank you for the continued feedback though it's definitely not what I had hoped to hear. Please continue to work with support and we'll get the issue tracked down. Thanks again for your patience

[Gostev]

I doubt this has anything to deal with the cluster load because there are dedicated HTTP error codes for overload, while error code 500 is used for generic/unspecified error. As a quick fix I suppose we can try and add retries on the Veeam side, and see if that helps.

Just wanted to remind what was the plan for a quick fix. Retries obviously won't help in situations when Nutanix cluster keeps returning errors persistently. So the next step would be to understand why API endpoint persistently errors out. This we won't be able to troubleshoot on our end, without help from Nutanix... I assume their would need to investigate impacted AHV clusters directly.

[arngrimurm]

@ronnmartin61 My Veeam backup server was using an Active Directory account to authenticate to the Nutanix cluster.
After I changed that to a local Nutanix account I was able to backup my two problematic large file servers.
https://helpcenter.veeam.com/docs/vbahv ... html?ver=6

I'm not going to open the champagne yet, but let's say I'm chilling it

[anbj01]

Thank you @arngrimurm !!! This solved my issues. We upgraded to latest version but same error. Our problem was also related to AD user. AD servers are located in an other datacenter ,and i our case it seems like VPN tunnel between was causing our errors. We move to an other network solution and now jobs seems to work normaly. But good to now that maybe its best to change and use a local Nutanix user insteed

Thanks to you at Veeam to always try to fix problems really fast.

[ronnmartin61]

@anbj01 yes we do indeed always strive to get these issues resolved as quickly as possible. Thanks to you and @arngrimurm for the information on resolution

[geetansh]

@ronnmartin61 Is there a permanent solution in progress to fix the AD authentication issue?

[ronnmartin61]

@geetansh we're looking into it now to assess overall remediation effort and potential fix delivery mechanics...

[geetansh]

Please let me know if there is an update on this issue. Is this fixed in 12.3?

[ronnmartin61]

@geetansh yes the authentication issue should be resolved with VBR 12.3 / AHV plugin 7.0