Veeam for AHV - Inability to restore and backup simultaneously in a chain (different RPs)

[HYF_JE]

Case 07248439. Opening up this thread on the advice of the support rep.

VBR 12.1.0.2131
AHV Plug-In 12.5.1.8

This will be a weird one, hopefully not one most admins will face. Maybe I have something fundamentally wrong here and this is working as designed.

I have a relatively new backup job/chain - a single VMware ESXi VM. I had successfully taken one full backup of that VM. I manually started the job again to get a fresh incremental RP (mostly to get an idea on how much data the VM churns).

While that incremental was running (which I expected take a while) I then attempted to start a restore of the VM to a Nutanix AHV cluster, using the restore point based on the full backup. What I found was that this failed. When trying to do an entire VM restore to AHV, I got this error immediately, the job never actually kicked off:

Code: Select all

Failed to start restoring virtual machine. Failed to execute restoreVM - Object reference not set to an instance of an object.

Failed to execute restoreVM - Object reference not set to an instance of an object.

I also tried doing an instant VM restore, and that error text complained about the .vib being locked. That didn't make sense to me, as I didn't ask for the restore of the VM from the in-progress incremental backup. I selected the full backup's restore point.

Code: Select all

Error Failed to start instant recovery session: Failed to publish disks. Failed to publish REDACTED Error: Failed to lock backup files
Item [REDACTED.vib] is locked by running session REDACTED [Backup]

Without doing a lot of research on my own, I'm guessing there's a limitation (unique to restores to AHV??) where you cannot restore from a chain when that chain is being used as part of a backup job to add an incremental.

This isn't a particular issue in our environment as our RPOs are pretty lenient but I could see this being problematic for customers who have aggressive/frequent RPOs. Here I'm thinking of a situation where an administrator wants to test a VM restore but doesn't want to disable the backup job in order to complete that restore.

Also to note it - while drafting this post, I took the same restore point in the chain (while the fresh incremental was running) and started a restore to a different VMware cluster and that seems to be working/in progress without error. I think that adds to the possibility of this being a limitation or bug.

[ronnmartin61]

Definitely a corner case but we will take a look...

[HYF_JE]

Quickly updating this thread:

I'm facing other issues with restores of VMware VMs to Nutanix AHV that I won't go into all the details on. The original issue I reported (inability to restore from a chain while the chain is being used for an active backup) may be a red herring on my end.

That said, if anyone from R&D would be able to expand on the factors that would lead to the first codeblock/error regarding "Object reference not set to an instance of an object" that would be much appreciated as it's stumping myself and support.

[HYF_JE]

Veeam support rep asked me to upgrade the VBR to 12.1.1.56 to try to solve the issue I'm having with restoring a VMware VM to Nutanix AHV. It was unsuccessful in doing so.

[HYF_JE]

Veeam support had me upgrade to what I'll describe as a "private build" of the AHV plugin - 12.5.1.17. Apparently the error message to do with "Object reference not set to an instance of an object" is a known issue and this newer build fixes it.

I wasn't told of any known downsides to installing this version, so I'm guessing Veeam treats this build as a "hotfix" of sorts where they only want customers with the issue installing it as it may not be as vetted/tested as the latest official build (5.1.0.7).

What's important is that this solved my issue. Again, the original title of this topic is a bit inaccurate - this didn't have to do with simultaneous use/access of restore points/chains as I thought it did. Instead it was entirely to do with a bug in the Veeam for AHV plugin.

Some more details on that specifically, my understanding is there's some (JSON?) structured data that the software is interpreting and one of the key/value pairs has the value of null (not the string "null", but null as a value). That apparently causes the issue, as explained by support below:

That "objectTag":null is what is causing the failure, as the Plugin is unable to handle it being null. This tag is created by VMware / HyperV backups that are created before AHV 5. The fix? To ignore the tag. The behavior of the "objectTag" being null is being addressed and will be properly handled in our next minor version of the Backup and Replication software.