CBT and Hot Add failing after Instant VM Recovery

[smile_dav]

After some testing, I can say that the Hot Add issue is caused by a File Level Restore or a Instant Recovery run. However, when I reboot the server after a FLR or IR run, the backup jobs are fine...

[dellock6]

I had the same problem with a customer today, yesterday we did both a instant recovery and a file level restore, and now I'm receiving warning log complaining they had to failover to network mode, while they usually run in hotadd mode.

My customer is using Veeam Backup v6 with patch #3. Anyone else with this behaviour? Maybe we can submit a bug.

[rawtaz]

I had the same problem with a customer today, yesterday we did both a instant recovery and a file level restore, and now I'm receiving warning log complaining they had to failover to network mode, while they usually run in hotadd mode.

My customer is using Veeam Backup v6 with patch #3. Anyone else with this behaviour? Maybe we can submit a bug.

This is what triggered it for me too. I did my first Insant Recovery just to try it out, and right after that the VM started not being able to do HotAdd. A reboot of the Veeam controller solved it.

So to clarify; You say that they were using patch #3, and that is the latest one, right? The one that was released just recently and was meant to address this problem? Or is the latest patch not #3?

[dellock6]

Yes, I'm using patch #3, the last one published some days ago. I installed it right before using Instant Recovery.

[Gostev]

Maybe we can submit a bug.

Yes, and please post the support case ID in this topic.

[dellock6]

Hi Anton,
case opened, assigned ID#5174618.
I'm available to do all tests we need to discover the cause of the issue.

[peter.dyer]

I can confirm that a reboot of the Veeam server resolved these issues for me.

I cannot confirm the issues were caused by an Instant Restore. In my case I suspected a SureBackup job (basically the same as an Instant Restore). However, since the reboot I've been unable to reproduce the issues.

[dellock6]

For now, I did not rebooted my customer's server so we can still see the issue.
Backup are failing over to network mode and they did complete, so we can stay for some days in this situation. Hopefully it would help in the resolution of the problem; I sent all the logs to support, waiting for other tests to do.

[smile_dav]

Anyone got any news from Veeam Support?
I have been asked to check whether there are any mounted disks on the Veeam Backup Proxy and whether Change Block Tracking was enabled...
Just wanted to know if other people are at a more advanced stage at getting a resolution.

I know the obvious workaround is to change the transport mode for the proxy to network, but Virtual Appliance mode is much faster.

[dellock6]

I did some tests with Support and they all failed, I exported further logs and send to them; now I'm still waiting for a showback from them.
I agreed with this customer to let the jobs failover to network mode, so we did not rebooted the Veeam Server, because seems that rebooting the server sometimes makes the error non reappearing even redoing a Instant Recovery, so we will not be able to eventually analyze the bug, if it's a bug.

[vmwarektnx]

Same error, reboot fixed the problem

[dellock6]

Quick update: I was able to recreate the error by doing a reboot and then another File Level Restore, without an Instant Recovery. So the error is lying somewhere there...

[rawtaz]

At least it's a very good thing that a reboot "fixes" it, I mean as opposed to having to redo the backups or jobs or whatever

[Gostev]

Thank you all for your patience. The research has been completed today, we now fully understand what happens.

Indeed, this has nothing to deal with Instant VM Recovery, but only File Level Restore. The FLR process reserves a block of mount points from the system (100 of those), making hot add to use those above 100, and there is a bug in vStorage API that makes hot add go crazy in such case. The immediate hotfix will probably be to reduce the number of reserved mount point from 100 to 20 or something. As far as the current workaround, I suggest disabling local VMware backup proxy on your backup server, and using another VM as your hot add proxy server.

Thanks!

[rawtaz]

Gostev; Is it correctly understood that 1) Veeam will issue a hot fix/patch for this in the near future; 2) Until then, one may either do the workaround you suggest, OR simply not do File Level Restores and the problem shouldn't appear at all?