PSA: Don't try to Migrate To Production if the VM is powered off!

[pvz]

Hi!

I would just like to share my experience with Veeam Backup & Replication, and perhaps save someone a phone call to support by posting about this experience.

The scenario: I use Veeam's integration with Nimble Storage snapshots, and I would like to "restore" a VM off a snapshot to a different VM for lab purposes. By leveraging Storage-level snapshots I don't have any impact on the production VM, and I also don't have to transfer the data off a backup, so it's faster. However, for Nimble snapshots, it's not possible to do a restore, so instead my plan is to do an instant VM recovery immediately followed by a migrate to production, without powering the VM on. We are licensed for Storage vMotion in our environment.

However, when trying to do this (Migrate to Production on a powered off VM), Veeam fails with the following error message:

2020-09-29 10:06:43 :: Relocating VM Error: Error caused by file /vmfs/volumes/5f72e515-0a22e2a2-755a-08f1ea705954/xxxx-dc01.xxxx.xxx/xxxx-dc01.xxxx.xxx.vmdk

Quick troubleshooting in vSphere just showed me the same error message, so I picked up the phone with Veeam support (ticket number #04413215). After a little bit of headscratching, the engineer on the phone suggested that it doesn't make sense and is not possible to do a vMotion on a powered off VM, so we tried powering the VM on and then trying a migrate to production, and lo and behold, it worked.

So, my PSA is: Don't try to do a "migrate to production" operation if the IVR VM is powered off. It doesn't make much sense in most cases anyway, since the whole point of an IVR is to have an instantly powered on VM that can be migrated to production online, so I don't really fault Veeam for not testing it. The only reason I did it this way is that Veeam doesn't have a workflow for a "restore" off a Nimble snapshot, which caused me to do it this way.

I would like to say I'm actually very satisfied with the level of support. Veeam is one of the few companies where you can just pick up the phone and get help with your issues immediately without dealing with ticket creation and then waiting days for a callback, and I really appreciate how my assigned support engineer decided to take initiative and help us quickly come up with a reason as to why what I was trying to do didn't work. I can think of several other vendors I'm a customer with where an issue like this would take weeks to even identify.

But what Veeam could do to improve here to make it not even neccessary to pick up the phone would be to add a check to make sure that a VM you're trying to "migrate to production" is actually powered on. (Usually you never even want to do that on a powered off VM anyway, since you won't be able to power the VM on during a migration process, and the only reason I hit this scenario because of a particular corner case involving Nimble snapshots.) Or at the very least add a better error message indicating that you can't Storage vMotion a powered-off VM.

This experience is with VBR 10a (10.0.1.4854) and ESXi 6.7.

[HannesK]

Hello,
thanks for the positive feedback on support. I will forward it to support!

I guess PSA means something like request for improvement or feature request. And not the car manufacturer (PSA).

Thanks for the feature request though. We count your vote as +1.

Best regards,
Hannes

[Andreas Neufert]

This is a VMware limitation that was introduced in the latest VMware updates. The reason for this is that the CBT is not in a clean state and VMware prevents VMs (independant from Veeam) from Storage vMotion. We are investigating how we can workaround this, so that it automatically would work.

[Gostev]

PSA = Public Service Announcement

[Rumple]

The reason for this is that the CBT is not in a clean state and VMware prevents VMs (independant from Veeam) from Storage vMotion. - This is an older post, but from reading this, it sounds like when a VM is powered off, you can no longer migrate the VM to different storage?
That would basically eliminate being able to move a VM between Sites where you are changing storage and compute at same time...

[Andreas Neufert]

There are many ways to achive the Migration (for example our Quick Migration or Replica Failover). Storage vMotion is used prefered if available. vMotion is I think not affected.

Usually the non consistent CBT state is forced by server downtime, specific actions against an offline VM (like multiple disk resizings)...
It has nothing to do with Veeam and how we do things.

At normal Backup (with VMware VM Snapshot processing), the CBT is in a consistent state and you will not face this.
If you just do Crash Consisten Backups on the Storage (without VMware VM Snapshots created before), the CBT is as well in Crash consistent state. When you restore from this Storage Snapshot with or without (manually) Veeam, you have to at least boot the VM once before Storage vMotion or reset CBT manually.