Veeam and Zerto do not play nice

[MikeK]

We've recently made the leap to Zerto and have decided to move away from VMware SRM and storage based replication. Our plan is to leverage Zerto in conjunction with Veeam to meet our data protection requirements. Over the past month we initiated our first Zerto test failover and encountered an issue that causes Veeam backups to fail. I would like to share and to inquiry if anyone else has seen the same.

Failover Process
Leveraging Zerto we initiate a "Move" operation. This allows us to to failover VMs to our DR site, power up at DR, validate apps, commit (which initiates a reverse sync), and cut the lines between Prod and DR. Once all testing is complete, the lines are re-connected and we initiate a Move operation to failback to Prod.

Problem
Once a commit is performed in Zerto, source side the VMs are removed from inventory in vSphere. When we perform a Move to failback to Prod and VMs are added to inventory, new UUIDs are created :-\. Veeam ,of course does, not like this and subsequently backups will fail. We can manually add the VMs to their respective jobs, however, new backup chains are created. New chains equals more data and in this particular instance we filled one of our repositories.

Options
- We can perform a Zerto Test failover aka Bubble test. This is not a good option for us as it does not achieve the level of validation we would like to from a DR perspective
- We can initiate a Move operation, power up at DR, validate apps and then DO NOT COMMIT. This would prevent VMs from being removed from inventory but presents other challenges with our DBs that replicate outside of Zerto. We're concerned about application consistency.

Questions
- Has anyone else encountered this scenario?
- What exactly is Veeam targeting in vSphere as the VM identifier? Is it the UUID, the InstanceUUID, moref or other? Is there a way to manually modify a Veeam job with an updated Instance UUID?

[DaveWatkins]

Someone will probably answer this specifically, but I'm pretty sure Veeam uses moref as it's ID for a VM, which will explain what you're seeing since new VM's will have new moref's. Veeam has a tool to update the Veeam DB when doing vcenter migrations that availble in a KB article I think, but I doubt it'll be useful to you since it's designed more as a one time use thing

[jmmarton]

As Dave pointed out, the vSphere MoRef is used. If that changes then the VM is a brand new VM unless the MoRef is updated in the Veeam database. This can be done using the vCenter Migration Utility from Veeam (https://www.veeam.com/kb2136).

Out of curiosity, are you doing realtime replication with Zerto? If not, then perhaps a combination of Veeam replication + Veeam Availability Orchestrator may be worth investigating as an alternative.

Joe

[MikeK]

Thanks for the responses. Makes sense the MoRef is used. It may be more of a VMware/Zerto issue but essentially if we perform a Move operation and commit in Zerto, the source side VMs will be removed from inventory and new MoRefs will be created when added :-\

Zerto is asynchronous replication providing RPOs that are only seconds old. It is widely used for DRAAS and when service providers tout their DRAAS/Gartner top right quadrant capabilities they are all undoubtedly leveraging Zerto. For us it complementary to Veeam. Veeam replication would not meet our requirements from an RPO/RTO perspective.

The only solutions we have really been able to come up with is fundamentally changing our DR failover process or possibly some unsupported workaround: http://robvit.com/veeam/backup_replicat ... not-found/
Again, Veeam and Zerto are really complementary products to us but this is a real challenge that we're struggling with and that is forcing us to evaluate our backup architecture.

[jon.france]

What happens if you use a folder (or even a datastore) to add your VMs to your Veeam backup job instead of adding the VMs individually by name?

When Zerto, performs its second Move operation to create the VMs at the primary site again, will it place them back in the original folder? If so, I would expect the Veeam backup job to pickup those 'new' VMs for you without requiring that you manually add them.

[FrenchBlue]

Hello,

Sorry, a bit of digging there, but very same issue 4 years later. Has anyone found a solution to this? Even in V10, Zerto can preserve UUID, but not MoRef ID.

[FrenchBlue]

Update: there's no way to preserve the MoRef ID on Zerto's side when a move or a failover are committed.

Is there a way in Veeam to force it to re-attach a "new" VM to an existing backup chain?

Thanks.

[Mildur]

Hi Frenchblue

A new VM has a new MoRef ID and cannot be manually attached to an existing backup chain of a foreign VM.

In one case Veeam preserves the backup chain. If you use Veeam Quick Migration to migrate a VM.
If the target VM is successful in the heartbeat test after the migration, we will replace the MoRefID in the Veeam database with the new target VM. Backup Chain will continue with incremental backups.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

In a quick migration scenario, we can read all the data (source and target metadata) required to verify that we map the backup chain to the correct VM MoRef ID.
Manual re-mapping through a backup administrator require additional checks we don't have today. Therefore we don't have such feature implemented. We will research this topic and what can be done at one point in the future. But no ETA I can provide today.

Best,
Fabian

[FrenchBlue]

Hello,

Thanks for the answer. But to do the failback we use Zerto's automatic reverse replication. So at the moment there's simply no workaround for this issue, we'll have to do with it.

[Mildur]

Yes.
Unfortunately there is no workaround available.

Best,
Fabian

[harrycarter]

Hello

Is there any update from Veeam as to whether this is on the roadmap? 2025 and still the same issues when failing back.

[Mildur]

No update regarding the Zerto use case.

But we have now an officially supported PowerShell cmdlets to update the MoRefID for vCenter migrations scenarios.
However, this command is designed for one-time migration of an entire vCenter where the VMs are not changed (just managed by a new vCenter) and the old vCenter will not be used anymore by the backup server.

Best,
Fabian

[tyler.jurgens]

I was thinking those PowerShell cmdlets would work here - The DR vCenter that the Zerto VMs get created on will create a new VM during a failover and destroy the VM after the failover. That said, the DR vCenter isn't really the concern here, because its transient and the Zerto customers may or may not be using Veeam to backup their VMs while on the destination vCenter. If they are backing up the VMs during a failover, the PowerShell cmdlets, in theory, should work fine.

However its more the fail back scenario that gives me pause. Specifically around:

Old vCenter must not be used by Veeam Backup & Replication after migration.

Can the old vCenter never be used again for any reason other than a concern around possible duplicate MoRefIDs? I would like to know more.

Worth testing, but *if* the "old" vCenter can be reused (Aka, the production vCenter remains):
1) Before running the Zerto failover, get the MoRefIDs of the existing VMs
2) Perform the Zerto failover.
3) Fail back the Zerto VMs
4) Use the PowerShell cmdlets to map the MoRefIDs from the old VMs to the "newly" created VMs that Zerto will deploy during the fail back.

The PowerShell cmdlets always reference "Old" and "New" vCenters - I am not sure if they can be the *same* vCenter instance - which is what you would need in this case. It would essentially be the same as cloning a VM in vCenter and wanting to map the backup chains to the cloned VM. I haven't tested this myself, but if this is possible in the PowerShell cmdlets then it should work fine for Zerto failover/failback.

Or just use VeeamCDP.