Veeam and Zerto do not play nice

[MikeK]

We've recently made the leap to Zerto and have decided to move away from VMware SRM and storage based replication. Our plan is to leverage Zerto in conjunction with Veeam to meet our data protection requirements. Over the past month we initiated our first Zerto test failover and encountered an issue that causes Veeam backups to fail. I would like to share and to inquiry if anyone else has seen the same.

Failover Process
Leveraging Zerto we initiate a "Move" operation. This allows us to to failover VMs to our DR site, power up at DR, validate apps, commit (which initiates a reverse sync), and cut the lines between Prod and DR. Once all testing is complete, the lines are re-connected and we initiate a Move operation to failback to Prod.

Problem
Once a commit is performed in Zerto, source side the VMs are removed from inventory in vSphere. When we perform a Move to failback to Prod and VMs are added to inventory, new UUIDs are created :-\. Veeam ,of course does, not like this and subsequently backups will fail. We can manually add the VMs to their respective jobs, however, new backup chains are created. New chains equals more data and in this particular instance we filled one of our repositories.

Options
- We can perform a Zerto Test failover aka Bubble test. This is not a good option for us as it does not achieve the level of validation we would like to from a DR perspective
- We can initiate a Move operation, power up at DR, validate apps and then DO NOT COMMIT. This would prevent VMs from being removed from inventory but presents other challenges with our DBs that replicate outside of Zerto. We're concerned about application consistency.

Questions
- Has anyone else encountered this scenario?
- What exactly is Veeam targeting in vSphere as the VM identifier? Is it the UUID, the InstanceUUID, moref or other? Is there a way to manually modify a Veeam job with an updated Instance UUID?

[DaveWatkins]

Someone will probably answer this specifically, but I'm pretty sure Veeam uses moref as it's ID for a VM, which will explain what you're seeing since new VM's will have new moref's. Veeam has a tool to update the Veeam DB when doing vcenter migrations that availble in a KB article I think, but I doubt it'll be useful to you since it's designed more as a one time use thing

[jmmarton]

As Dave pointed out, the vSphere MoRef is used. If that changes then the VM is a brand new VM unless the MoRef is updated in the Veeam database. This can be done using the vCenter Migration Utility from Veeam (https://www.veeam.com/kb2136).

Out of curiosity, are you doing realtime replication with Zerto? If not, then perhaps a combination of Veeam replication + Veeam Availability Orchestrator may be worth investigating as an alternative.

Joe

[MikeK]

Thanks for the responses. Makes sense the MoRef is used. It may be more of a VMware/Zerto issue but essentially if we perform a Move operation and commit in Zerto, the source side VMs will be removed from inventory and new MoRefs will be created when added :-\

Zerto is asynchronous replication providing RPOs that are only seconds old. It is widely used for DRAAS and when service providers tout their DRAAS/Gartner top right quadrant capabilities they are all undoubtedly leveraging Zerto. For us it complementary to Veeam. Veeam replication would not meet our requirements from an RPO/RTO perspective.

The only solutions we have really been able to come up with is fundamentally changing our DR failover process or possibly some unsupported workaround: http://robvit.com/veeam/backup_replicat ... not-found/
Again, Veeam and Zerto are really complementary products to us but this is a real challenge that we're struggling with and that is forcing us to evaluate our backup architecture.

[jon.france]

What happens if you use a folder (or even a datastore) to add your VMs to your Veeam backup job instead of adding the VMs individually by name?

When Zerto, performs its second Move operation to create the VMs at the primary site again, will it place them back in the original folder? If so, I would expect the Veeam backup job to pickup those 'new' VMs for you without requiring that you manually add them.

[FrenchBlue]

Hello,

Sorry, a bit of digging there, but very same issue 4 years later. Has anyone found a solution to this? Even in V10, Zerto can preserve UUID, but not MoRef ID.

[FrenchBlue]

Update: there's no way to preserve the MoRef ID on Zerto's side when a move or a failover are committed.

Is there a way in Veeam to force it to re-attach a "new" VM to an existing backup chain?

Thanks.

[Mildur]

Hi Frenchblue

A new VM has a new MoRef ID and cannot be manually attached to an existing backup chain of a foreign VM.

In one case Veeam preserves the backup chain. If you use Veeam Quick Migration to migrate a VM.
If the target VM is successful in the heartbeat test after the migration, we will replace the MoRefID in the Veeam database with the new target VM. Backup Chain will continue with incremental backups.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

In a quick migration scenario, we can read all the data (source and target metadata) required to verify that we map the backup chain to the correct VM MoRef ID.
Manual re-mapping through a backup administrator require additional checks we don't have today. Therefore we don't have such feature implemented. We will research this topic and what can be done at one point in the future. But no ETA I can provide today.

Best,
Fabian

[FrenchBlue]

Hello,

Thanks for the answer. But to do the failback we use Zerto's automatic reverse replication. So at the moment there's simply no workaround for this issue, we'll have to do with it.

[Mildur]

Yes.
Unfortunately there is no workaround available.

Best,
Fabian