unable to add replica vms to cloud failover plan

[rhys.hammond]

Hey Everyone,

Just working on a project that involves a brand new Veeam Backup & Replication v9.5U3 configured with a Service Provider consuming CC-R.
Replication is working pretty well, we had a couple small wrinkles that needed to be ironed out but everything feels stable at the moment.

One issue that I cannot solve is that when we create a Cloud Failover Plan, we are unable to add any objects to the failover plan. See attached image.
I performed some basic troubleshooting to no avail so opened a support Case # 02446414

If I right-click a cloud replica VM I can 'Add to failover plan' and select 'New cloud failover plan' but if I try and add another cloud replica no luck. When I right click and expand the 'add to failover plan' the only option is to create a new failover plan.

Our next steps are to contact the Service Provider to troubleshoot on their end and collect logs from them but I wanted to post on the forums first to see if anyone else had seen this issue?
To me it feels like it could be a firewall issue but we know the replication to the SP is working (working after we enabled TCP/UDP for their required port).
The reason being is when we tested from a Veeam BnR Console that wasn't installed on the VBR server, we can't see any Cloud replicas at all.

Looking at the firewall logs we can't see anything being blocked though.

Cheers Rhys Hammond

[dellock6]

I've never seen this issue before honestly. Just so we are sure everything is up to date, that both provider and client are at 9.5u3?

[rhys.hammond]

Hey Luca,

Yep, both tenant side and SP are both at 9.5u3

[rhys.hammond]

Veeam support has been escalated to the cloud team, will post an update when I know more.

[n.zahid]

I have got the same issue and Veeam support is investigating. Is there any update on this yet ?

[dellock6]

Not that I'm aware, please keep both updated about what support will find out.

Thanks,
Luca

[rhys.hammond]

status of my open support ticket is currently: Research/Testing

[rhys.hammond]

just heard back from support, they found some orphaned entries on the service provider database that could be preventing us from creating the failover plan. (two job entry from the service provider database (for the tenant) that has no OIB's.)
We need to clean up this entry on the service provider database and tenant database and then try to create the failover plan again.

Will report back once cleaned and tested.

[rhys.hammond]

ok Veeam support were able to successfully resolve this issue. The root cause were few a backup ids with zero OIB. After we removed the orphaned entries from the provider database we were able to create the failover plan successfully.
In my case the only entries that were cleaned up were on the SP side, not the tenant.

[dellock6]

Thanks Rhys for updating the thread, and glad it has been fixed.

Luca

[rhys.hammond]

This issue has reoccured after the SP and tenant upgraded to 9.5u3a
Have opened a new support ticket with Veeam to resolve

[veremin]

Kindly, let us know the support ticket ID, so we can follow the investigation. Thanks.

[rhys.hammond]

The case# was 03141448,

Same resolution steps as last time but support shared how to troubleshoot these myself for next time.
For those that are interested, from the SP side you run the following query to find a stale job entry and then delete it.

Code: Select all

select T1.id as backup_id, T1.job_name, t2.id as point_id, T3.id as OIB_id, t3.point_id from dbo.[Backup.Model.Backups] as t1 full outer join dbo.[Backup.Model.Points] as t2 on t1.id = t2.backup_id left outer join dbo.[Backup.Model.OIBs] as t3 on t2.id = t3. point_id where t1.id in ( select backup_id from [Backup.Model.ViCloudTenantBackups] where [quota_id] in ( select id from [Backup.Model.ViHardwareQuotas] where tenantId = (select id from dbo.tenants where name = [i]'%TenantName%[/i]'))) and t3.id is null

Replace %TenantName% with the tenant name in CC

The output should be something like

Code: Select all

064bfb1c-7a1e-42c9-bad4-40629f6c4048f2ca3a17-18b5-4690-82f8-530834453b3a%VMNAME%

%VMNAME% will depend on the VM object that is stale.

Query to delete stale entry:

Code: Select all

EXEC [dbo].[Backup.Model.DeleteBackupChildEntities] '064BFB1C-7A1E-42C9-BAD4-40629F6C4048'
EXEC [dbo].[Backup.Model.DeleteBackup] '064BFB1C-7A1E-42C9-BAD4-40629F6C4048'

Output:
1 Row affected