Backup jobs slow to initialise

[SteelContainer]

Hi There,
Certain backup jobs will take up to 20 minutes per VM just to start backing up. Once backup begins speed is ok.
Errors in the task logs for a vm show errors relating to a node in a different hyper-v cluster. eg: VM1 resides on the CLUST1 cluster but has multiple errors in its task log for the CLUST2 cluster.
Errors in the task log spend about 20 minutes trying to query a cluster that the VM doesn't reside on. It starts with a [HvSnapshotTracker] info/error. It says it it closing snapshots for that job but none of the VMs in this job reside on that cluster anymore.
Have raised a case but so far it hasn't been helpful - ID# 02751988
Is there someway to stop the job from trying to close old snapshots that dont exist? or maybe clear the HvSnapshotTracker? This other cluster does exist but it only has a few nodes and is slowly being decommissioned. No VMs in this job are on the second cluster.

Code: Select all

[11.04.2018 12:01:12] <12> Info     [HvSnapshotTracker] closing snapshots on Job: f319030b-e2b9-4486-a191-125b06552daa
[11.04.2018 12:01:12] <12> Info     [HvSnapshotTracker] Closing snapshot id: '3785d845-e0df-4f88-827b-3bbb6c578b39' on Host: 'CLUST2NODE1'
[11.04.2018 12:01:14] <12> Error    [HvSnapshotTracker] Failed to close snapshot id: '3785d845-e0df-4f88-827b-3bbb6c578b39' on Host: 'CLUST2NODE1'
[11.04.2018 12:01:14] <12> Error    Failed to resolve host name 'CLUST2NODE1' (System.Exception)
[11.04.2018 12:01:14] <12> Error       at Veeam.Backup.Common.CIpAddress.ResolveConnectionPoints(String serverName)
[11.04.2018 12:01:14] <12> Error       at Veeam.Backup.Core.Common.CRemoteMachineAddress.TryGetAddressesFromName(String someNameOrIp, IPAddress[]& addresses)
[11.04.2018 12:01:14] <12> Error       at Veeam.Backup.Core.Common.CRemoteMachineAddress.GetFor(CDBHost host)
[11.04.2018 12:01:14] <12> Error       at Veeam.Backup.Core.Common.CHost.GetManagmentAddresses(CDBHost server)
[11.04.2018 12:01:14] <12> Error       at Veeam.Backup.Core.SHvUtils.CreateHvIntegrator(CHost host)
[11.04.2018 12:01:14] <12> Error       at Veeam.Backup.Core.CHvSnapshotTrackerUtils.CloseSnapshot(CHvSnapshotDbInfo snapshotInfo)
[11.04.2018 12:01:14] <12> Error    No such host is known (System.Net.Sockets.SocketException)
[11.04.2018 12:01:14] <12> Error       at System.Net.Dns.GetAddrInfo(String name)
[11.04.2018 12:01:14] <12> Error       at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
[11.04.2018 12:01:14] <12> Error       at System.Net.Dns.GetHostEntry(String hostNameOrAddress)
[11.04.2018 12:01:14] <12> Error       at Veeam.Backup.Common.CIpAddress.GetIpAddresses(String hostName)
[11.04.2018 12:01:14] <12> Error       at Veeam.Backup.Common.CIpAddress.ResolveConnectionPoints(String serverName)

These errors repeat for 20 minutes and I believe this is what is slowing down the backups.

Thanks,
Cullan

[SteelContainer]

Also the following SQL query only returns results for CLUST1, so I'm not sure why it would think there are snapshots on the other cluster.

SELECT TOP 1000 [job_id]
,[session_id]
,[snapshot_id]
,[host_name]
,[vm_id]
,[collection_path]
FROM [VeeamBackup01].[dbo].[HvRecoveryCheckpoints]
where [job_id] = 'f319030b-e2b9-4486-a191-125b06552daa';

[SteelContainer]

Ahh looks like they are are in the HVVolumeSnapshots table. I'll ask the support tech if there is a safe way to clear these.

SELECT TOP 1000 [session_id]
,[snapshot_id]
,[host_name]
,[job_id]
FROM [VeeamBackup01].[dbo].[HvVolumeSnapshots]
where [job_id] = 'f319030b-e2b9-4486-a191-125b06552daa';

[SteelContainer]

FYI to those who have the same issue "[HvSnapshotTracker] Failed to close snapshot id" in the task logs slowing down backups.

Confirmed that this is an issue with the data in the [dbo].[HvVolumeSnapshots] table and a few other Veeam DB tables. Support gave us the queries to clear these tables and now our backup duration times have increased by 10x!, some even faster. If you have the same issue just log a case and reference my case number to confirm if you have the same issue.
I have also recommended to the support tech to notify the devs in case this is a more widespread problem as it could be slowing down other backups without people noticing.