Random timeout to start agent

[jmarshall]

We have a long standing issue with our Veeam backups that we have been troubleshooting on and off for over a year and still not go to the bottom of it. I've opened Veeam tickets in the past, but we still haven't come to a resolution, so I'm hoping that this forum might be able to help. (As per forum rules, the latest case number is: Case #04420162)

The issue we are seeing is VM backups are regularly failing with the error: Timeout to start agent. When the backup re-runs, it nearly always completes successfully.

The key point is that the VMs which fail are nearly always different, so it's almost certainly not a guest OS issue.

The current setup:

- Veeam B&R 10 running on a physical PowerEdge R530 server.
- VMs are running on a 2012 R2 Hyper-V 3 Node Failover Cluster, with storage spaces used for the CSV
- The backup job is configured with an on-host backup proxy (have also tested with Off-host backup)
- Hype-V guest quiescence is enabled and the option to take grash consistent backup selected
- CBT enabled
- Volume snapshots disabled (have also tried running with enabled)
- Backup storage repository is set to limit concurrent tasks to 2

- Reviewed Veeam logs for VM in question
- At [01.10.2020 19:11:44] there was an error logged "Failed to copy change tracking files from host 'HyperV_Host3' to proxy 'HyperV_Host1'."
- Reviewed System Event logs of HyperV_Host1 and during the time of the backup there is an event logged ID 1:

Code: Select all

Log Name:      System
Source:        VDS Basic Provider
Date:          01/10/2020 19:01:47
Event ID:      1
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      HyperV_Host1
Description:
Unexpected failure. Error code: 48F@01000003

- Researched this error and found article which advises this can be safely ignored:
https://support.microsoft.com/en-us/hel ... perating-s

- Reviewed Application event logs of HyperV_Host3 and found the following VSS error

Code: Select all

Log Name:      Application
Source:        VSS
Date:          01/10/2020 19:01:34
Event ID:      12289
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      HyperV_Host3
Description:
Volume Shadow Copy Service error: Unexpected error while calling GetStorageDependencyInformation.  hr = 0x8007048f, The device is not connected.
. 

Operation:
   Add a Volume to a Shadow Copy Set

Context:
   Execution Context: Coordinator

Error-specific details:
   Error: (0x8007048f) The device is not connected.

- After this error there is:

Log Name:      Application
Source:        VSS
Date:          01/10/2020 19:02:39
Event ID:      12289
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      HyperV_Host3
Description:
Volume Shadow Copy Service error: Unexpected error while calling GetStorageDependencyInformation.  hr = 0x8007048f, The device is not connected.
. 

Operation:
   Automatically choosing a diff-area volume
   Processing EndPrepareSnapshots

Context:
   Execution Context: System Provider

Error-specific details:
   Error: (0x8007048f) The device is not connected.

- All the hosts have had firmware updates to the latest version and 2012R2 is regularly patched so are also up to date
- Any help would be greatly appreciated

[Vitaliy S.]

Hi Jonathan,

You might have already gone through this with our support team, but given there is a VSS error, have you tried creating VSS snapshots manually? Is there any pattern in time when these backups fail? For example, some external activity starts at the same time when you jobs start and this overlap causes this issue.

Thanks!

[jmarshall]

Hi Vitaliy,

VSS snapshots create fine manually. Also the backup works fine on the re-run. There, is no pattern that I can find so far. It's not even every day for example. The backup completed successfully the two days prior. I just can't seem to find any correlation.

[Vitaliy S.]

Jonathan, apart from updating all the firmware (storage) I have no other ideas (reviewed your case for suggested options). Everything points to Microsoft unfortunately.

[lbardoul]

Hello,

Just in case, did you find something related to this random "Error: Timeout to start agent" ?
We are having the same issue recently.

Regards,
Ludovic

[PetrM]

Hi Ludovic,

Please don't forget to share a support case ID, as requested when you post about an error or technical issue. The error above is quite generic and can occur due to many different reasons.
You may ask our support team to review the above mentioned request 04420162 and to check either the solution provided is applicable in your case or not.

Thanks!