Backup copy job – Immediate Copy (mirroring) – Cannot be started issue

[SteelContainer]

Case # 04910549

We had an issue where a subset of backup copy jobs were not working correctly for a while.
Three of the backup copy jobs (immediate mirror mode) were stuck in a state where they were failing but not sending any alerts, either email or to our linked VSPC console. Veeam was not even writing to the copy jobs log files, the only indication of a failure was the following error in the VBR console:

14/07/2021 5:48:21 PM :: Job BACKUPCOPYJOBNAME cannot be started. Timeout: 904.6279091 sec
14/07/2021 5:48:24 PM :: Job has failed unexpectedly

We managed to get the jobs working again by removing the backup chains from the Veeam config, rescanning, creating new jobs and remapping the backups. This also resynced the 28 day retention for the job from the source backup job, but I assumed it would also resync the missing GFS monthly fulls which it did not even when using the sync all option. I think the ability to sync all backups from the source backup job would be a great feature. I actually thought that was how it already worked (keeping an exact copy in a second location), but now since the job had been failing for months we are now missing months of full backups in the second location.

The other thing we really need is the alerting to work as we expect to be notified on failures either via mail or VSPC. Perhaps this is an edge case or just a failure that had not been catered for. Alerts for the other backup\backup copy jobs were still working.
The Veeam tech we dealt with said he would raise this with the back end team but I thought I should post here for added visibility.

Thanks

[HannesK]

Hello,
for the alarm in VSPC... I assume the reason is that VBR didn't show any error. VSPC alarms if there is an issue in VBR. But as far as I understood the case, there was nothing in VBR.

As for the root cause: can you maybe check with the support engineer whether your issue is related to the BCJ issue in case 04882991? It sounds similar.

For the backup copy job feature request, just to clarify: all days were copied correctly, just the GFS points were not generated. correct?

Thanks,
Hannes

[SteelContainer]

Hi!
Thanks. That makes sense.

For the feature request: The source backup job has 28 days retention set in the job and it also had 12 months + 4 years in the GFS retention. The copy job (immediate/mirror) was set to the same, 28 days retention with 12months + 4 years in the GFS settings. Only the 28 days retention seems to get mirrored. We lost a lot of the monthly retention when the BCJs had the issue and then we couldn't mirror it back from the source backup job. It would be quite helpful if we could just set the job to mirror the GFS retention of the source job as well as the simple retention.

[SteelContainer]

Hi! Just FYI we had this issue again with another backup copy job (mirroring). Again we didn't get any alerts in VSPC since the job in the VBR console was not showing an error, also no alerts from the jobs RPO alarms so this issue went unnoticed.
We fixed the job again by using the following steps:
• Disable the job
• Backup the Veeam config and copy the Veeam job VBM
• Navigate to Home > Backup > Disk (Copy)
• Hold down the Ctrl key
• Right click the chain belonging to a job and, while still holding down Ctrl, click 'Remove from configuration.
• Rescan the repository
• Check it reappears
• Recreate a new job with the same settings and in the Target page, click Map Backup and select the corresponding backup chain.
• Enable the job and see if that allows the new job to run which will perform incremental if it's mapped.
Hopefully this issue is on the roadmap to be fixed in an update, missing these failed backups and then having to recreate the jobs is not ideal and we should be able to rely on Veaam's alerting.
Thanks!

[HannesK]

sounds like this issue - confirmed bug that is planned to be fixed.

I'm not merging the thread because that might cause confusion.

[SteelContainer]

Great, yes that sounds like the same issue. Look forward to the update to fix it.

Thanks!

[spiritie]

Hi! Just FYI we had this issue again with another backup copy job (mirroring). Again we didn't get any alerts in VSPC since the job in the VBR console was not showing an error, also no alerts from the jobs RPO alarms so this issue went unnoticed.
We fixed the job again by using the following steps:
• Disable the job
• Backup the Veeam config and copy the Veeam job VBM
• Navigate to Home > Backup > Disk (Copy)
• Hold down the Ctrl key
• Right click the chain belonging to a job and, while still holding down Ctrl, click 'Remove from configuration.
• Rescan the repository
• Check it reappears
• Recreate a new job with the same settings and in the Target page, click Map Backup and select the corresponding backup chain.
• Enable the job and see if that allows the new job to run which will perform incremental if it's mapped.
Hopefully this issue is on the roadmap to be fixed in an update, missing these failed backups and then having to recreate the jobs is not ideal and we should be able to rely on Veaam's alerting.
Thanks!

FYI: You can also just rename the job and the rename it back again, this resolves this issue.

[spiritie]

Can Veeam comment on why this happens sometimes? @HannesK

We have seen this many times on v10 installations across many different VBR installations, all same behaviour:

- Seems to happen randomly, usually only affect 1 job
- Job doesn't ever start, no logging occurs in the jobs folder, only timeout errors in the "Svc.VeeamBackup.log"
- Rescan repo doesn't fix it
- Run active full doesn't fix it (if I remember correctly)
- Pressing Sync Now either with "latest" of "all" doesn't fix it
- Disable and enable job doesn't fix it
- Rebooting VBR server doesn't fix it
- No SMTP email is sent, (as I understand this bug is fixed in 11a, since the logic has been changed to that the timeout starts will also give emails now)

[foggy]

Hi Gert, in the OP's case, the reason for such behavior was the fact that all the proxy servers were fully loaded and the job didn't even get to start and fail. As the job session didn't start, it did not send any notification. Could this be the case in your environment as well? As mentioned, in v11a the job will now send the notification in this case.

[spiritie]

Hi Foggy

No this is not the case. I'm sitting with a VBR installation that is doing this right now, and the only job running is the "broken" one, which times out after 15 minutes.

So for you question, barely any load is present currently on the VBR server, proxies and repository servers.

EDIT:
This has been the case on all of the servers where I've seen this issue. Log in to check daily backup status.
1. See job has failed during the night.
2. Pressing "sync" now to test the job again during the day where no other backup load is running, same error.
3. Rename job and rename back again, and wupti it works again

[foggy]

I've checked the case mentioned in the adjacent thread and this issue was also supposed to be addressed in v11a, not only the notification one. Since you're still seeing this behavior, please open a new case for a closer look.

[spiritie]

Hi Foggy

I believe the that we haven't seen this issue in v11. We hadn't noticed the fails for a few days and the server was upgraded only 2 days ago to v11a (from 10).

I believe the jobs got bugged in v10 and the broken states of these jobs are now present in our v11 installation.. I don't believe I will see this bug again now that we are on v11a.

The SMTP is a whole different bug in my opinion, these should have sent us an SMTP no matter what.