BUG: Failed / cancelled / multiple Backup Move operations cause "Error: More than one backup was found for the same job"

[AlexHeylin]

Case 06377207

Backup Move fails ungracefully if backup files a locked open, for example by Backup Copy Job. While it disables source backup jobs it does not disable backup copy hobs. As a result the attempt to move screws up database entries and results in:

- After stopping all BCJ Subsequent move does move the files (though as per case, towards the end this can be so slow that data flow cannot be seen) but the DB is screwed up so the Backup Move job fails with:
Failed Updating configuration database records Error: More than one backup is attached to job after orphaninig [sic] and deletion of source backups during moving
The name of the Backup is shown on the left with a status of Success, but the overall job fails. This appears to be because the DB updates are processed by the parent job, not the per-backup task. Obviously the move of the backups was not successful because of the DB issue.

- After move completes (according to Backups view) - Backups fail with:
Error: More than one backup was found for the same job (0f176e29-a70c-452b-bc83-697751a6b63d):
Backup NAME HERE (7da2af27-da8e-4aaf-8117-7c3fc72150e6)
Backup NAME HERE (c668d781-8226-427e-8816-97f629a1891b)

There's also an issue where going back and forth while editing a job having changed the repo can cause multiple Backup Move jobs to be queued up for the same backups.

Thanks

[HannesK]

Hello Alex,
just to be sure... you did not ask support to confirm a bug and there is no bug number yet, right?

I will check, but it will take some time (12.1 needs to be finished )

Best regards,
Hannes

[AlexHeylin]

Hi Hannes,
In the case I asked for three things to be done

1. Urgent - Need to get DB fixed so backups can run. Backup job cannot be edited due to the duplicate records in the DB.
2. Medium priority – find cause of database record duplication and open bug report / enhancement request to ensure this does not happen.
See veeam-backup-replication-f2/bug-failed- ... 90619.html
3. Now low priority - Investigation into why Backup Move slowed down so much that data movement appeared to have ceased. Create enhancement request etc to improve this in future release.

It looks like 1 was done, and 2 & 3 ignored. So as far as I can tell there's no bug filed for the move failing and messing up the DB, nor has there been any investigation into why the move appears to have slowed down to almost nothing for several hours near the end of the process.

I hoped that having reported them, and asked for the work to be done that support would have spun these tasks off to the appropriate people. Possibly they did, but did not confirm this or report on status. If you're able to get someone to pick these up, I think it would be a good idea.

Thanks

Alex

[HannesK]

Hello,
I checked with Support. We were not able to find out / reproduce the duplicate database record. The duplicate record was the problem (could have existed since longer time) and the "move" then created the issue. The case logs and DB unfortunately don't show the exact way that the backups got to such a state. According to Support it's possible this was the result of another issue that may have left a not-good state, but with the info on the case it wasn't possible to confirm.

In general: once a ticket is closed, the case is done. Logs are deleted relatively quickly and then nobody can check anything anymore.

Best regards,
Hannes

[AlexHeylin]

Hi Hannes,
I can tell you what happened from a user interface point of view, and I think it rules out the duplicate record existing prior to the Move command being used.
Prior to the move, the backup job could be edited - so there was no duplicate record at that time.
The move command was used and it disabled the backup job, but did not stop or disable a backup copy which I had not noticed was still running. Due to the RPs being locked by the BCJ, the subtask in the move job failed. However it appears that the DB update happens in the move JOB not the subtask, and despite the move subtask failing the DB update proceeded and this did partially update the DB which created the duplicate.
From this point the backup job becomes uneditable due to the duplicate. At this point we raised the case. While in our scenario of what we were doing the impact of the workaround was acceptable, there are scenarios where the workaround would have had unacceptable impact and obviously the whole issue and workaround are best avoided anyway.

If what I think happened, it should be very easy to reproduce by moving any backup from one on-disk repo to another on-disk repo while a BCJ is reading from the backup. It should also be easy to confirm that the Move command does not gracefully handle a running BCJ, by either looking at the code or reproducing.

I'm now uploading the logs and DB to Case #06405825. Please use these for your investigation. I'm not pushing this for my benefit, but to help you improve the product if it does what it seemed to do to me.

Thanks very much
Alex

[dougy]

Alex,
They provide a solution, I believe we are experiencing the exact/similar same issue.

[AlexHeylin]

Dougy - they provided a workaround which was good enough to get us out of the hole.

Please can you add your case number to this thread, and if you've not already the please upload logs going back to a few days before this occurred. Please also take a DB backup BEFORE trying to resolve, and another afterwards and submit those to your case too. https://www.veeam.com/kb1471

Read this in full and consider the impacts on YOUR scenario before doing ANY of these steps. This is from memory so may not be correct. Use at your own risk! I recommend contacting support before proceeding.

1. Wait for or force stop the BCJ so it's not running. Disable it.
2. Redo the move and wait until it fully completes.
3. Right click the original backups (in new location) and select Detach from job. This will make them orphans.
4. If you still can't edit the backup job targeting the backups you just moved you can right click on it and clone the backup job, point it to the NEW backup location, remap the backups, and continue. It's possible you'll need to do an active full backup (or that it will do an active full on the first run). I think in theory this may not be required, but we needed to run active full anyway as we were changing from SMB to ReFS storage and wanted to activate block cloning.
5. Run the new job and wait for it to complete to make sure everything is OK.
6. Update any BCJ etc which are using the old job as a source so they're using the new backup job instead
7. Once you're sure you don't need to copy any settings over from the old job to the new job, delete the old job.
8. Ensure all jobs are now enabled.

Good luck!

[HannesK]

Hello Alex,
fair points... thanks for pushing again... I talked to support to check again.

Thanks for helping us to improve the product

Best regards,
Hannes

[AlexHeylin]

Hi Hannes,

Did this get looked at?

We're using the Copy Backup function as part of a migration for another client and we appear to be seeing similar stalling after the data itself has finished copying.
Case #06405825 — Case logs for Hannes, follow on from Case 06377207 - if you'd like to refresh your memory.

Many thanks

Alex

[HannesK]

Hello,
yes, R&D looked at it and could not find the reason. As far as I see, the storage was decommissioned before the root cause could be investigated.

Best regards,
Hannes

[ExanSoftware]

We are just hitting this same message but for a different reason after the upgrade to v12.

We have 1 backup job (We have 12 others working as expected) which is linked to 2 Backup Copy jobs to separate cloud repo's for each backup job. After the upgrade receive a similar message to your original. Reaching out to support shortly but it was interesting and wanted to post.

Failed to process "JobNameHere" (Incremental) (0 B) at 2/23/2024 7:49:25 AM Error: More than one backup was found for the same job (4324a5b9-ee65-4c27-a648-b40ca1ff117a): "JobNameHere" (0b82bb57-1ff6-4d41-9b2a-278aa2d356df) "JobNameHere" (a99b6d4c-af26-4a83-9cde-9ce92fb7cf8d)

[AlexHeylin]

Thanks for adding this @ExanSoftware. It's good to know that some of these things break other than when I'm near them...

[HannesK]

Hello,
@ExanSoftware: can you please share the support case number?

Thanks,
Hannes

[ExanSoftware]

Case # 07149423

[HannesK]

thanks, I will check with support