Comprehensive data protection for all workloads
Post Reply
BertM
Influencer
Posts: 13
Liked: 7 times
Joined: Aug 18, 2016 6:16 pm
Full Name: Bert
Contact:

SOBR setup failure can result in backup job failure

Post by BertM » 1 person likes this post

Hello all,

TL:DR - If you have jobs with spaces in the name and add the repo to a SOBR, but the SOBR setup fails, it might make your jobs fail due to renaming the job folders by changing spaces to underscores.

Just wanted to bring some awareness to an issue we had. We are moving from tape to cloud (Backblaze), so yesterday I was setting up a SOBR that would use our primary on-prem repo (Windows ReFS) and Backblaze. We already have a lot of backups in the repo. While Veeam was configuring the SOBR, it failed due to a running tape backup job. The error said it couldn't rename D:\VeeamBackups\Backup Job Folder Name\ to D:\VeeamBackups\Backup_Job_Folder_Name\, because a file inside was locked by the tape backup job. No big deal, I'll wait until tomorrow.

Get to the office today, and a lot of VM backup jobs have failed. They all say they can't find the .vibs from yesterday. I looked at the repo, and the only files I saw are a .vbm and .vib for each VM. It appeared all my previous backups were gone. So, after a brief internal panic, I notified my boss, opened a ticket with Veeam, and started digging. After realizing the drive still had the same 120TB of data as yesterday, I noticed that most of my backup jobs had two folders. One with spaces and one with underscores (ie - "Backup Job Folder Name" AND "Backup_Job_Folder_Name"). The folder with the underscores had all my previous backups.

So, it appears the SOBR setup process had renamed most of them, hit the folder that was locked, and threw an error. But it didn't put the renamed folders back to their previous state or update the backup jobs to use the new folder name. I updated my support case, but missed the reply from support and decided to just try to fix it myself. I stopped any jobs that were running (and disabled SQL log jobs), moved the newly created folders to another place, renamed all my old folders back to the original, and ran a repo rescan. After that, the backup jobs work again.

FWIW, support suggested I use the Map Backup feature within the job config to map it to the new folder location, which would have probably been faster than waiting 30 minutes for the rescan. Oh well.

Veeam folks, could you consider adding a bit of logic to prevent this if the SOBR config fails part way through. Either have it return the folder names back or update the jobs that had folders renamed? Might save some future admin a mental kernel panic.

For reference, my support case was #07502153.

Thanks,
BertM
Post Reply

Who is online

Users browsing this forum: No registered users and 55 guests