DB full corrupts SOBR offload?

[BrianBuchanan]

I have case 05101914 open, but for the second time my SOBR offload has broken after running out of Configuration DB Space. The first time I hit the 10-GB Limit of SQL Express and the second time was a log file filled the data disk.

First case was with v11 and this second time is with completely fresh build with V11a. In both cases I resolved the disk space issue but the SOBR Offload process hangs for hours and hours, backing up several offloads and more importantly it blocks the backup jobs from running. The backup job that used to run in 38 minutes now completes in 6 hours. The SOBR Offload task is near impossible to stop.

The last backup that completed in normal time was on the 19th and I'm considering restoring the configuration database from that day because I feel that the DB lost transactions causing the SOBR process to just not work properly anymore.

Both times the backup job failed saying that the metadata file was inconsistent. On the V11 db, a rescan of the SOBR took over 24 hours on the performance tier, but on the fresh v11a install the performance scan completed within 30 minutes and the capacity tier took 4 hours.

I feel that it's an integrity problem with the configuration database, but how can I fix it?

[HannesK]

Hello,
SOBR offload and the configuration database are relatively independent. So I assume that database issues are a coincidence, but not the root cause of the SOBR offload issues. But only support can verify that - I can only do assumptions on a forum.

Looking at the number of licenses related to that case, a rescan of 24h or even 30min sounds like an infrastructure issue to me.

You can always start "clean", from a configuration restore, but I'm not sure whether that really solves the root cause of your issues.

I would assume the SQL express should be okay in your environment. We usually say "up to 500 machines if no file-to-tape is used". So my main question is, why your database is that large (maybe very long session history? High amount of SQL jobs? Also a "classic" thing to check if your backup server is virtual: does the VM have virtual cores or virtual CPUs? Because SQL express only uses the first 4 cores of the first CPU.

Best regards,
Hannes

[BrianBuchanan]

Support also suggested a backup/restore of the configuration database. I'm just waiting the 4-6 hours for the jobs to stop It feels like the Backup Repository is the problem, but I have 2 other VBR servers using different paths on the same server and they are working. From the previous experience I know that if I setup a fresh VBR and back up the same target to the same repository, I'll get about 2.0GB/sec processing speed.

[BrianBuchanan]

Oh something else. My first server's DB itself was only ~7-GB, but the log was ~2.9-GB (SQL Express installed) and that started the problem. I have a single backup job for a 4-node Hyper-V Cluster with 130 VMs. Retention is about 30-days and offload to Azure immediately and keep 14-days on-site.

[BrianBuchanan]

The configuration backup and restore seems to have resolved the issue. There was an initial configuration database synchronization process that failed on the capacity tier and sealed it, and since it is sealed nothing is offloading. I'm waiting to hear back what the error was and how to proceed with the sealed capacity tier.

[BrianBuchanan]

Spoke too soon. Console is still showing old jobs until I press F5, successfully Resynch the SOBR (4 hours 25 minutes with a status of Skipped: 1) and unsealed. Now the SOBR Offload job is running at 2MB/s and preventing the backup job from running (1 hour 15 minutes, 0%, 2MB/s, bottleneck: detecting.

[BrianBuchanan]

It's starting to look like there's a bug in Veeam when the database hits the 10-GB limit of SQL Express, or the disk fills with SQL Enterprise that causes this behaviour. I don't know which function is affected, but it feels like it involves SOBR overloading the on-prem backup repository (is it constantly recalculating checksum of blocks because it wasn't recorded in the database?). If I still had a configuration database backup before that happened I could have probably restored with that. Continuing to work with Support.

[HannesK]

I have 2 other VBR servers using different paths on the same server and they are working

keep in mind that the three VBR servers don't know about each other and the repository server could be overloaded by too many tasks which might cause these issues.

If the VBR database hits the 10GB limit, then the results are unpredictable. I would not consider that as "expected behavior" instead of a bug. In simple mode, log files also should not grow to 2.9 GB.