After v12 upgrade, backup to tape jobs started blocking backup jobs

[DonZoomik]

#05903470

After v12 upgrade I noticed that some backup jobs were occasionally stalling without obvious reason. While analyzing the problem I noticed that it only occurred when tape was running. I'm quite sure that this didn't happen in v11.
Currently I have a 25TB VM being written to tape that takes over 24 hours so it's quite annoying - tape job task for this VM has being running for ~14 hours while backup has been blocked for ~12. In other cases I saw that single VM being written to tape can block whole backup job, eg other VMs in the same job also don't progress (in that case ~10 out of ~20 were stalling).

Backup chains are pre-VM, upgraded to new v12 format. Forward incrementals with weekly syntheticals so it should never block as all the files should stay static on disk until retention hits.

[Mildur]

Hi Mihkel

Have you enabled the option "Prevent this job from being interrupted by source backup jobs" in your Tape jobs?
If yes, the source job will wait until the tape job has finished.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Best,
Fabian

[DonZoomik]

Yes, it is enabled. However I'm quite sure it did not block non-transforming source jobs in v11 as looking at previous concurrent backup and tape sessions shows no blocking. In this case this option is required as the Tape job includes both transforming agent self-managed jobs and non-transforming VMware jobs.

By transforming I mean Forever Forward Incremental and Reverse Incremental that require exclusive lock on chain for merge operations (that would disrupt tape jobs). In contrast non-transforming Forever Incremental with synthetics and Backup Copy with GFS result in static chains that don't require locks once source job ends.

I could possibly separate transforming source jobs to a separate tape jobs. However just after v12 upgrade (between tape sessions) I also changed Tape job's include objects from specific source jobs to include whole repositories. The effective dataset is the same but the tape job separation would defeat the purpose of the change - to include practically everything being backed up by default to tape jobs. However i doubt that changing include objects from job->repository had any effect and it's some subtle behavioral change. There are a few other tape jobs that still include specific jobs (backup copy jobs that in turn include whole reposories) and they also now see blocking - they don't need this option TBH but nevertheless they didn't block on v11.

[DonZoomik]

Support got carried off by tape drive locking (another tape job took over the drives at some point) and some tape write failures (probably a known hardware problem).

I tried disabling the "Prevent..." option on one running tape job that was locking out a source Backup Copy job. Immediately the tape job was terminated although the source job didn't need to do any transforming actions (Backup Copy with weekly GFS - practically the same as backup with weekly synthetics) listed here https://helpcenter.veeam.com/docs/backu ... ml?ver=120 :

Sometimes, the source job may start when the tape job is still running. This may cause a conflict if the source job needs to perform the following operations:
* Merge of backup files
* Reverse incremental backup transformation
* Compact of full backup file
* Retention Policy maintenance.
By default, the source job has priority. In this case, the tape job terminates with error and no data is written to tape. Select the Prevent this job from being interrupted by source backup jobs option if you want to give the tape job a higher priority. If this option is selected, the tape job will not terminate and will finish writing the data. If the source job needs to perform the listed above operations, it will wait for the tape job to finish. Note that in this case the source job may finish with a significant delay.

I currently suspect that new v12 True Split Machine chains are for some reason treated as transforming job. Purely a speculation but maybe this lock behavior is the default from old times and around ~v9.5 when locking improved and the option was introduced, some backup types started to be treated specially with no implicit locking, maybe someone forgot to update the special treatment list...
Unfortunately I don't think I can create a classic split machine chain anymore to test my theory as all my chains have been upgraded. Maybe a logical repository with per-machine files will help out, will try.

[DonZoomik]

After some back-and-forth and testing some options, there are other cases with similar behavior and the case has been escalated.

[alex.nhs]

Hiya,

Have you had any further update on this? We are having the exact same issue with a tape job that usually takes 60 hours. Since upgrading to V12 the backup job stops the tape job!

[Mildur]

Hi Alex

Latest state is, that a hotfix is prepared. Please open a support case and provide your log files.
If we can confirm that you are also affected by the same issue, you will also get the private hotfix when the fix is ready.

Best,
Fabian

[alex.nhs]

Hi Fabian,

Thank you this sounds very promising!!

We have Case #05959346 open at the moment if you can do anything to speed up this process that would be wonderful as currently we are unable to go to tape.

Thanks
Alex

[Mildur]

Hi Alex

Thanks for the case number.
I cannot promise an ETA yet. They started with the fix this week.
I will monitor our internal ticket.

Best,
Fabian

[JohnnyCache]

Same issue here. Like Alex, we have "Prevent this job from being interrupted by source backup jobs" disabled and our tape job is terminated by running disk backup jobs.
Case # 05951523

[DonZoomik]

Got the hotfix. The next row of tape jobs starts on Sunday so we'll see then.

[DonZoomik]

It seems at first glance that the hotfix has cured the issue.

[Mildur]

Hi Mihkel

Thanks for your feedback. I'm glad the hotfix helped.

Best,
Fabian

[semiller]

I'm also in need of the hotfix - Case # 05950713
Thank you!

[Mildur]

Hi

Thanks for the case number.
Your support engineer will confirm the issue in your environment and provide you with the hotfix.

Best,
Fabian

[Suzii]

I'm also having this issue post v12 upgrade - case #06015261

Could you please arrange for me to get this hotfix

[Mildur]

Hi Suzii

Welcome to the forum.
I replaced the contract number in your comment with the case number.

The hotfix which you were provided is not required with the most current cumulative patch:
https://www.veeam.com/kb4420

Best,
Fabian

[DonZoomik]

The custom registry flag is still required (in addition to hotfix or CP2).

[Mildur]

Hi Mikhel

Suzii got an another fix. I believe it was not the correct one for his case.
I escalated it to support management this morning.

Best,
Fabian

[Suzii]

Thanks Fabian,

I applied the first hotfix and my tape backups ran successfully for the first time since I upgraded to v12 - over 2 weeks ago.
Hopefully I will not need to manually rescan the repositories after the Synth Fulls run over the weekend which happened last weekend.
I will apply the cumulative patch after out EOM Offsite tape backups run on Sunday night so probably Monday or Tuesday.

case #06015261

[Mildur]

Hi Suzii

Thank you for the feedback. I'm glad the provided hotfix has solved it.
It's possible that it contains fixes for both issues.

Best,
Fabian

[Suzii]

Hi Fabian,

I have successfully installed the patch.
Unfortunately since I installed the hotfix we have found, particularly on the Daily Tape job, that the jobs seem to be taking a long time to complete. For example the Daily Tape job backs up under 1TB Tuesday to Friday and used to take about 45mins or so. Today was 870GB and took 104 mins. This is post cumulative update.

case #06015261

[Mildur]

Hi Suzii

Thank you for the update.
I'm sorry to hear that you see a lower tape performance. Was it for every tape session the same after the upgrade?
Let's wait till our support engineer had analyzed the new log collection.

Best,
Fabian

[Suzii]

Hi Fabian,

Sorry for the delay in responding, I have not been at work.
It appears to have impacted all the tape jobs but is most obvious in the dailys which are taking at least twice as long as prior to the upgrade.
I have uploaded 28 days of logs for the daily job which goes back prior to the upgrade.

case #06015261

[Mildur]

Hi Suzii

No worries.
I was informed this week, that we have a known issue with locking the new backup chain format in v12. We plan to solve it with v12a. For v12 Cumulative Patch 2 we have a workaround. Can you please try the jobs again with the following registry key?

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Name: TapeBackupLocksResolution
Type: DWORD
Value: 0

Best,
Fabian

[Suzii]

Hi Fabian,

Didn't see this until after the Daily tape job started yesterday but today it ran in just over 60 mins for 1TB - not quite as fast as it used to but considerably faster than since the upgrade. This is using 1 tape drive.
The Weekly job ran in slightly more than the usual time - under 15 hours for 24.5TB - used to take around 12-13 hours from memory. This uses 2 tape drives but one file is over 12 TB.

case #06015261

[ibarizz]

Hi Suzii

No worries.
I was informed this week, that we have a known issue with locking the new backup chain format in v12. We plan to solve it with v12a. For v12 Cumulative Patch 2 we have a workaround. Can you please try the jobs again with the following registry key?

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Name: TapeBackupLocksResolution
Type: DWORD
Value: 0

Best,
Fabian

Hey Fabian,

I just wanna know if that registry key may solve the situation I'm facing. Currently my tape backup jobs are stopped when the source backup job is started, and later they are not being resumed to where they got stopped like it used to happen in version 11a.
We have version 12 with the last patch applied.

Thank you!

[ibarizz]

Coming back from my last post, the registry key worked like charm!

[Mofamike]

Support got carried off by tape drive locking (another tape job took over the drives at some point) and some tape write failures (probably a known hardware problem).

I tried disabling the "Prevent..." option on one running tape job that was locking out a source Backup Copy job. Immediately the tape job was terminated although the source job didn't need to do any transforming actions (Backup Copy with weekly GFS - practically the same as backup with weekly synthetics) listed here https://helpcenter.veeam.com/docs/backu ... ml?ver=120 :

I currently suspect that new v12 True Split Machine chains are for some reason treated as transforming job. Purely a speculation but maybe this lock behavior is the default from old times and around ~v9.5 when locking improved and the option was introduced, some backup types started to be treated specially with no implicit locking, maybe someone forgot to update the special treatment list...
Unfortunately I don't think I can create a classic split machine chain anymore to test my theory as all my chains have been upgraded. Maybe a logical repository with per-machine files will help out, will try.

Thank you for this thread and for providing the patch + regkey.

Did I understand correctly that the option "job from being interrupted by source backup jobs" can remain activated after the patch and then only puts the disk job in the queue if the following happens in the disk job:

* Merge backup files
* Reverse incremental backup transformation
* Compact of full backup file
* Retention Policy maintenance.

If the disk job only performs an incremental backup and none of the above points apply, the disk job will run as normal without waiting for the tape job. Correct?

[DonZoomik]

To run without disruption, chain must be (synth) full + incremental. "Only incremental" implies forever forward incremental that requires merge on every session, therefor conflicting with tape job.

There's some other issue with "Changes in source backup files detected, rescan required". Chain is Synthetic Full + Incrementals so relevant files never are never updated, however this still triggers quite often (not always). Mostly an annoyance (jobs are retried) so I never investagated deeper.