-
- Service Provider
- Posts: 371
- Liked: 120 times
- Joined: Nov 25, 2016 1:56 pm
- Full Name: Mihkel Soomere
- Contact:
After v12 upgrade, backup to tape jobs started blocking backup jobs
#05903470
After v12 upgrade I noticed that some backup jobs were occasionally stalling without obvious reason. While analyzing the problem I noticed that it only occurred when tape was running. I'm quite sure that this didn't happen in v11.
Currently I have a 25TB VM being written to tape that takes over 24 hours so it's quite annoying - tape job task for this VM has being running for ~14 hours while backup has been blocked for ~12. In other cases I saw that single VM being written to tape can block whole backup job, eg other VMs in the same job also don't progress (in that case ~10 out of ~20 were stalling).
Backup chains are pre-VM, upgraded to new v12 format. Forward incrementals with weekly syntheticals so it should never block as all the files should stay static on disk until retention hits.
After v12 upgrade I noticed that some backup jobs were occasionally stalling without obvious reason. While analyzing the problem I noticed that it only occurred when tape was running. I'm quite sure that this didn't happen in v11.
Currently I have a 25TB VM being written to tape that takes over 24 hours so it's quite annoying - tape job task for this VM has being running for ~14 hours while backup has been blocked for ~12. In other cases I saw that single VM being written to tape can block whole backup job, eg other VMs in the same job also don't progress (in that case ~10 out of ~20 were stalling).
Backup chains are pre-VM, upgraded to new v12 format. Forward incrementals with weekly syntheticals so it should never block as all the files should stay static on disk until retention hits.
-
- Product Manager
- Posts: 9327
- Liked: 2475 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hi Mihkel
Have you enabled the option "Prevent this job from being interrupted by source backup jobs" in your Tape jobs?
If yes, the source job will wait until the tape job has finished.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120
Best,
Fabian
Have you enabled the option "Prevent this job from being interrupted by source backup jobs" in your Tape jobs?
If yes, the source job will wait until the tape job has finished.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Service Provider
- Posts: 371
- Liked: 120 times
- Joined: Nov 25, 2016 1:56 pm
- Full Name: Mihkel Soomere
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Yes, it is enabled. However I'm quite sure it did not block non-transforming source jobs in v11 as looking at previous concurrent backup and tape sessions shows no blocking. In this case this option is required as the Tape job includes both transforming agent self-managed jobs and non-transforming VMware jobs.
By transforming I mean Forever Forward Incremental and Reverse Incremental that require exclusive lock on chain for merge operations (that would disrupt tape jobs). In contrast non-transforming Forever Incremental with synthetics and Backup Copy with GFS result in static chains that don't require locks once source job ends.
I could possibly separate transforming source jobs to a separate tape jobs. However just after v12 upgrade (between tape sessions) I also changed Tape job's include objects from specific source jobs to include whole repositories. The effective dataset is the same but the tape job separation would defeat the purpose of the change - to include practically everything being backed up by default to tape jobs. However i doubt that changing include objects from job->repository had any effect and it's some subtle behavioral change. There are a few other tape jobs that still include specific jobs (backup copy jobs that in turn include whole reposories) and they also now see blocking - they don't need this option TBH but nevertheless they didn't block on v11.
By transforming I mean Forever Forward Incremental and Reverse Incremental that require exclusive lock on chain for merge operations (that would disrupt tape jobs). In contrast non-transforming Forever Incremental with synthetics and Backup Copy with GFS result in static chains that don't require locks once source job ends.
I could possibly separate transforming source jobs to a separate tape jobs. However just after v12 upgrade (between tape sessions) I also changed Tape job's include objects from specific source jobs to include whole repositories. The effective dataset is the same but the tape job separation would defeat the purpose of the change - to include practically everything being backed up by default to tape jobs. However i doubt that changing include objects from job->repository had any effect and it's some subtle behavioral change. There are a few other tape jobs that still include specific jobs (backup copy jobs that in turn include whole reposories) and they also now see blocking - they don't need this option TBH but nevertheless they didn't block on v11.
-
- Service Provider
- Posts: 371
- Liked: 120 times
- Joined: Nov 25, 2016 1:56 pm
- Full Name: Mihkel Soomere
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Support got carried off by tape drive locking (another tape job took over the drives at some point) and some tape write failures (probably a known hardware problem).
I tried disabling the "Prevent..." option on one running tape job that was locking out a source Backup Copy job. Immediately the tape job was terminated although the source job didn't need to do any transforming actions (Backup Copy with weekly GFS - practically the same as backup with weekly synthetics) listed here https://helpcenter.veeam.com/docs/backu ... ml?ver=120 :
Unfortunately I don't think I can create a classic split machine chain anymore to test my theory as all my chains have been upgraded. Maybe a logical repository with per-machine files will help out, will try.
I tried disabling the "Prevent..." option on one running tape job that was locking out a source Backup Copy job. Immediately the tape job was terminated although the source job didn't need to do any transforming actions (Backup Copy with weekly GFS - practically the same as backup with weekly synthetics) listed here https://helpcenter.veeam.com/docs/backu ... ml?ver=120 :
I currently suspect that new v12 True Split Machine chains are for some reason treated as transforming job. Purely a speculation but maybe this lock behavior is the default from old times and around ~v9.5 when locking improved and the option was introduced, some backup types started to be treated specially with no implicit locking, maybe someone forgot to update the special treatment list...Sometimes, the source job may start when the tape job is still running. This may cause a conflict if the source job needs to perform the following operations:
* Merge of backup files
* Reverse incremental backup transformation
* Compact of full backup file
* Retention Policy maintenance.
By default, the source job has priority. In this case, the tape job terminates with error and no data is written to tape. Select the Prevent this job from being interrupted by source backup jobs option if you want to give the tape job a higher priority. If this option is selected, the tape job will not terminate and will finish writing the data. If the source job needs to perform the listed above operations, it will wait for the tape job to finish. Note that in this case the source job may finish with a significant delay.
Unfortunately I don't think I can create a classic split machine chain anymore to test my theory as all my chains have been upgraded. Maybe a logical repository with per-machine files will help out, will try.
-
- Service Provider
- Posts: 371
- Liked: 120 times
- Joined: Nov 25, 2016 1:56 pm
- Full Name: Mihkel Soomere
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
After some back-and-forth and testing some options, there are other cases with similar behavior and the case has been escalated.
-
- Novice
- Posts: 3
- Liked: never
- Joined: Mar 22, 2023 8:31 am
- Full Name: Alex Sparks
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hiya,
Have you had any further update on this? We are having the exact same issue with a tape job that usually takes 60 hours. Since upgrading to V12 the backup job stops the tape job!
Have you had any further update on this? We are having the exact same issue with a tape job that usually takes 60 hours. Since upgrading to V12 the backup job stops the tape job!
-
- Product Manager
- Posts: 9327
- Liked: 2475 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hi Alex
Latest state is, that a hotfix is prepared. Please open a support case and provide your log files.
If we can confirm that you are also affected by the same issue, you will also get the private hotfix when the fix is ready.
Best,
Fabian
Latest state is, that a hotfix is prepared. Please open a support case and provide your log files.
If we can confirm that you are also affected by the same issue, you will also get the private hotfix when the fix is ready.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Novice
- Posts: 3
- Liked: never
- Joined: Mar 22, 2023 8:31 am
- Full Name: Alex Sparks
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hi Fabian,
Thank you this sounds very promising!!
We have Case #05959346 open at the moment if you can do anything to speed up this process that would be wonderful as currently we are unable to go to tape.
Thanks
Alex
Thank you this sounds very promising!!
We have Case #05959346 open at the moment if you can do anything to speed up this process that would be wonderful as currently we are unable to go to tape.
Thanks
Alex
-
- Product Manager
- Posts: 9327
- Liked: 2475 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hi Alex
Thanks for the case number.
I cannot promise an ETA yet. They started with the fix this week.
I will monitor our internal ticket.
Best,
Fabian
Thanks for the case number.
I cannot promise an ETA yet. They started with the fix this week.
I will monitor our internal ticket.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Novice
- Posts: 7
- Liked: 1 time
- Joined: Aug 12, 2020 12:26 pm
- Full Name: Thomas S.
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Same issue here. Like Alex, we have "Prevent this job from being interrupted by source backup jobs" disabled and our tape job is terminated by running disk backup jobs.
Case # 05951523
Case # 05951523
-
- Service Provider
- Posts: 371
- Liked: 120 times
- Joined: Nov 25, 2016 1:56 pm
- Full Name: Mihkel Soomere
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Got the hotfix. The next row of tape jobs starts on Sunday so we'll see then.
-
- Service Provider
- Posts: 371
- Liked: 120 times
- Joined: Nov 25, 2016 1:56 pm
- Full Name: Mihkel Soomere
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
It seems at first glance that the hotfix has cured the issue.
-
- Product Manager
- Posts: 9327
- Liked: 2475 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hi Mihkel
Thanks for your feedback. I'm glad the hotfix helped.
Best,
Fabian
Thanks for your feedback. I'm glad the hotfix helped.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Lurker
- Posts: 1
- Liked: never
- Joined: May 05, 2015 11:47 am
- Full Name: Stanley M.
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
I'm also in need of the hotfix - Case # 05950713
Thank you!
Thank you!
-
- Product Manager
- Posts: 9327
- Liked: 2475 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hi
Thanks for the case number.
Your support engineer will confirm the issue in your environment and provide you with the hotfix.
Best,
Fabian
Thanks for the case number.
Your support engineer will confirm the issue in your environment and provide you with the hotfix.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Novice
- Posts: 6
- Liked: never
- Joined: Apr 26, 2023 10:34 pm
- Full Name: Suzii Stevens
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
I'm also having this issue post v12 upgrade - case #06015261
Could you please arrange for me to get this hotfix
Could you please arrange for me to get this hotfix
-
- Product Manager
- Posts: 9327
- Liked: 2475 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hi Suzii
Welcome to the forum.
I replaced the contract number in your comment with the case number.
The hotfix which you were provided is not required with the most current cumulative patch:
https://www.veeam.com/kb4420
Best,
Fabian
Welcome to the forum.
I replaced the contract number in your comment with the case number.
The hotfix which you were provided is not required with the most current cumulative patch:
https://www.veeam.com/kb4420
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Service Provider
- Posts: 371
- Liked: 120 times
- Joined: Nov 25, 2016 1:56 pm
- Full Name: Mihkel Soomere
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
The custom registry flag is still required (in addition to hotfix or CP2).
-
- Product Manager
- Posts: 9327
- Liked: 2475 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hi Mikhel
Suzii got an another fix. I believe it was not the correct one for his case.
I escalated it to support management this morning.
Best,
Fabian
Suzii got an another fix. I believe it was not the correct one for his case.
I escalated it to support management this morning.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Novice
- Posts: 6
- Liked: never
- Joined: Apr 26, 2023 10:34 pm
- Full Name: Suzii Stevens
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Thanks Fabian,
I applied the first hotfix and my tape backups ran successfully for the first time since I upgraded to v12 - over 2 weeks ago.
Hopefully I will not need to manually rescan the repositories after the Synth Fulls run over the weekend which happened last weekend.
I will apply the cumulative patch after out EOM Offsite tape backups run on Sunday night so probably Monday or Tuesday.
case #06015261
I applied the first hotfix and my tape backups ran successfully for the first time since I upgraded to v12 - over 2 weeks ago.
Hopefully I will not need to manually rescan the repositories after the Synth Fulls run over the weekend which happened last weekend.
I will apply the cumulative patch after out EOM Offsite tape backups run on Sunday night so probably Monday or Tuesday.
case #06015261
-
- Product Manager
- Posts: 9327
- Liked: 2475 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hi Suzii
Thank you for the feedback. I'm glad the provided hotfix has solved it.
It's possible that it contains fixes for both issues.
Best,
Fabian
Thank you for the feedback. I'm glad the provided hotfix has solved it.
It's possible that it contains fixes for both issues.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Novice
- Posts: 6
- Liked: never
- Joined: Apr 26, 2023 10:34 pm
- Full Name: Suzii Stevens
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hi Fabian,
I have successfully installed the patch.
Unfortunately since I installed the hotfix we have found, particularly on the Daily Tape job, that the jobs seem to be taking a long time to complete. For example the Daily Tape job backs up under 1TB Tuesday to Friday and used to take about 45mins or so. Today was 870GB and took 104 mins. This is post cumulative update.
case #06015261
I have successfully installed the patch.
Unfortunately since I installed the hotfix we have found, particularly on the Daily Tape job, that the jobs seem to be taking a long time to complete. For example the Daily Tape job backs up under 1TB Tuesday to Friday and used to take about 45mins or so. Today was 870GB and took 104 mins. This is post cumulative update.
case #06015261
-
- Product Manager
- Posts: 9327
- Liked: 2475 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hi Suzii
Thank you for the update.
I'm sorry to hear that you see a lower tape performance. Was it for every tape session the same after the upgrade?
Let's wait till our support engineer had analyzed the new log collection.
Best,
Fabian
Thank you for the update.
I'm sorry to hear that you see a lower tape performance. Was it for every tape session the same after the upgrade?
Let's wait till our support engineer had analyzed the new log collection.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Novice
- Posts: 6
- Liked: never
- Joined: Apr 26, 2023 10:34 pm
- Full Name: Suzii Stevens
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hi Fabian,
Sorry for the delay in responding, I have not been at work.
It appears to have impacted all the tape jobs but is most obvious in the dailys which are taking at least twice as long as prior to the upgrade.
I have uploaded 28 days of logs for the daily job which goes back prior to the upgrade.
case #06015261
Sorry for the delay in responding, I have not been at work.
It appears to have impacted all the tape jobs but is most obvious in the dailys which are taking at least twice as long as prior to the upgrade.
I have uploaded 28 days of logs for the daily job which goes back prior to the upgrade.
case #06015261
-
- Product Manager
- Posts: 9327
- Liked: 2475 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hi Suzii
No worries.
I was informed this week, that we have a known issue with locking the new backup chain format in v12. We plan to solve it with v12a. For v12 Cumulative Patch 2 we have a workaround. Can you please try the jobs again with the following registry key?
Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Name: TapeBackupLocksResolution
Type: DWORD
Value: 0
Best,
Fabian
No worries.
I was informed this week, that we have a known issue with locking the new backup chain format in v12. We plan to solve it with v12a. For v12 Cumulative Patch 2 we have a workaround. Can you please try the jobs again with the following registry key?
Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Name: TapeBackupLocksResolution
Type: DWORD
Value: 0
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Novice
- Posts: 6
- Liked: never
- Joined: Apr 26, 2023 10:34 pm
- Full Name: Suzii Stevens
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hi Fabian,
Didn't see this until after the Daily tape job started yesterday but today it ran in just over 60 mins for 1TB - not quite as fast as it used to but considerably faster than since the upgrade. This is using 1 tape drive.
The Weekly job ran in slightly more than the usual time - under 15 hours for 24.5TB - used to take around 12-13 hours from memory. This uses 2 tape drives but one file is over 12 TB.
case #06015261
Didn't see this until after the Daily tape job started yesterday but today it ran in just over 60 mins for 1TB - not quite as fast as it used to but considerably faster than since the upgrade. This is using 1 tape drive.
The Weekly job ran in slightly more than the usual time - under 15 hours for 24.5TB - used to take around 12-13 hours from memory. This uses 2 tape drives but one file is over 12 TB.
case #06015261
-
- Service Provider
- Posts: 53
- Liked: 6 times
- Joined: Jul 28, 2021 2:36 pm
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Hey Fabian,Mildur wrote: ↑May 12, 2023 7:43 am Hi Suzii
No worries.
I was informed this week, that we have a known issue with locking the new backup chain format in v12. We plan to solve it with v12a. For v12 Cumulative Patch 2 we have a workaround. Can you please try the jobs again with the following registry key?
Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Name: TapeBackupLocksResolution
Type: DWORD
Value: 0
Best,
Fabian
I just wanna know if that registry key may solve the situation I'm facing. Currently my tape backup jobs are stopped when the source backup job is started, and later they are not being resumed to where they got stopped like it used to happen in version 11a.
We have version 12 with the last patch applied.
Thank you!
-
- Service Provider
- Posts: 53
- Liked: 6 times
- Joined: Jul 28, 2021 2:36 pm
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Coming back from my last post, the registry key worked like charm!
-
- Influencer
- Posts: 20
- Liked: never
- Joined: Aug 29, 2019 12:23 pm
- Full Name: Mofamike
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
Thank you for this thread and for providing the patch + regkey.DonZoomik wrote: ↑Mar 04, 2023 12:54 pm Support got carried off by tape drive locking (another tape job took over the drives at some point) and some tape write failures (probably a known hardware problem).
I tried disabling the "Prevent..." option on one running tape job that was locking out a source Backup Copy job. Immediately the tape job was terminated although the source job didn't need to do any transforming actions (Backup Copy with weekly GFS - practically the same as backup with weekly synthetics) listed here https://helpcenter.veeam.com/docs/backu ... ml?ver=120 :
I currently suspect that new v12 True Split Machine chains are for some reason treated as transforming job. Purely a speculation but maybe this lock behavior is the default from old times and around ~v9.5 when locking improved and the option was introduced, some backup types started to be treated specially with no implicit locking, maybe someone forgot to update the special treatment list...
Unfortunately I don't think I can create a classic split machine chain anymore to test my theory as all my chains have been upgraded. Maybe a logical repository with per-machine files will help out, will try.
Did I understand correctly that the option "job from being interrupted by source backup jobs" can remain activated after the patch and then only puts the disk job in the queue if the following happens in the disk job:
* Merge backup files
* Reverse incremental backup transformation
* Compact of full backup file
* Retention Policy maintenance.
If the disk job only performs an incremental backup and none of the above points apply, the disk job will run as normal without waiting for the tape job. Correct?
-
- Service Provider
- Posts: 371
- Liked: 120 times
- Joined: Nov 25, 2016 1:56 pm
- Full Name: Mihkel Soomere
- Contact:
Re: After v12 upgrade, backup to tape jobs started blocking backup jobs
To run without disruption, chain must be (synth) full + incremental. "Only incremental" implies forever forward incremental that requires merge on every session, therefor conflicting with tape job.
There's some other issue with "Changes in source backup files detected, rescan required". Chain is Synthetic Full + Incrementals so relevant files never are never updated, however this still triggers quite often (not always). Mostly an annoyance (jobs are retried) so I never investagated deeper.
There's some other issue with "Changes in source backup files detected, rescan required". Chain is Synthetic Full + Incrementals so relevant files never are never updated, however this still triggers quite often (not always). Mostly an annoyance (jobs are retried) so I never investagated deeper.
Who is online
Users browsing this forum: No registered users and 24 guests