mkretzer's known V12 issues

[mkretzer]

Since we upgraded to V12 we had several issues (and for which I already created several threads and cases) which might be edge cases – still in this post I want to document the current status of all our V12 issues - this might help some of you who still have not decided on V12 upgrade.
All in all V12 runs smoothly - but there were a lot of workaround needed:

Slow BCJ performance on BCJ with a lot of VMs. Case 06232794
- Status: Workaround available.
- For BCJ using V11 chains switching between job objects is very slow. This can be fixed by upgrading backup chains to V12 format (both source & target).

Very slow tape object scanning for upgraded chains. Was investigated within the case 06252634
- Status: Will be fixed on V12.1, (bad) Workaround available.
- After upgrading our backup chains to V12 format it takes 1-2 Minutes for the needed backup files to be detected at the start of a tape job. With thousands of VMs this can be an issue as the writing to tape starts after the scan is complete. Veeam cannot provide a patch for V12 but told me it will be fixed in V12.1. A manual start of an active full tape backup does not show this behavior and can be a valid workaround – which still brings problems like no way to retry after a backup failure. For us this would have been a "breaking change" .

Background retention job blocks/fails backups. Case 06264521
- Status: Hotfix available, Workaround available, optimization planned for V12.1 .
- The background retention job can block primary backups. These Backups might fail/need retry. A hotfix is available (reference case 02476388) which changes the priority of the retention job. Another workaround is to disable the background retention altogether (which we did) via a registry setting (SystemRetentionDisable). Further optimization limiting background retention to Orphaned backups might be implemented in V12.1 (not yet certain from what I understand).

General interface slowness, especially when trying to restore Objects. Case 06252634
- Status: Hotfix available, will be fixed in V12.1
- A fix for slow console is available, but we did not test this yet. Reference case 06252634.

“Corrupted” restore Points when backing up templates that have incremental backups and “Exclude templates from incremental backup” on after chains have been upgraded Case 06258440
- Status: Workaround available, will be fixed in “further Versions”
- The per-VM V12 metadata seems to get corrupted when a template gets changed (converted to VM and back) and thus job finds an incremental backup when trying to create a synthetic backup. This causes the backup job to fail and display a corruption message. In fact, the backup points are not corrupted (verified by heath check and validator). By disabling “Exclude templates from incremental backup” in the job excludes (https://helpcenter.veeam.com/docs/backu ... ml?ver=120) this is fixed.

Primary backup jobs fail tape jobs Case 06298510
- Status: Workaround available
- V12 tape jobs can be blocked/failed by primary backup jobs, even with periodic synthetic backups enabled. This can be fixed by setting “TapeBackupLocksResolution“ to 0 in registry (see tape-f29/tapes-fail-when-primary-backup ... 89466.html). This can happen for non-upgraded and upgraded chains!

Command line tools like “Veeam.Backup.Manager.exe -SHOWPROXYUSAGES” do no longer work with MFA enabled. Case 06233911
- Status: Documentation updated
- When using command line tools MFA must be temporarily disabled for the user running the tools.

V12 Job starts jobs on days no backup is scheduled when synthetic is configured for that day Case 06234650
- Status: Works as designed, Workaround available
- V12 automatically starts a backup on days which are excluded for scheduling – for us that was an issue as on days with synthetic backups we did start the jobs earlier via Powershell (to be able to start the tape GFS scan before 23:59:59 which is the last moment a GFS scan is able to run on a day). By blocking backup run at specific times of the day (the time the backup normally starts) we were able to still use the old logic.

Even if some of these bugs were very frustrating my thanks go out to Veeam support who were willing to help with each of these issues. I still hope for some fixes before V12.1 but currently we can live with all but one of these issues...

Markus

[Dima P.]

Hello Markus,

Big thanks to you while working on these issues and root cause analysis. I've update your post with the case IDs for the future reference!

[Dima P.]

Hello Markus,

12.1 is out and the slow processing rate issue fix is included (as well as other features, improvements and fixes of course), please give it a spin and let us know how it goes. Thank you!

[mkretzer]

Will do. Should we wait days or weeks before we update (how bad is the support load currently)?

[bct44]

Interested by the answer of Mkretzer question , I need to update related to a case, support told me no hotfix because it will be fixed in v12.1.

[Dima P.]

The support team is fully prepared for this release, so should be business as usual. Regarding the fixes, indeed some of the heavy ones were not provided previously via standalone private fix delivery due to amount of changes required in the product, instead such fixes were aimed at 12.1 as delivery vehicle.

As for deployment or upgrade, I'd recommend to deploy the build in the lab on the first place, get a hands on experience on what's new/changed and then plan upgrade for the production servers.

[mkretzer]

For us all worked in Lab, i just worry that the issues are again with scalability and our bigger environment.
I found it interesting that the changes for copy job performance scalability which Veeam support helped us with made it to the release notes: "Improved scalability — following many optimizations, V12.1 quadruples the recommended scalability limit for backup copy jobs to a maximum of 2,000 machines per job". For the first time our 1800 VM copy jobs are now fully supported

[Dima P.]

For us all worked in Lab

Sounds great!

i just worry that the issues are again with scalability and our bigger environment.

Is it possible to run a test backup via the lab deployment?

the changes for copy job performance scalability which Veeam support helped us with made it to the release notes

Cant thank enough all you guys enough for submitting these cases to help us with investigations and further improvements.

[mkretzer]

Sadly, i do not have thousands of VMs in the test environment. But knowing us we will update next week or so...

[bct44]

@mkretzer i'm curious how did you notice "Slow BCJ performance on BCJ with a lot of VMs"? I hadn't noticed because even my jumbo jobs (around 1k VMs) finish within 24 hours, but on closer inspection, it seems quite slow. Good to know, i was outside best pratices for backup copy jobs before v12.1... True per-vm backup files allowed to rebalance the organisation of certain jobs.
The only thing i had noticed was the memory usage was more high during backup copy jobs windows than other time interval of the day on the repo, much more than during the backup windows.

About "'Background retention job blocks/fails backups" Have you encountered any full storage not found errors or orphaned VIB after rotating backup chains?

I will probably planned to update the dev part with vbr with with several thousand vms. Even if less necessarily reprensative than the production vbr with less load and veeam functionalities used.

Sorry if I'm squatting on your topic, but I'm also very interested in your problems

[mkretzer]

In May 2021 we consolidated our BCJs and a maximum of 1800 VMs in one source and copy job. That caused Veeam to not be able to copy the backups in under 24 hours. Our goal was that the copy should be finished in < 6 hours. We worked with support and in the then-current version we tested several patches which optimized this (which we are very happy about).

We disabled background retention as recommended by support. Since than all is fine.

[bct44]

Hello,

I did the upgrade on production today, for now so far so good.
Jumbo BCJ goes skyrocket, i love seeing 1-2GB/S throughput on multiple jumbo BCJ

[mkretzer]

We upgraded to 12.1 and now all our Backups with non-upgraded chains fail!! Case 07052866

[Dima P.]

Markus,

Sorry to hear that, checking your case. Is that backup copy, primary backups or both? Thank you.

[mkretzer]

Primary, copy won't copy failed points. We need it working in the next few hours!!!!

[Dima P.]

Thank you for the update. Passed all the details to scale-out repository team, meanwhile please continue to work with your support engineer.

[mkretzer]

I called in because i heard nothing from support. The new engineer tells me now that there are notes on the case and it looks like no one from Dev is working on this. I asked him to escalate....

[Dima P.]

Markus,

QA team is aware and investigating the uploaded logs.

[Dima P.]

Markus,

A quick question from QA team (since we suspect this one to be a scale-out repository issue), can you please clarify the type of the extents that are included in scale-out repository (Windows / Linux / Object Storage etc)? Thank you!

[mkretzer]

Hello,

Windows ReFS.

Markus

[mkretzer]

FYI we just backed up over thousand VM to the same SOBR to chains which were upgraded before 12.1.
Still waiting for a workaround for legacy chains.

[mkretzer]

Problem solved by Chain Upgrade. According to support V12.1 basically does no longer support the old chain format. @Gostev this should be updated in documentation/checked at upgrade time!!

[Gostev]

Actually, I can't imagine this being intentional or intended because the majority of our customer base will go to V12.1 directly from V11. And will thus need to follow the same process as people who upgraded from V11 to V12.0, slowly upgrading all of their jobs one by one while already on V12.1.

But glad to hear you're good now. Frankly speaking, these transition phases we provide are never meant to be very long and span multiple releases in any case. You always want to complete them as fast as possible, to quickly get into the "better QA-ed" envelope. Which is always the "default" configuration!

[mkretzer]

Yes i would really have preferred upgrading all chains on 12.0 already. But the tape bug prevented that. I just hope this bug is really solved with 12.1. We will see in about 24 hours...

[mkretzer]

Just so others can find the error we had - the message was:

Processing xxxx Error: Cannot transfer meta to the master extent. All failbacks failed. Failed to rename file 'X:\Backups\XXXXX\XXX.vbm.temp' to 'XXX.vbm'

Workaround is to upgrade the backup chains!

[Dima P.]

Additionally, QA suspects that this error couple caused by Windows Defender blocking the files on scale-out repository. If you faced similar behavior please make sure that the path to the repository on the scale-out extent is whitelisted in the repository properties. Thank you!

[mkretzer]

We don't use Defender but use another AV which never showed this issue before. I find it very strange that this happens only with non-upgraded chains and only with 12.1!

[jessey316]

Hi Dima,

I have encountered the same issue using Crowdstrike. We have an all-in-one setup, so the Veeam server and repository are on the same server.

Could you confirm the path that needs to be whitelisted or excluded?

[mkretzer]

https://www.veeam.com/kb1999
But according to Reddit thread https://www.reddit.com/r/Veeam/comments ... il/setting the exclusions does not help!

[Dima P.]

Unfortunately that seems to be true. Based on this discussion looks like Crowdstrike AV still interferes during our backup operations while single storage format is used.

Can confirm that for other AVs we've tested excluding the path to a repo folder works like a charm but with Crowdstrike the backup chain upgrade seems to be the only workaround.