Veeam Security Bulletin (September 2024)

[laurent.b]

We are exactly on the same case than the subject. We are block to th first version of 12.0 since we have datadomain too much full to be update to be compatible with the last update (Strange than 12 is compatible but now 12.2...) I've contacted the support, they said me to keep my version 12.0...how is possible? We need the security patch different than the update software.

[pirx]

We were not urged to release 12.2 as soon as possible because most of the severe vulnerabilities were found during the internal testing by our AppSec QA team and thus not known to anyone outside Veeam. And the external researcher who found the other was very collaborative and didn't rush us either.

I'm pretty sure I commented on that yesterday evening, but the comment was deleted without any note.

Anyway, assuming that a bug is not known to others just because the vendor doesn't know it, is just an assumption.

I'm also not happy how Veeam handles security updates and it different than other companies do it. We were affect by bugs in minor updates before and it is just not true that the forum is always full with comments about these after 2-3 days.

[np-mast]

We always expect customers to be on the latest update or release of their major version. To facilitate this, we try our best never to change system requirements within a major release, and avoid architecture changes of any significance (unless they are required to fix a hot support issue).

What was done by Veeam with Version 12.1 is that licensing was changed (I think it was for file-to-tape backup which now consumes way more licenses then previously). That't a major no-no to me, especially within a major release where there is no other way to get bug and security fixes.

[m.novelli]

We are exactly on the same case than the subject. We are block to th first version of 12.0 since we have datadomain too much full to be update to be compatible with the last update (Strange than 12 is compatible but now 12.2...) I've contacted the support, they said me to keep my version 12.0...how is possible? We need the security patch different than the update software.

Yeah, changing third-party firmware requirements inside the same release 11.x , 12.x , etc is not a good practice... Veeam should provide security fixes for all previous 12.x versions and, my 2 cents, also for 11.x versions (some "little and poor" customers are still on very old ESXi versions)

Marco

[Gostev]

I admit Data Domain DD OS version support situation in 12.1 was a huge oversight.

This was the first time in the history of Veeam when a change of system requirements happened within a major release. The root case was that Dell added support for the latest DD OS version while at the same time removing support for the oldest DD OS version from the DDBoost library (hey they are also pushing customers to upgrade) and the team felt forced by Dell to make the same change, so they never came to discuss. I'm not sure what could have been done, perhaps we could just load old DDBoost library when detecting old DataDomain OS version, however we never had a chance to discuss. @laurent.b perhaps we could still deliver a private fix like that to use with 12.2, please request it through support and we go from there (devs will need a support case to register their work to regardless).

The good thing is, the DD OS situation in 12.1 became a great learning experience and the prime example for the entire PM team why system requirements should never be changed within a major release.

[pirx]

@Gostev but wasn't it similar with IBM storage integration lately? It was failing for us after an update because it was suddenly using a new method and we needed first to find out that there is a regkey that restores old behaviour. In my opinion it should be the other way around and it was again a sign that even updating minor versions lead to impacts in production because functionality was changed.

[Gostev]

As long as fallback to the "old way" is available then I actually am supportive updating components, modules and APIs in immediate release vehicles, even in maintenance releases.

In case of Data Domain, we don't want anyone to remain on no longer maintained/patched DDBoost version, because it adds security risk to the entire customer base over just a few customers with very old systems.

Same with primary storage arrays integration: if IBM forced us to change the API we were using going forward, then I like to think there was a good reason for this (as trust me, no one likes to touch what's working, especially our R&D).

So changes themselves are fine within a major release when they are justified - we must always strive to keep 3rd party components up to date for better security posture - but in such cases I ask the team to always maintain a fallback option at least until through the given major release lifecycle.

[Dooh]

Today's security landscape requires a bit more transparency that we needed in the past. The update was available for a week before it was made known that the update addresses such high severity vulnerabilities. That is a week where we took this as a feature update and did not update deployments with a sense of urgency, but attackers were taking that update and looking for anything that got patched to see how they can exploit it. Had we known that there were such issues being addressed, we would have been updating our customers, but now we are a week behind our attackers, and that is never a good place to be. We have seen too often where after a vulnerability is released that attackers start exploiting it within hours.

Additionally, there was no info about what part of Veeam was vulnerable or how it was vulnerable, nor was listed any other way to mitigate the vulnerability via anything like a firewall rule or something similar. This means I have customer that will be vulnerable for a lot longer than we would like. Between the download being so large, and the fact that it is a full upgrade, some environments will take a long time to get this update deployed.

Most of my customers we will be fine to have this deployed quickly, but others will not be so lucky. For more challenging or sensitive deployments, might I suggest making a version of 12.x that an LTSB build? Customers with the LTSB build would not get new features, but would get security updates that can be deployed in a quick manner without needing to go through internal QA testing?

Lastly, is there any way to get a version of the deployment that does not have all of the plugins? That is half of the download size right there

[m.novelli]

I agree with all your points Dooh

Especially the .ISO size...
And the update time is sooooo long... the installer run half hour? I should measure this

Marco

[Gostev]

Dooh,

The public disclosure was delayed by a few days on purpose due to the number and complexity of security changes, some of which took a few months to implement and test due to potentially impacting much functionality. So we gave it a few days until 12.2 reached 10'000 installs to ensure we did not miss any big regressions. Unfortunately that got us into the weekend followed by the U.S. public holiday, which would certainly give bad actors the above-mentioned advantage to make a disclosure then. This set of factors resulted in the public announcement on the next week after the release only.

We do document mitigations when they are available. In this case, mitigations are unfortunately not available.

Your LTSB suggestion assumes all security updates are something minor and can patched "in a quick manner". Sometimes it is the case but unfortunately the opposite is also common with mitigation affecting almost every component and feature. As a result, at least in the current situation, the "security-only" build would not have significantly less changed lines of code comparing to the current 12.2. But if we had one, our Dev/QA would have to work on two very large updates separately, which was deemed ineffective and thus just did not make sense in this particular case.

Whenever we encounter a severe vulnerability that is quick to fix, we just do a quick maintenance release, like for example 12.1.2 was. Thing is, we always have a "working branch" for the current release where all the private hotfixes produced for our Customer Support are being merged. So we can "seal" it at any point in time, add fixes for those newly discovered vulnerabilities and, following a quick regression cycle, ship it as a maintenance release. But again, this requires vulnerabilities which can be quickly patched with a granular, isolated fix - which are historically speaking the majority, but not all vulnerabilities we encountered to date.

Unfortunately, at this time minor and major releases are available in a form of full ISO only. Only maintenance releases have a "smaller" option. It's hard to have a "smaller" option for more significant releases because all agents and plug-ins are updated, but each customer is using a different subset of those agents and plug-ins, so which of those do we even put on a "smaller" option if not all of them?

Thanks

[tyler.jurgens]

Lastly, is there any way to get a version of the deployment that does not have all of the plugins? That is half of the download size right there

I'd even take a web installer with each optional item as something that can be selected. If selected for install, it will download those at that time.
Keep that along side a full installer for those air gapped environments that can't connect to the internet.

[m.novelli]

Yeah, I think that with VBR 12.2 we reached the biggest ISO offense ever and something from Veeam side should be done, even on the Windows platform, not only on the next v13 Linux appliance.
In my SMB environment I don't use any plugin (Azure / GCP / AWS / Kasten / oVirt / Proxmox / Veeam Agent for multiple platforms / Nutanix...)
The wasted time downloading the ISO, the wasted space on disk and the wasted time for installation has reached an offensive time / size
Some cherry picking is needed now, like the suggested web installer

Marco

[Gostev]

Wait until you have to download OVA/OVF, Marco

[Dooh]

The public disclosure was delayed by a few days on purpose due to the number and complexity of security changes, some of which took a few months to implement and test due to potentially impacting much functionality. So we gave it a few days until 12.2 reached 10'000 installs to ensure we did not miss any big regressions. Unfortunately that got us into the weekend followed by the U.S. public holiday, which would certainly give bad actors the above-mentioned advantage to make a disclosure then. This set of factors resulted in the public announcement on the next week after the release only.

Please do not take this as an attack, I am very thankful for everything you do and your engagement on here, and I love the Veeam platform, I just want to help make it better (and not have my teams hate me for pushing Veeam)

In today's world, we can't wait days to know something critical was patched. Have a look at this breakdown: https://labs.watchtowr.com/veeam-backup ... 4-40711-2/

In 4 days since the vulnerabilities were released, this team was able to determine what the vulnerabilities were and write it up in a presentable manner. The attackers can just skip that last part and get to writing an attack probably on day 2.
For the attackers, that was August 30th or maybe September 1st.
For us on defense, that was 4-5 days before we knew we even had a problem.

This also really highlights something that I hope can be addressed: Easy of upgrade. For those of us with hundreds of customers, having to manually connect to each one and download this massive update and install it is a major effort if we can't automate things. I am excited for the things you mentioned about v13 though!

I agree with all your points Dooh

Especially the .ISO size...
And the update time is sooooo long... the installer run half hour? I should measure this

Marco

We had some customers take 6 hours to upgrade from v11 to v12, not all CPUs are created equal unfortunately or under our control. I forget who said it earlier in this thread, but some customers are constrained by budget and stuck on less-than-optimal deployments

[m.novelli]

One more devastating issue that I've got is that (so far) on two customers the SUPER BIG ISO download was corrupted and the upgrade failed in the middle. So I needed to re-download the ISO, uninstall Veeam, reboot, reinstall Veeam, reboot, check...

Marco

[Gostev]

Marco, we provide checksums for all downloads. Just check it before running the upgrade. It's the best practice with unreliable Internet connections regardless of the download size.

"Reboot required" discussion is not specific to this update either so I split it into the dedicated topic.

To future posters: please avoid derailing this discussion and do create separate topics for everything that is not directly related to this topic (Security Bulletin). They are completely free

[ferrus]

In case of Data Domain, we don't want anyone to remain on no longer maintained/patched DDBoost version, because it adds security risk to the entire customer base over just a few customers with very old systems.

I can understand the reasoning, but consider the position now. We're now stuck on an outdated, insecure version of Veeam B&R (managed to at least patch the Veeam EM vulnerability as it's a separate 12.2 update), and also prevented from the latest VMware 8u3 patches.
The one decision to drop support for EoL DD in 12.2 has left us in a much less secure position that with an older unmaintained DDBoost.

I take responsibility that we shouldn't as an organisation be using the DataDomains beyond their support date, but they were quite a significant investment, still working perfectly well - and though the processes are in place to select their replacement - they just aren't happening quick enough for us.

The trickle down effect is significant for us, and without any backported patching it's going to be an uncomfortable time reading Veeam and VMware security bulletins, until the DD replacements get the green light.

[Gostev]

The one decision to drop support for EoL DD in 12.2 has left us in a much less secure position that with an older unmaintained DDBoost.

This was in 12.1 actually, so almost 1 year ago soon... no changes in 12.2 at it comes to DD.

[pesos]

Have to agree here - for multiple sites the ISO size has just gotten absurd and clearly some rethinking of the distribution approach is necessary.

Ridiculous to have to download gigs upon gigs of plugins we will never use - why not at least split the plugin download from the main product as one consideration?

[Gostev]

All customers use a different set of agents and plug-ins. Deployments which use none are quite rare these days...

[huddie]

Please note that Veeam never provided "small upgrade option" for minor releases. It's a new product build so "smaller" is not an option even in theory: no files in the distribution are unchanged.

Gostev, this simply isn't true. 11.x had numerous small security hotfix patches released during its lifetime. These were not just released as 12GB ISO in-place upgrades, they were made available to install automatically and quickly via the VB&R Console Upgrade button. I think we need to be clear about something here, a 12GB ISO in-place upgrade is not a security hotfix patch. 12.2.0.334 did include the security hotfixes but was a full in-place upgrade. Veeam need to explain why they've stopped issuing security hotfix patches and replaced this patching method with full in-place upgrades. It's taken me all day today to patch from v12.1.2.172 to v12.2.0.334 and it's caused issues and disruption.

[Gostev]

It's true, you're just not reading carefully what I write...

We did provide "small upgrade options" for maintenance releases, but never for minor or major releases. Even V12 had such "small" options available for the most recent 12.1.1 and 12.1.2 maintenance releases. So neither did we "stop issuing" anything, nor we're doing things ANY differently comparing to what we've been doing all the time. I'm sorry but these are just facts, no matter how inconvenient they are. This does not mean there's no place for improvement of course, but to claim we're doing something different with 12.2 than before is simply unfair.

If you look at the full build number, this is how you can determine the release vehicle designation depending on what octet has changed:

Starting from V12.1 we're using industry standard way
New build of the same release: 4th octet (only used to differentiate RTM and GA builds when they are different, for example at V12 launch)
Maintenance release: 3rd octet (examples are above-mentioned V12.1.1 and V12.1.2)
Minor release: 2nd octet (currently discussed V12.2)
Major release: 1st octet (V13 next year)

Before V12.1 we used custom/weird Veeam way
New build of the same release: 4th octet (only used to differentiate RTM and GA builds when they are different, for example at V11 launch)
Maintenance release: PDDDDMMYY designation of unchanged build number
Minor release: 3rd octet (V11a)
Major release: 1st octet (V12)
Not used at all: 2nd octet

It's taken me all day today to patch from v12.1.2.172 to v12.2.0.334 and it's caused issues and disruption.

Technically speaking, the installation of 12.2 should have been faster than "small upgrade options" of 12.1.2 for example, because 12.2 does not require unpacking self-extracting EXE first before the installation can actually start. Would you like to open a support case to let us investigate why the installation of 12.2 took a full day while your upgrade logs are still available?

[huddie]

Gostev, you're missing the point. Patches should have been issued as out-of-band security patches. I'm not concerned about how you name such releases or how build numbers change. Customers should not have to do a full in-place upgrade every time a Veeam vulnerability needs to be patched. If Microsoft did what Veeam are doing users would have to do a full in-place upgrade of Windows every month.

We did provide "small upgrade options" for maintenance releases, but never for minor or major releases.
Even V12 had such options available for the most recent 12.1.1 and 12.1.2 maintenance releases.

The 'small upgrade option' for v12.1.1.56 to v12.1.2.172 never appeared on the Upgrade button in VB&R Console for me. This had to be done as a full ISO / in-place upgrade.

[Gostev]

There could only be two reason for this update notification not appearing:
a/ You have "Check for product and hypervisor updates periodically" check box on the Options > Notifications tab unchecked.
b/ Your firewall is blocking backup server from accessing Veeam update servers.

But this is a moot point I guess, what matters is 12.1.2 does exist, it had a "small upgrade option" and it did address security vulnerabilities. So we did exactly what you just said you expect Veeam to do.

Unfortunately, and here I'm repeating what I already explained once or twice earlier in this topic, not all vulnerabilities can be addressed in a quick and small maintenance release. Some require major architecture changes which in turn touch and break/affect almost every single product feature, resulting in a big rewrite and requiring at least a minor release vehicle to ship due to the sheer number of code changes.

[huddie]

a/ You have "Check for product and hypervisor updates periodically" check box on the Options > Notifications tab unchecked.

Nope. This box is still checked, as it always has been.

b/ Your firewall is blocking backup server from accessing Veeam update servers.

Also nope. Our firewall blocks no outgoing connections and this has always been the case. Its config hasn't changed.

The update button worked in v11 and now doesn't work in v12.

It would be good to hear from others in Veeam who can clarify the changes in how security hotfix patches are released.

[Gostev]

Nope. This box is still checked, as it always has been.
Also nope. Our firewall blocks no outgoing connections and this has always been the case. Its config hasn't changed.

Try opening from your backup server https://dev.veeam.com/backup/vbrupdate.vdf
If this downloads the file correctly, then check if the file contains <ComponentVersion>12.1.2.172</ComponentVersion>
If yes, next check if you can see this same file/content in C:\ProgramData\Veeam\vbrupdate.vdf (it is downloaded there automatically)
If it's there, then you should have received a notification about 12.1.2 available from the backup console. Let me know if you're convinced you did not receive it and I can ask QA to try installing v12.1.1.56 and see if they get the notification or not.

It would be good to hear from others in Veeam who can clarify the changes in how security hotfix patches are released.

Actually I run R&D at Veeam, so you won't get any different information from others in Veeam. There has been no changes to how security hotfix patches are released: whenever possible, they are and will continue to be made available as a part of maintenance releases like 12.1.2 was. Otherwise, they will be included in the next minor and/or major releases only, depending on a scale of the architectural changes required. Nevertheless, the latest build of every major release will receive remediations for security vulnerabilities for as long as the major release is actively supported - and usually even a bit longer in case of critical vulnerabilities (both V10a and V11a received such courtesy security patches when they were out of support already).

[pirx]

Unfortunately, and here I'm repeating what I already explained once or twice earlier in this topic, not all vulnerabilities can be addressed in a quick and small maintenance release. Some require major architecture changes which in turn touch and break/affect almost every single product feature, resulting in a big rewrite and requiring at least a minor release vehicle to ship due to the sheer number of code changes.

Even if a security fix requires a major rewrite, it does _not_ justify to bundle it with a bunch of new features. Have the latest fixes been such major rewrites?

We are really having more and more issues with Veeam's policies regarding security. I wrote it a while ago, other vendors provide their older releases much longer with security fixes, even without LTS releases. Much written here from Gostev focus on Veeam's R&D aspects but not much on customer view.

[Gostev]

I don't know if it's fair to say that "not much on customer view" has been written in this thread so far!

Have the latest fixes been such major rewrites?

Yes, they have. Almost half a year of working through areas impacted by certain fixes. The good news is, these architectural changes will prevent the whole class of vulnerabilities going forward.

[pirx]

If the vulnerability would have been exploited and not reported to Veeam, would the fix then also have been this major rewrite?

I'm not feeling really any better knowing that there was such a high score CVE and Veeam needed months to fix this. Just because Veeam was not aware of it, does not automatically mean that it's not known to others.

[Gostev]

This vulnerability just did not have an isolated fix available - unlike for example those vulnerabilities addressed in 12.1.2. But you should not be shocked about complex changes around security requiring a few months - keep in mind it took Intel 2 years after the initial report to deliver first remediations for Spectre/Meltdown vulnerabilities.

Luckily, such vulnerabilities are less common while the majority can be addressed with a fairly quick and isolated fix. If I remember correctly, previous vulnerability that took a few months to address was back in 10a (so almost 4 years ago).

There were no known exploitation of this vulnerability at the time we disclosed it. Our Information Security team monitors darknet and uses many other special sources not available to ordinary mortals like me, but they do notify us the moment something pops up on their radar. I'm sure this will not always be the case though, for example among 79 Windows vulnerabilities of this week's Patch Tuesday, 4 were already being actively exploited in the wild at the time of disclosure. But that's life.

In any case, folks this has been already a full week of this "interview", I gave it a fair go and invested much of my time to answer many questions and concerns, some multiple times at this point I would appreciate that we do not continue belaboring this, as I have no mental or physical capacity to continue at this level of engagement (some of you know there are multiple similar threads in private subforums as well and I had to actively participate in all of them).

Overall the release seem to be a successful one with close to 70'000 unique downloads, 40'000 backup servers reporting to be running 12.2 already (lower number than actual due to backup servers sending this data once a week only) and yet very few known issues in the sticky topic, most of which are hard for me to blame QA for missing. Probably it would be fair to declare this release a success, aside of the download size of course!