Veeam Security Bulletin (September 2024)

[apolloxm]

We had received an email from Veeam said that Veeam released updates for all the products. Currently, we are using Veeam backup and replication and the version is 12.0.0.1420 P20230718. How can we install this security update? please advice us. Thanks! https://www.veeam.com/kb4649

[Mildur]

Hi Apollo

As mentioned in the KB you shared in your topic, all security vulnerabilities are fixed in our latest release.
You have to update your backup environment to v12.2:
https://www.veeam.com/kb4600

Download the iso from our website and upgrade your v12.0 installation to v12.2. The upgrade procedure is documented in our helpcenter:
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Best,
Fabian

[apolloxm]

We can't accept veeam provide such bad solution to customer.Per Veeam support policy, you need provide a hotfix to all the V12 veeam products. In your lifecycle support policy, you never said that we only need to use v12.2 then we can get the security fix. Currently we are using the V12 version. And You just released the 12.2 on the August 28, 2024 and you force asked customer to upgrade this version. As you know, when the new version released, the new bug or issue will be happened and we can’t say just upgrade the sever. We need to do more testing and planning to a go live window to install the new update. This need to take a lot of times to verify it. Please provide a smooth solution to us. Thanks!
https://www.veeam.com/product-lifecycle.html

[apolloxm]

I am not the only customer who need such solution. Please hear the voice of the customer.Thanks!

[DanielJ]

I agree. We just upgraded to 12.1 and want to avoid being early adopters. Also 12.2 is only offered as a full ISO, why is there no smaller upgrade option for 12.x?

I am also worried that the hotfix we received recently, for a long standing problem which was now finally solved (ticket 07324220), won't be included in 12.2 so we will have the same problems again.

[mjr.epicfail]

Can I also add that it would be a great courtesy if Veeam would inform the Veeam Cloud Service Providers before they announce these major vulnerabilities and that they are fixed in a version they released over a week ago...

[Gostev]

We can't accept veeam provide such bad solution to customer.Per Veeam support policy, you need provide a hotfix to all the V12 veeam products. In your lifecycle support policy, you never said that we only need to use v12.2 then we can get the security fix. Currently we are using the V12 version. And You just released the 12.2 on the August 28, 2024 and you force asked customer to upgrade this version. As you know, when the new version released, the new bug or issue will be happened and we can’t say just upgrade the sever. We need to do more testing and planning to a go live window to install the new update. This need to take a lot of times to verify it. Please provide a smooth solution to us. Thanks!
https://www.veeam.com/product-lifecycle.html

Please note that Veeam never in its history provided updates for earlier minor or maintenance releases. Always only for the current build. We do not have infrastructure nor resources to make it happen. If we were to ever engage in this, the entire R&D would be just backporting the same fixes to multiple branches of the same major release and doing nothing else.

Here are some relevant quotes from the Support Policy:

Fixes and resolutions are often rolled into the next product release, and others are included as part of the next maintenance release, and most urgent are addressed with a hotfix that can be applied on specific product version.
[...]
Hotfix development is only available to customers who are upgraded to the most recent publicly available build of impacted products.

We always expect customers to be on the latest update or release of their major version. To facilitate this, we try our best never to change system requirements within a major release, and avoid architecture changes of any significance (unless they are required to fix a hot support issue).

By the way, if you are still on V12.0 then you have already missed a number of maintenance and minor releases which included fixes to high severity vulnerabilities.

[Gostev]

Can I also add that it would be a great courtesy if Veeam would inform the Veeam Cloud Service Providers before they announce these major vulnerabilities and that they are fixed in a version they released over a week ago...

That's not possible unfortunately, vulnerability disclosure best practices require us to make vulnerabilities known to all users at once.

The reasons are pretty logical actually: there are no way to guarantee bad actors are excluded from the "preview" recipients and they will thus have ample time to create exploits and start exploiting before vulnerabilities before the majority of users had a chance to mitigate them or even learn about their existence.

Also, it doesn't work in practice. The moment first people get something from Veeam (those subscribed to instant security bulletins on the Veeam Support KB), they post it on Reddit first and read it later. With all other customers commenting how upset they are not to have received this from Veeam yet. True story from another vulnerability disclosure a few months ago.

[apolloxm]

Please note that Veeam never in its history provided updates for earlier minor or maintenance releases. Always only for the current build. We do not have infrastructure nor resources to make it happen. If we were to ever engage in this, the entire R&D would be just backporting the same fixes to multiple branches of the same major release and doing nothing else.

I can remembered that veeam provided Vulnerability solution for both of V11a and V12.
post479406.html#p479406

Veeam never think about what customer need. you think about yourself R&D.

You just released the 12.2 on the August 28, 2024 and you force asked customer to upgrade this version. As you know, when the new version released, the new bug or issue will be happened and we can’t say just upgrade the sever. We need to do more testing and planning to a go live window to install the new update. This need to take a lot of times to verify it. we need more time to upgrade, not like this.


Security patch priority should be highest priority. Veeam need to provide all the security patch to all supported version

[Gostev]

I agree. We just upgraded to 12.1 and want to avoid being early adopters.

You won't be an early adopter, we had nearly 10'000 backup servers reporting to be running 12.2 just 3 full business days after the release also, we of course checked with our Customer Support before making the announcement and it's dead quiet there as it comes to 12.2 (thus no sticky Top Support Issues tracker topic for 12.2 still).

Also 12.2 is only offered as a full ISO, why is there no smaller upgrade option for 12.x?

Please note that Veeam never provided "small upgrade option" for minor releases. It's a new product build so "smaller" is not an option even in theory: no files in the distribution are unchanged.

I am also worried that the hotfix we received recently, for a long standing problem which was now finally solved (ticket 07324220), won't be included in 12.2 so we will have the same problems again.

If you received it in the last few days then it has to be rebuilt for 12.2 (not a problem and can be done fast).

If it was a few weeks ago, then it should be automatically included in 12.2 (exceptions are rare, only for fixes that are perceived to be potentially dangerous to roll out to all customers).

[Gostev]

I can remembered that veeam provided Vulnerability solution for both of V11a and V12.
post479406.html#p479406

That is correct, we always provide security patches for the latest build of every major release that is still supported. If V11 was still supported today, it would have received the solution in the form of a new V11 build.

[apolloxm]

From now, I don't like veeam.!!!!!

[bbittner]

Hi,
I'm having trouble in finding more details about the security issues, especially about the critical one (9.8, CVE-2024-40711, - remote code execution).
Are there at least bit more information, for example, what api/interface is effected?
I have to rate this issue for our very specific installations.

[Spex]

Please note that Veeam never in its history provided updates for earlier minor or maintenance releases. Always only for the current build. We do not have infrastructure nor resources to make it happen. If we were to ever engage in this, the entire R&D would be just backporting the same fixes to multiple branches of the same major release and doing nothing else.

My guess is that the security issues have been known internally for several weeks - or even longer at Veeam. During that time, 12.1 was the current version, so it would be "normal" to provide a fix for 12.1. However, you have chosen to implement the fixes in 12.2, which is still in development/testing and to first release 12.2 before announcing the security ony some days later to stay in compliance with your support statement.

This is a risky move since a) what if you find major problems in 12.2 code during testing and b) you are urged to release 12.2 as soon as possible since high CVE scores. c) Also, for internal security compliance reasons, Veeam end users cannot wait for feedback from early adopters of 12.2.

Not the proceeding I would expect.

[apolloxm]

Yes, Veeam12.2 just released a few days , we don't know what is the bug. if V12.2 released for several months, then I can accept this.

[Gostev]

My guess is that the security issues have been known internally for several weeks - or even longer at Veeam. During that time, 12.1 was the current version, so it would be "normal" to provide a fix for 12.1. However, you have chosen to implement the fixes in 12.2, which is still in development/testing

Correct. And as you're aware, many security vulnerabilities were indeed resolved in 12.1.2 maintenance release. But not every vulnerability can be fixed in a few weeks. Some require many months of significant security architecture changes (think Specter/Meltdown that took Intel 2 years until first remediations) and they also tend to impact every component and feature, thus requiring at least a minor release vehicle which gets to enjoy multiple regression testing cycles due to it's relatively long release cycle.

These regression testing cycles is where we determine all existing functionality impacted by security changes, fix those regressions, then perform another regression testing cycle to ensure those last fixes did not break something else (in particular, interoperability with other functionality), fix those other newly introduced bugs, rinse and repeat a few more times. On top of that, this activity has to be coordinated with "normal" on-going bugfix/enhancements which touch the same modules and are often in the very same code that addresses the vulnerability, so it all has to be carefully coordinated.

As a result, some security fixes result in changes in most product modules and it's no longer just a small granular patch no matter how you look at it or name it.

Although I do get a feeling that this thread would not exist if we just called this release 12.1.3 instead of 12.2 - which we totally could because most of it is just new workload / new platform versions support and on-going bug fix, making it a pretty usual Veeam maintenance release. However, our marketing would not agree to this naming

[Spex]

Maybe you are right regarding the wording of the patchlevel and the existance of this thread. Also the what's new pdf for 12.2 is only 13 pages in comparison with 12.1 that has 24 pages.....

[Gostev]

Apologies, I forgot to address these:

This is a risky move since a) what if you find major problems in 12.2 code during testing

I've covered our regression testing process and how we address major problems found in 12.2 code testing in my previous response to you.

b) you are urged to release 12.2 as soon as possible since high CVE scores.

We were not urged to release 12.2 as soon as possible because all severe vulnerabilities were found during the internal testing by our AppSec QA team and thus not known to anyone outside Veeam. You can expect many more vulnerabilities found internally going forward as I tripled our AppSec QA team and it's not like we're talking about going from 2 to 6 people here it's a very large team now doing nothing but analyzing source code for vulnerabilities.

c) Also, for internal security compliance reasons, Veeam end users cannot wait for feedback from early adopters of 12.2.

This is not entirely true, the wait has already happened. This is the very reason why we did not publish the Security Bulletin right on the 12.2 release day, but waited for a week. Now, if you're a long term R&D Forums member you already know that thanks to VBR having close to a million active installs, any real issues with a new release result in a 1-2 pages topic on these forums within 24 hours after GA but there are none still after a week, nor our Customer Support have observed any significant issues reported on 12.2 to date. While as I've already shared earlier, we had nearly 10'000 backup servers reporting to run 12.2 in just 3 full business days after the release.

[Gostev]

Maybe you are right regarding the wording of the patchlevel and the existance of this thread. Also the what's new pdf for 12.2 is only 13 pages in comparison with 12.1 that has 24 pages.....

Great observation, to which I would also add that What's New for 12.1.2 (previous maintenance release) would not be much smaller either if we were to put it in a nicely formatted PDF and add more feature/benefit type of information

[Gostev]

I'm having trouble in finding more details about the security issues, especially about the critical one (9.8, CVE-2024-40711, - remote code execution).
Are there at least bit more information, for example, what api/interface is effected?
I have to rate this issue for our very specific installations.

As an industry best practice, vulnerabilities reporting and CVE creation in particular is handled by a different team here at Veeam, and that team reports directly to CISO. The logic is about adding checks and balances to the whole process, as R&D has its own priorities (release dates etc.) which addressing newly found vulnerabilities conflicts with. So it makes sense to have another team doing the vulnerability assessment and reporting, but it also means we in R&D do not control CVE articles content.

I will notify the corresponding team of your feedback but I highly recommend not being too mathematical in this particular case. First due to the number of vulnerabilities disclosed. And also due to one significant security architecture change that should prevent 80% of new vulnerabilities going forward (as over 80% of vulnerabilities that were found in VBR in the past couple of years were all around one particular technology and more or less the same attack vector). This was the big change I've been talking about in one of my previous responses which makes the entire class of vulnerabilities impossible to exploit going forward.

[bbittner]

Thanks for your reply!
It's really fantastic to have someone from Veeam here for such a kind of questions and answers. Please keep doing this.

[k00laid]

Please note that Veeam never in its history provided updates for earlier minor or maintenance releases. Always only for the current build. We do not have infrastructure nor resources to make it happen. If we were to ever engage in this, the entire R&D would be just backporting the same fixes to multiple branches of the same major release and doing nothing else.

@gostev I will say I know have -1 versions patched all the time. For service providers it is very typical for the absolute soonest upgrades begin is measured in months, much less weeks or days. We've had 1 week with these builds, there's too much QA and development work, for at least 2 major products for us, that is going to have to happen for this to be doable. Will be happy to discuss more in a more private setting.

[Gostev]

Hi, Jim. I'm sure that as technologists we can agree that "-1 version" approach is meaningless because we can be like Google Chrome and have Veeam Backup & Replication V128 now, and V129 in 3 months so while we did start with the "-1 version" approach in the early days of Veeam, we since switched to the "specific time period" approach like the majority of the industry is doing.

By the way, as you remember Veeam actually used to do annual major releases up to version 9, which meant just 2 years effective support cycle when using "-1 version" approach. Whereas V11 was supported for 3 years and V12 will likely be supported for 4 years (due to major architecture changes in V13, we may need to give our customers more time to migrate).

So, be careful what you wish for going back to the "-1 version" approach because advancing a major release version once a year is totally a fair game given the typical size of our annual releases. And who knows if our marketing decides to start doing this exactly next year?

[Gostev]

VCC/VSPC specific queries have been moved to the corresponding private forum > Veeam Cloud & Service Providers Forum

[SnakeSK]

This is seriously unheard of in the industry that the new version comes out as an excuse to not to backport critical RCE fix into the previous versions. Just my 2 cents

[m.novelli]

Please note that Veeam never provided "small upgrade option" for minor releases. It's a new product build so "smaller" is not an option even in theory: no files in the distribution are unchanged.

Hi Gostev, I have 60+ customers on VBR 12.1 , can you provide a 2 GB installer to upgrade them?

Downloading a 12 GB ISO is't totally a pain

Thanks, Marco

[Gostev]

There is no "new version of V12", what came out is the next build of V12 (just like the previous 12.1.2 was). It's an integral part of the continuous stream of cumulative V12 updates.

It *is* the fix you seek to be used to update any previous V12 builds in the list and get RCE fixes. Never in the history of Veeam did we have a different approach of patching a major release other than continuous stream of new builds that are effectively cumulative patches and include RCE fixes as well. It's been like that for over 16 years now so definitely not "unheard of".

[Gostev]

Hi Gostev, I have 60+ customers on VBR 12.1 , can you provide a 2 GB installer to upgrade them?

Downloading a 12 GB ISO is't totally a pain

Hi, Marco.

As it stands right now R&D can create smaller update packages for maintenance releases, but not for minor and major releases.

12.1 weren't any different by the way, it came as a full ISO only - so if you have your customers on any 12.1 build then you did it at least once before already.

Thanks

[m.novelli]

I think you should implements something like "Windows Update" on VBR and VBO (also VMware should do it for vCenter) so all SMB Customer can be set to "automatic update" just like Windows

Marco

[Gostev]

This will be the feature of Linux-based VBR appliance in V13. Users will be given a chance to initiate an update manually first, but will be forced into automated updated after X days if they don't. But only within a major release of course, no forced updates across major versions.