Include Security Fixes in Release Notes

[controlfreak]

We have compliance requirements to evaluate software patches/updates for security fixes within 30 days of release. Currently, Veeam product release notes do not contain a list of bug fixes or security fixes. This is somewhat unusual for software release notes. Please include at a minimum, security fixes in the release notes.

These documents don't contain any bugfix/security fix informtion.
https://www.veeam.com/veeam_agent_windo ... _en_rn.pdf
https://www.veeam.com/veeam_backup_9_5_ ... tes_rn.pdf

This document points out (page 19) that OpenSSL vulnerabilities have been fixed, but do not clarify if it is in Veeam One, Veeam Backup and Replication, or with Veeam Agent for windows. As such, I have to assume that the agent 3.0 release contains security fixes for OpenSSL.
https://www.veeam.com/veeam_backup_9_5_whats_new_wn.pdf

I worked with my account manager to get this information, and they suggested creating a forum post to generate interest for the "Feature". Please add security and bug fix information to release notes.

Thanks,

-Control

[Gostev]

Thanks, good suggestion. I do remember mentioning some security fixes in the What's New for Update 4, particularly around storage integrations. I will see how can we do a better job tracking them as regular "new features", so that they don't get lost. I have an idea that should work without causing too much pain to anyone in R&D.

[controlfreak]

Great, thanks. Looking forward to what you come up with.

[controlfreak]

Hey Gostev,

I am doing patch evaluation for NERC CIP Compliance today (every 30 day process). Agent version 3.0.1 released, but the release notes do not contain any fix information. I can't determine if it is a security patch due to this lack of information. Can you please confirm for me if the agent update contains any security or vulnerability fixes?

More on this topic, KB2926 states: "In addition, Update 4a addresses over 300 minor Update 4 bugs reported by customers and found during the internal testing."

It would be extremely helpful to have access to the full bugfix list so I can effectively evaluate veeam updates for security fixes. If not, I really need a way to confirm that those 400 bugfixes do not contain a security fix.

Due to the inadequate release notes for the new agent 3.0.1, I'll be forced to install the newest agent on all our CIP classified critical assets unless I can confirm by Thursday that the update doesn't contain a security fix. Due to these compliance requirements, I would be running a very new agent version that released yesterday on critical bulk electric system assets. It would be AWESOME if I could say without a doubt whether or not an agent update contains security fixes. If they don't, I can choose to deploy to critical systems after a thorough test period.

Thanks for listening to my compliance woes!

Control

[PTide]

Hi,

Can you please confirm for me if the agent update contains any security or vulnerability fixes?

Which agent are you referring to?

It would be extremely helpful to have access to the full bugfix list so I can effectively evaluate veeam updates for security fixes. If not, I really need a way to confirm that those 400 bugfixes do not contain a security fix.

That is, you are ok with either of ways:

a) An official statement that says "no security fixes included". In this case you'll be able to "skip" the update and perform your own tests for as long as you please in order to feel comfortable with installing the update.

b) An official list of fixed bugs with descriptions OR an official statement that says: "contains security fixes". If the latter, then you'll have to install it right away. If the former, you'll have to examine the list in order to figure if there are any security fixes.

Is that correct?

I would be running a very new agent version that released yesterday on critical bulk electric system assets.

That doesn't sound right to me. IMO, when things come to critical systems, every new software release has to brew for while in a test polygon, no matter what the software vendor claims. One doesn't simply install the newest stuff on the the next day after release.

Thanks!

[controlfreak]

Hi PTide, Happy to clarify.

I am looking at the newest agent release, version 3.0.1. We are required by NERC CIP-007 R2.2 to evaluate security patches for applicability within 35 days of release.

Acceptable evidence for the patch evaluation process includes release notes containing bugfix information that we can review to make a determination, or a statement from the vendor that release x.y does not contain any security fixes. We could use either.

In the case of a security patch being released, I do need technical information about the vulnerability or a CVE number. The reason for this is that NERC CIP R2.3 requires that we install security patches within 35 days of the evaluation date, or that we create a security patch mitigation plan within 35 days of the evaluation. We have the option to mitigate security patches if we are concerned about production impact, but we can't create a mitigation plan without technical details of the vulnerability. The mitigation plan requires us to document details of the vulnerability, technical controls to mitigate the vulnerability, and a date for the end of the mitigation (the end of mitigation requiring patch installation or software version update to non-vulnerable version). The mitigation plan has to be signed off by upper management and submitted to the regulators, so it must be adequately detailed and accurate.

Regarding testing prior to deployment, we always test on dev systems prior to deployment to production. Our patch cycle includes an evaluation install week and multiple production install weeks, with the most critical system being patched several weeks after the evaluation assets.

Ultimately, in order to make informed decisions regarding our compliance patch program, we need timely information about security fixes contained in veeam software releases. We do this every month, so a self-help option is preferred to opening a support ticket every month. Veeam is the only vendor for which I evaluate patches that doesn't put this information in the release notes.

Let me know if you need any other specific detail.

Thanks,

Control

[PTide]

Thank you for the clarification. I guess you want such lists to be maintained for both agents (we have two of them - for Windows and for Linux, and both go with the same version number 3.0.1).

Thanks!

[controlfreak]

Currently we only use the windows agent, but we would support the change on both to be future proof!

[michaelsbak]

This would be very helpful for me as well.

[controlfreak]

This is still an issue for me and affects our reliability compliance requirements. Veeam Support - Case # 04030080.

The Release Notes and What's New documents for Veeam agent 4.0 do not list what, if any, security vulnerabilities were fixed. Support recommended I discuss this on the forum. We need this information to remain compliant with Federal Critical Infrastructure Regulations.

Thanks,

Control

[Gostev]

We did make this a part of our release process based on your earlier feedback, and I've seen drafts of KB articles floating around, so they should be published soon.

[controlfreak]

Hey Gostev,

We are on very tight timelines for our regulatory requirements. We need that information at the time of release. When and where will this document be published?

Also, please tell me right now, does the windows agent update to 4.0 contain any security fixes from the 3.0.2.1170 version?

Thanks,

Control

[bcampbell]

Hey Gostev,
I am working with veeam using the same CIP processes controlfreak is using. It is very difficult to show our compliance team what any of these released actually involve. Having a detailed release note at the time of being published on the veeam website would be the best.

I currently use this KB https://www.veeam.com/kb2683 to basically check every month if there is an update. Since version 4.0 came out this month I need to asses this version within 35 days to see if it is security related.

[Gostev]

Bryan, roger that.

We need that information at the time of release. When and where will this document be published?

Normally, going forward the plan is to make this information available at the time of GA. This first time it will be a bit later, since the whole process is just being established. But, it won't take very long.

Unfortunately, I can't provide any specific detailed information at the moment. You will have to wait until the official KB articles are published.

[controlfreak]

Why can't you tell me if there are security fixes in the latest release?

[Gostev]

Because I don't have the information for Veeam Agent for Windows.

[controlfreak]

Can you find the draft KB you mentioned and look for me? I opened a ticket with support, but they directed me back here. I need to do my due diligence to meet my compliance requirements, but I can't do that if you guys withhold information.

[bcampbell]

Support also directed me to here and my account manager. But a community Forum is also not a place that my compliance team likes to grab important information from. It needs to be posted somewhere credible.

Are controlfreak and I the only NERC CIP companies using Veeam?

[PTide]

@bcampbell

But a community Forum is also not a place that my compliance team likes to grab important information from. It needs to be posted somewhere credible.

Would you ask them to define what kind of resources they find 'credible', please? This forum is constantly monitored and moderated by Veeam development, security, and product management teams. The same applies to Veeam website. If that is not 'credible' enough, then I am not even sure what would be : )

Thanks!

[Gostev]

No, I agree that despite of these facts, a community forum will never be perceived as the credible source. This is exactly why we're intended to publish this information in the support KB articles.

Can you find the draft KB you mentioned and look for me?

I only saw a draft of a KB article for Veeam Backup & Replication. But, it is safe to assume that our agents also had some security related fixes, because big parts of code base are shared.

Are controlfreak and I the only NERC CIP companies using Veeam?

Likely not, but I don't recall seeing similar requests in the past 12 years, before this topic was created.

[controlfreak]

Just to chime in, the veeam forum would be credible enough for our organization, if the post came from an official Veeam account. We would also accept the information in a support request, or on the mentioned KB.

[bcampbell]

Thanks Gostev. I understand this may not be in your normal process yet. I just started using veeam in the past few months and have only recently been doing our Monthly security update checks. Veeam is not the only company that has issues releasing information like this.

[Gostev]

All, thank you for you patience. Here are the KB articles:

Veeam Backup & Replication > KB3103

Veeam Agent for Microsoft Windows > KB3108

Veeam Agent for Linux > KB3109

Please let us know if these address regulatory requirements you're facing.

[controlfreak]

Thanks Gostev, these are excellent and meet our needs completely!

Thank you again for putting this together for what is likely a pretty small group of customers overall. We greatly appreaciate the effort on your part.

Thanks,

Dan

[Coldfirex]

When will these links be updated for the new releases?

[Gostev]

Yes, always.

[Coldfirex]

Thanks. When should we expect those to be updated?

[Gostev]

Soon!

[maartene]

Veeam Backup & Replication > KB3103
lease let us know if these address regulatory requirements you're facing.

I think it would be beneficial if there would be links to what the fixes/improvements actually are ("improved" sounds vague).

maarten

[Gostev]

I was told that for these changes, it is not possible to explain their details without fully documenting the previous and the new architecture, which is something we cannot do publicly. Not just for know-how reasons, but also because it will give hackers a great documentation and ideas on how to attack Veeam customers who did not upgrade to the latest version yet.

On the other hand, whenever changes are simple and contained - they will be explained specifically, like a number of other bullets already are.