Meltdown - How to patch your system?

[frankive]

Not Veeam related but it would be interesting to see how other IT-professionals is dealing with the meltdown and spectre flaws which has been going viral the last 2 weeks.
I have read quite a few interesting article the last days, but most of them are focusing on the antivirus vendor and their compatibility before we can patch the Windows Computers.
I am also reading that we need to patch the hardware; is a bios update enough or is it other firmware which also needs updating here?
I have not had time to read Gostevs newsletter but will do it later this evening; I just had a quick look and saw it was related to this topic so I am hoping for interesting reading (as always!).

Anyway; if other IT-profesisonals have some tips to deal with this for now, it would be interesting to hear how you deal with this.

Thanks.

[edirschedl]

Hi,

any plans from VEEAM regarding a Meltdown / Spectre OS patch for the deployed Linux appliances, such as Linux Helper appliance for Linux FLR?

Are there any other VEEAM components affected from this issue and will be pachted?

Thank you,
Emanuel

[nitramd]

Hi Frank.

Microsoft has mandated that AntiVirus vendors set a reg key in their software; this will show which vendors have updated their software and those who have not; apparently, current AV engines won't stop Meltdown or Spectre. If the reg key has not been set and you're running AV software other than Microsoft's, you will not receive January updates or subsequent updates.

I've been reading that firmware/microcode will be updated by hardware manufacturers and, therefore, should be installed.

A strategy we're employing is to install patches on a few servers and see what happens - making a snapshot first, of course. Then if all goes well continue to roll out patches.

The overarching theme is to patch now and continue patching, which I presume means keep patching until the current afflicted hardware is replaced with CPUs that are not susceptible to these two flaws. This brings up a number of questions in my mind, however.

Anyway, if you would like to review a brief guide on how to protect your machines follow this link: https://thehackernews.com/2018/01/meltd ... tches.html

Hope this helps.

[PTide]

any plans from VEEAM regarding a Meltdown / Spectre OS patch for the deployed Linux appliances, such as Linux Helper appliance for Linux FLR?

Are there any other VEEAM components affected from this issue and will be pachted?

Hi,

Only root can login into appliance as there are no other users. Once you've logged in as root you don't need to exploit anything. To protect neighbour VMs from getting into each other's memory it is sufficient to patch the host.

All other Veeam components are installed on the machines provided by user, therefore it depends on system administrator whether or not those machines are vulnerable.

Additional info on the subject can be found here.

Thanks

[ChuckS42]

We're actually turning this effort into an actual Project to plan remediation. It's a many-headed beast, not just fixing (actually working around) the vulnerabilities, but avoiding the (inevitable?) performance hit after patches are applied. Physical processor family counts, as does the version of vSphere and whichever EVC mode you've selected for your clusters. If you need to raise the EVC mode (or disable it entirely) one must power off VMs and update VM hardware levels for it to take effect. Not pretty!

[DaveWatkins]

The BIOS update patches one of the vulnerabilities of Spectre, it does nothing for Meltdown. Technically it only delivers an updated microcode for the CPU which could actually be delivered by the OS on every boot and work (VMWare is doing just that with it's patches for ESXi).

The windows key is simply to tell MS that the AV vendors have tested the patches and it doesn't cause any issues. Basically the patches make changes to some fairly low level kernel code that AV vendors are notorious for using undocumented features. That will result in blue screens and so MS have taken this extra step so patches aren't applied that will cause blue screens when the AV software tries to do something stupid.

So, if you're running VMWare for example, you need the ESXi patches to stop VM's being able to access each others memory. The first round of ESX patches stop Meltdown from doing that. The second lot of patches released yesterday present up to the VM's the same flags as a BIOS updated physical host (ie they tell the VM that the BIOS is running the new microcode to block one Spectre vulnerability).

Once you have those installed you can then look to actually patch Spectre and Meltdown to stop processes reading the local machine/VM's kernel memory.

At least thats my current understanding

[cdownum]

Has anyone noticed a degradation of VEEAM performance in medium-large environments after patching for Spectre/Meltdown?

[opg70]

This is exactly my concern, especially with the reports of very high cpu usage on some servers. Particularly as the load increases with network and storage accesses - the main type of work of Veeam servers. Hopefully the fact that Veeam works with larger blocks will help minimize this. The fact that older cpu's are affected more than newer ones will also be bad for most users as Veeam servers were often older machines repurposed for backup duties.

[DVTNZ]

I have applied the all the MS and the Dell BIOS update to a customer's R740xd host with (Dual Xeon Silver 4110 CPUs) VBR host and compared the backups before and after their seems to be an overall 14% slowdown.

[ITP-Stan]

VMWare have pulled the patches for now. They may cause instability issues, anyone experience this so far?

I'll guess we'll have to wait for new patches.

[InFrance]

Hi,

My VBR serveur is a physical HP Prioliant D380 G7 running windows server 2012
I understand from HP that there might not be any BIOS updates for this server
As yet, there is also no patch for windows 2012

Other than admins, there are no other application or users on this server

In order to fully protect my VBS server from these vulnerabilities, is my only option to migrate my VBR server to a new server (with the BIOS update) running Windows 2016?

I would very much appreciate your comments.

Regards

[Pat490]

afaik you do not have to go to Server 2016.
there should be patches for 2012r2 already or will be soon.

at the moment all news changes daily. I will wait a few more days before patching!

[InFrance]

Thanks for your reply

Unfortunately, I am running Windows 2012 and not Windows 2012r2

[cbc-tgschultz]

Honestly I'm not sure it is worth patching against for many use cases. The vulnerability requires that you run untrusted code on your machine, so the primary vectors will be web browsing and VMs you don't control. For providers this is a pretty big deal, and for desktops... well it doesn't really change the risk factors of web browsing all that much in my opinion. If your infrastructure is entirely under your control and you don't do a lot of browsing from your servers, the cost/benefit of patching against is not very appealing, especially if any of your workloads happen to fall into the most affected kinds. At the very least, I think it is worth waiting for everyone else to shake out all the flaws and more performance information to become available.

[jonhutton434]

If you have any HPE hardware this link is worth a visit

https://support.hpe.com/hpsc/doc/public ... 39267en_us