Meltdown - How to patch your system?

Availability for the Always-On Enterprise

Meltdown - How to patch your system?

Veeam Logoby frankive » Tue Jan 09, 2018 9:25 pm

Not Veeam related but it would be interesting to see how other IT-professionals is dealing with the meltdown and spectre flaws which has been going viral the last 2 weeks.
I have read quite a few interesting article the last days, but most of them are focusing on the antivirus vendor and their compatibility before we can patch the Windows Computers.
I am also reading that we need to patch the hardware; is a bios update enough or is it other firmware which also needs updating here?
I have not had time to read Gostevs newsletter but will do it later this evening; I just had a quick look and saw it was related to this topic so I am hoping for interesting reading (as always!).

Anyway; if other IT-profesisonals have some tips to deal with this for now, it would be interesting to hear how you deal with this.

Thanks.
frankive
Service Provider
 
Posts: 773
Liked: 93 times
Joined: Tue May 14, 2013 8:35 pm
Location: Norway
Full Name: Frank Iversen

[MERGED] Meltdown / Spectre Patches for VEEAM Linux applianc

Veeam Logoby edirschedl » Wed Jan 10, 2018 1:42 pm

Hi,

any plans from VEEAM regarding a Meltdown / Spectre OS patch for the deployed Linux appliances, such as Linux Helper appliance for Linux FLR?

Are there any other VEEAM components affected from this issue and will be pachted?

Thank you,
Emanuel
edirschedl
Influencer
 
Posts: 23
Liked: 2 times
Joined: Thu Jul 21, 2016 12:29 pm

Re: Meltdown - How to patch your system?

Veeam Logoby nitramd » Wed Jan 10, 2018 3:08 pm

Hi Frank.

Microsoft has mandated that AntiVirus vendors set a reg key in their software; this will show which vendors have updated their software and those who have not; apparently, current AV engines won't stop Meltdown or Spectre. If the reg key has not been set and you're running AV software other than Microsoft's, you will not receive January updates or subsequent updates.

I've been reading that firmware/microcode will be updated by hardware manufacturers and, therefore, should be installed.

A strategy we're employing is to install patches on a few servers and see what happens - making a snapshot first, of course. Then if all goes well continue to roll out patches.

The overarching theme is to patch now and continue patching, which I presume means keep patching until the current afflicted hardware is replaced with CPUs that are not susceptible to these two flaws. This brings up a number of questions in my mind, however.

Anyway, if you would like to review a brief guide on how to protect your machines follow this link: https://thehackernews.com/2018/01/meltd ... tches.html

Hope this helps.
nitramd
Enthusiast
 
Posts: 67
Liked: 9 times
Joined: Thu Feb 16, 2017 8:05 pm

[MERGED] Meltdown / Spectre Patches for VEEAM Linux applianc

Veeam Logoby PTide » Wed Jan 10, 2018 6:11 pm

edirschedl wrote:any plans from VEEAM regarding a Meltdown / Spectre OS patch for the deployed Linux appliances, such as Linux Helper appliance for Linux FLR?

Are there any other VEEAM components affected from this issue and will be pachted?

Hi,

Only root can login into appliance as there are no other users. Once you've logged in as root you don't need to exploit anything. To protect neighbour VMs from getting into each other's memory it is sufficient to patch the host.

All other Veeam components are installed on the machines provided by user, therefore it depends on system administrator whether or not those machines are vulnerable.

Additional info on the subject can be found here.

Thanks
PTide
Veeam Software
 
Posts: 3409
Liked: 283 times
Joined: Tue May 19, 2015 1:46 pm

Re: Meltdown - How to patch your system?

Veeam Logoby ChuckS42 » Wed Jan 10, 2018 6:56 pm

We're actually turning this effort into an actual Project to plan remediation. It's a many-headed beast, not just fixing (actually working around) the vulnerabilities, but avoiding the (inevitable?) performance hit after patches are applied. Physical processor family counts, as does the version of vSphere and whichever EVC mode you've selected for your clusters. If you need to raise the EVC mode (or disable it entirely) one must power off VMs and update VM hardware levels for it to take effect. Not pretty!
ChuckS42
Enthusiast
 
Posts: 72
Liked: 5 times
Joined: Wed Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens

Re: Meltdown - How to patch your system?

Veeam Logoby DaveWatkins » Thu Jan 11, 2018 1:45 am

The BIOS update patches one of the vulnerabilities of Spectre, it does nothing for Meltdown. Technically it only delivers an updated microcode for the CPU which could actually be delivered by the OS on every boot and work (VMWare is doing just that with it's patches for ESXi).

The windows key is simply to tell MS that the AV vendors have tested the patches and it doesn't cause any issues. Basically the patches make changes to some fairly low level kernel code that AV vendors are notorious for using undocumented features. That will result in blue screens and so MS have taken this extra step so patches aren't applied that will cause blue screens when the AV software tries to do something stupid.

So, if you're running VMWare for example, you need the ESXi patches to stop VM's being able to access each others memory. The first round of ESX patches stop Meltdown from doing that. The second lot of patches released yesterday present up to the VM's the same flags as a BIOS updated physical host (ie they tell the VM that the BIOS is running the new microcode to block one Spectre vulnerability).

Once you have those installed you can then look to actually patch Spectre and Meltdown to stop processes reading the local machine/VM's kernel memory.

At least thats my current understanding :)
DaveWatkins
Expert
 
Posts: 285
Liked: 71 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: Meltdown - How to patch your system?

Veeam Logoby cdownum » Fri Jan 12, 2018 5:53 pm

Has anyone noticed a degradation of VEEAM performance in medium-large environments after patching for Spectre/Meltdown?
cdownum
Novice
 
Posts: 3
Liked: never
Joined: Tue Mar 18, 2014 9:28 pm
Full Name: Chad Downum

Re: Meltdown - How to patch your system?

Veeam Logoby opg70 » Sun Jan 14, 2018 10:28 pm

This is exactly my concern, especially with the reports of very high cpu usage on some servers. Particularly as the load increases with network and storage accesses - the main type of work of Veeam servers. Hopefully the fact that Veeam works with larger blocks will help minimize this. The fact that older cpu's are affected more than newer ones will also be bad for most users as Veeam servers were often older machines repurposed for backup duties.
opg70
Influencer
 
Posts: 15
Liked: 1 time
Joined: Sun Oct 06, 2013 8:48 am

Re: Meltdown - How to patch your system?

Veeam Logoby DVTNZ » Sun Jan 14, 2018 11:28 pm

I have applied the all the MS and the Dell BIOS update to a customer's R740xd host with (Dual Xeon Silver 4110 CPUs) VBR host and compared the backups before and after their seems to be an overall 14% slowdown.
DVTNZ
Lurker
 
Posts: 1
Liked: never
Joined: Tue Jan 09, 2018 3:26 am
Location: South Island - New Zealand

Re: Meltdown - How to patch your system?

Veeam Logoby ITP-Stan » Mon Jan 15, 2018 8:58 am

VMWare have pulled the patches for now. They may cause instability issues, anyone experience this so far?

I'll guess we'll have to wait for new patches.
ITP-Stan
Service Provider
 
Posts: 77
Liked: 8 times
Joined: Mon Feb 18, 2013 10:45 am
Full Name: Stan (IF-IT4U)

Re: Meltdown - How to patch your system?

Veeam Logoby InFrance » Mon Jan 15, 2018 9:47 am

Hi,

My VBR serveur is a physical HP Prioliant D380 G7 running windows server 2012
I understand from HP that there might not be any BIOS updates for this server
As yet, there is also no patch for windows 2012

Other than admins, there are no other application or users on this server

In order to fully protect my VBS server from these vulnerabilities, is my only option to migrate my VBR server to a new server (with the BIOS update) running Windows 2016?

I would very much appreciate your comments.

Regards
InFrance
Influencer
 
Posts: 15
Liked: never
Joined: Thu Jun 07, 2012 11:27 am

Re: Meltdown - How to patch your system?

Veeam Logoby Pat490 » Mon Jan 15, 2018 10:35 am

afaik you do not have to go to Server 2016.
there should be patches for 2012r2 already or will be soon.

at the moment all news changes daily. I will wait a few more days before patching!
Pat490
Expert
 
Posts: 146
Liked: 24 times
Joined: Tue Apr 28, 2015 7:18 am
Location: Germany
Full Name: Patrick

Re: Meltdown - How to patch your system?

Veeam Logoby InFrance » Mon Jan 15, 2018 10:37 am

Thanks for your reply

Unfortunately, I am running Windows 2012 and not Windows 2012r2
InFrance
Influencer
 
Posts: 15
Liked: never
Joined: Thu Jun 07, 2012 11:27 am

Re: Meltdown - How to patch your system?

Veeam Logoby cbc-tgschultz » Mon Jan 15, 2018 2:54 pm 1 person likes this post

Honestly I'm not sure it is worth patching against for many use cases. The vulnerability requires that you run untrusted code on your machine, so the primary vectors will be web browsing and VMs you don't control. For providers this is a pretty big deal, and for desktops... well it doesn't really change the risk factors of web browsing all that much in my opinion. If your infrastructure is entirely under your control and you don't do a lot of browsing from your servers, the cost/benefit of patching against is not very appealing, especially if any of your workloads happen to fall into the most affected kinds. At the very least, I think it is worth waiting for everyone else to shake out all the flaws and more performance information to become available.
cbc-tgschultz
Enthusiast
 
Posts: 48
Liked: 10 times
Joined: Fri May 13, 2016 1:48 pm
Full Name: Tanner Schultz

Re: Meltdown - How to patch your system?

Veeam Logoby jonhutton434 » Mon Jan 15, 2018 2:59 pm

If you have any HPE hardware this link is worth a visit

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00039267en_us
jonhutton434
Lurker
 
Posts: 1
Liked: never
Joined: Tue Apr 18, 2017 10:22 am
Full Name: Jon


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google [Bot], KevinJ and 1 guest