Comprehensive data protection for all workloads
Post Reply
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Meltdown - How to patch your system?

Post by frankive »

Not Veeam related but it would be interesting to see how other IT-professionals is dealing with the meltdown and spectre flaws which has been going viral the last 2 weeks.
I have read quite a few interesting article the last days, but most of them are focusing on the antivirus vendor and their compatibility before we can patch the Windows Computers.
I am also reading that we need to patch the hardware; is a bios update enough or is it other firmware which also needs updating here?
I have not had time to read Gostevs newsletter but will do it later this evening; I just had a quick look and saw it was related to this topic so I am hoping for interesting reading (as always!).

Anyway; if other IT-profesisonals have some tips to deal with this for now, it would be interesting to hear how you deal with this.

Thanks.
edirschedl
Enthusiast
Posts: 44
Liked: 4 times
Joined: Jul 21, 2016 12:29 pm
Full Name: Emanuel Dirschedl
Contact:

[MERGED] Meltdown / Spectre Patches for VEEAM Linux applianc

Post by edirschedl »

Hi,

any plans from VEEAM regarding a Meltdown / Spectre OS patch for the deployed Linux appliances, such as Linux Helper appliance for Linux FLR?

Are there any other VEEAM components affected from this issue and will be pachted?

Thank you,
Emanuel
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Meltdown - How to patch your system?

Post by nitramd »

Hi Frank.

Microsoft has mandated that AntiVirus vendors set a reg key in their software; this will show which vendors have updated their software and those who have not; apparently, current AV engines won't stop Meltdown or Spectre. If the reg key has not been set and you're running AV software other than Microsoft's, you will not receive January updates or subsequent updates.

I've been reading that firmware/microcode will be updated by hardware manufacturers and, therefore, should be installed.

A strategy we're employing is to install patches on a few servers and see what happens - making a snapshot first, of course. Then if all goes well continue to roll out patches.

The overarching theme is to patch now and continue patching, which I presume means keep patching until the current afflicted hardware is replaced with CPUs that are not susceptible to these two flaws. This brings up a number of questions in my mind, however.

Anyway, if you would like to review a brief guide on how to protect your machines follow this link: https://thehackernews.com/2018/01/meltd ... tches.html

Hope this helps.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

[MERGED] Meltdown / Spectre Patches for VEEAM Linux applianc

Post by PTide »

edirschedl wrote:any plans from VEEAM regarding a Meltdown / Spectre OS patch for the deployed Linux appliances, such as Linux Helper appliance for Linux FLR?

Are there any other VEEAM components affected from this issue and will be pachted?
Hi,

Only root can login into appliance as there are no other users. Once you've logged in as root you don't need to exploit anything. To protect neighbour VMs from getting into each other's memory it is sufficient to patch the host.

All other Veeam components are installed on the machines provided by user, therefore it depends on system administrator whether or not those machines are vulnerable.

Additional info on the subject can be found here.

Thanks
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: Meltdown - How to patch your system?

Post by ChuckS42 »

We're actually turning this effort into an actual Project to plan remediation. It's a many-headed beast, not just fixing (actually working around) the vulnerabilities, but avoiding the (inevitable?) performance hit after patches are applied. Physical processor family counts, as does the version of vSphere and whichever EVC mode you've selected for your clusters. If you need to raise the EVC mode (or disable it entirely) one must power off VMs and update VM hardware levels for it to take effect. Not pretty!
Veeaming since 2013
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: Meltdown - How to patch your system?

Post by DaveWatkins »

The BIOS update patches one of the vulnerabilities of Spectre, it does nothing for Meltdown. Technically it only delivers an updated microcode for the CPU which could actually be delivered by the OS on every boot and work (VMWare is doing just that with it's patches for ESXi).

The windows key is simply to tell MS that the AV vendors have tested the patches and it doesn't cause any issues. Basically the patches make changes to some fairly low level kernel code that AV vendors are notorious for using undocumented features. That will result in blue screens and so MS have taken this extra step so patches aren't applied that will cause blue screens when the AV software tries to do something stupid.

So, if you're running VMWare for example, you need the ESXi patches to stop VM's being able to access each others memory. The first round of ESX patches stop Meltdown from doing that. The second lot of patches released yesterday present up to the VM's the same flags as a BIOS updated physical host (ie they tell the VM that the BIOS is running the new microcode to block one Spectre vulnerability).

Once you have those installed you can then look to actually patch Spectre and Meltdown to stop processes reading the local machine/VM's kernel memory.

At least thats my current understanding :)
cdownum
Novice
Posts: 3
Liked: never
Joined: Mar 18, 2014 9:28 pm
Full Name: Chad Downum
Contact:

Re: Meltdown - How to patch your system?

Post by cdownum »

Has anyone noticed a degradation of VEEAM performance in medium-large environments after patching for Spectre/Meltdown?
opg70
Influencer
Posts: 24
Liked: 3 times
Joined: Oct 06, 2013 8:48 am
Contact:

Re: Meltdown - How to patch your system?

Post by opg70 »

This is exactly my concern, especially with the reports of very high cpu usage on some servers. Particularly as the load increases with network and storage accesses - the main type of work of Veeam servers. Hopefully the fact that Veeam works with larger blocks will help minimize this. The fact that older cpu's are affected more than newer ones will also be bad for most users as Veeam servers were often older machines repurposed for backup duties.
DVTNZ
Service Provider
Posts: 1
Liked: never
Joined: Jan 09, 2018 3:26 am
Location: South Island - New Zealand
Contact:

Re: Meltdown - How to patch your system?

Post by DVTNZ »

I have applied the all the MS and the Dell BIOS update to a customer's R740xd host with (Dual Xeon Silver 4110 CPUs) VBR host and compared the backups before and after their seems to be an overall 14% slowdown.
ITP-Stan
Service Provider
Posts: 201
Liked: 55 times
Joined: Feb 18, 2013 10:45 am
Full Name: Stan (IF-IT4U)
Contact:

Re: Meltdown - How to patch your system?

Post by ITP-Stan »

VMWare have pulled the patches for now. They may cause instability issues, anyone experience this so far?

I'll guess we'll have to wait for new patches.
InFrance
Influencer
Posts: 17
Liked: never
Joined: Jun 07, 2012 11:27 am
Contact:

Re: Meltdown - How to patch your system?

Post by InFrance »

Hi,

My VBR serveur is a physical HP Prioliant D380 G7 running windows server 2012
I understand from HP that there might not be any BIOS updates for this server
As yet, there is also no patch for windows 2012

Other than admins, there are no other application or users on this server

In order to fully protect my VBS server from these vulnerabilities, is my only option to migrate my VBR server to a new server (with the BIOS update) running Windows 2016?

I would very much appreciate your comments.

Regards
Pat490
Expert
Posts: 170
Liked: 29 times
Joined: Apr 28, 2015 7:18 am
Full Name: Patrick
Location: Germany
Contact:

Re: Meltdown - How to patch your system?

Post by Pat490 »

afaik you do not have to go to Server 2016.
there should be patches for 2012r2 already or will be soon.

at the moment all news changes daily. I will wait a few more days before patching!
InFrance
Influencer
Posts: 17
Liked: never
Joined: Jun 07, 2012 11:27 am
Contact:

Re: Meltdown - How to patch your system?

Post by InFrance »

Thanks for your reply

Unfortunately, I am running Windows 2012 and not Windows 2012r2
cbc-tgschultz
Enthusiast
Posts: 65
Liked: 11 times
Joined: May 13, 2016 1:48 pm
Full Name: Tanner Schultz
Contact:

Re: Meltdown - How to patch your system?

Post by cbc-tgschultz » 1 person likes this post

Honestly I'm not sure it is worth patching against for many use cases. The vulnerability requires that you run untrusted code on your machine, so the primary vectors will be web browsing and VMs you don't control. For providers this is a pretty big deal, and for desktops... well it doesn't really change the risk factors of web browsing all that much in my opinion. If your infrastructure is entirely under your control and you don't do a lot of browsing from your servers, the cost/benefit of patching against is not very appealing, especially if any of your workloads happen to fall into the most affected kinds. At the very least, I think it is worth waiting for everyone else to shake out all the flaws and more performance information to become available.
jonhutton434
Lurker
Posts: 1
Liked: never
Joined: Apr 18, 2017 10:22 am
Full Name: Jon
Contact:

Re: Meltdown - How to patch your system?

Post by jonhutton434 »

If you have any HPE hardware this link is worth a visit

https://support.hpe.com/hpsc/doc/public ... 39267en_us
Post Reply

Who is online

Users browsing this forum: No registered users and 161 guests