New build physical VBR server and O/S

[Frosty]

Hi all,
I'm just about to start a build of a new physical server for our VBR 9.5 and have a question about operating system.
In the past we have been running Windows Server 2008 R2 and it has been rock solid.
I'm now weighing up whether to do the new build on Server 2012 R2 or on Server 2016.
I'm nervous about the patching for Server 2016 as I want everything to be stable (don't want large, disruptive updates such as "Spring" and "Autumn" like you get with Windows 10 environments).
Q. am I right to be nervous about that, or should I plunge in?
(I have loads of experience with Server 2012 R2 but absolutely zero experience with Server 2016).
Regards to all,
Steve

[Pat490]

2012R2 has extended support until 2023/10/10, so much time left and my preferred server os is still 2012r2!

[DGrinev]

Hi Stephen,

Please can you elaborate how many backup components will reside on the physical server and which exactly?
You might notice the major consideration on the forum for Server 2016 in terms of backup infrastructure is to use it as the Repository server with ReFS. Thanks!

[nitramd]

I have a VBR 2016 server and it runs well. It is, however, virtual.

I have a few 2016 server proxies and have had issues installing the large updates; resorted to installing these manually which is a pain and is time consuming.

Is there a chance that you could virtualize your VBR server? If not, I would suggest using 2012 R2 server. Will you need to purchase an extended support contract? I would also recommend using VAW 2.0 for your physical server - it makes life easier.

Good luck.

[Frosty]

Thanks all for the feedback so far.
In answer to the questions asked:

* initial plan was to place all backup components on the one physical server, however I am prepared to deploy some proxies as VMs if that becomes necessary for performance reasons (we only have about 50 VMs to back up, so pretty small environment). We currently run VBR 9.5 on a VM with 6 x vCPUs and all backup components on that single VM; it is performing fine.

* we will definitely be using this (new) physical server for VBR for the foreseeable future. It is a Dell R740XD with a single 12-core CPU, 32GB RAM and 8 x 10TB nearline SAS HDDs (configured as a 40TB RAID10 at present). I haven't installed VBR on it yet, so I haven't done any performance testing of that configuration.

* because this is a new server, we have 5-year onsite hardware support pre-paid, so that will take us up until end December 2022. We haven't considered operating system support yet. But there's a fair chance that I will mandate Server 2016 for all new server builds shortly (once I get some experience with it and understand the pro's and con's).

[nezzer]

Hey Steve!

Are you planning on running this physical machine as a backup server or as a proxy/repo?

I'm running a full 2016 Enviroment and it works really great.
If you want to go down the Storagespaces path you do NOT want 2k12.

1 Virtual Veeam Backup server
2 Physical primary repo and proxy w. storagespaces and paralell rebuild acitvated. 2 w Mirror.
1 Physical Secondary repo w. storagespaces and paralell rebuild acitvated. 2 w Mirror.

Using only 10GbE No hotadd needed except for a few machines with single disks larger than 1 TB.


I would recommend going full SD storage as you will get alot more thoughput and incredible latency compared to hardware raid.

Good luck with your our new setup! (:

[Frosty]

Hi nezzer,
Am intending this physical server to be a one-stop-shop for all things backup. It will run VBR but also perform some other useful functions for us. Am definitely leaning now to Server 2016 instead of 2012 R2. It is going to be a very simple setup though, nothing fancy. As it is physical, yes, network backup instead of Hot Add (which we used with our previous VM of VBR). We have 10GbE as well, so it should be very quick to move the data. It'll be the disk storage which will be slowish, but as our environment is small it isn't a big concern.

[chris.childerhose]

Our setup I have deployed is a physical VBR server which has the LTO-6 autoloader attached for tape backup. We have 1 proxy VM per host separated by affinity rule. The proxies have 8 vCPU and 16GB. We run 10GB network but have the Enterprise Plus for Nimble Integration. The server has two 8 core CPUs and 48GB of RAM. I am using Windows 2016 for ReFS integration which is great for synthetic fulls. Also use Veeam Agent 2.0 for server backup.

[AlexB82]

I'd investigate using Server 2016 which has support for ReFS. There's more on the Veeam blog here: https://www.veeam.com/blog/advanced-ref ... suite.html After chatting with some of the guys at Veeam, it seems like the right way to go.

[Frosty]

So I think I'm just about ready to make a call on this, and I don't think I'm prepared to go with Server 2016 at this point.

My main objection is the brain-dead way Microsoft have chosen to do patching with the Server 2016 (Windows 10) approach. Biggest problem is that you can no longer review the list of available patches before choosing which ones to install. Second issue is the 12-hour reboot window thing. Just unacceptable.

So I think I am going to have to build everything on Server 2012 R2. Disappointed as hell with Microsoft. Numbskulls.

[DGrinev]

Hi Stephen,

Your objections look quite reasonable, so feel free to deploy infrastructure based on 2012 R2. Thanks!

[lando_uk]

So I think I'm just about ready to make a call on this, and I don't think I'm prepared to go with Server 2016 at this point.

My main objection is the brain-dead way Microsoft have chosen to do patching with the Server 2016 (Windows 10) approach. Biggest problem is that you can no longer review the list of available patches before choosing which ones to install. Second issue is the 12-hour reboot window thing. Just unacceptable.

So I think I am going to have to build everything on Server 2012 R2. Disappointed as hell with Microsoft. Numbskulls.

I'm in the same boat, we have some funds to replace our old Veeam servers.

For patching, nowadays 2016 works well with WSUS. 2008R2 and 2012R2 have the same rollups as 2016, so you cant choose updates on those, just what rolllup (April/May etc) Server 2016 LTSC will never update to the latest version like Windows 10 does, its stuck on v1607 until 2027 now.

The lifespan of your server is 7 years, if you choose 2012R2 today, you're going to have to upgrade at some point before you throw the server away.

[Frosty]

The problem at the moment is that one cannot pick and choose Microsoft updates. If there is a compatibility question between various applications, you get a problem.
e.g. .NET Framework versions are often released, but they are either compatible or NOT with other applications such as Exchange, SharePoint, Skype For Business and so on.
This might or might not be a problem with this particular server and software combination, but I'm planning 5 years ahead and I don't want to have some kit on 2016 and some on 2012 R2 just because of patching issues. Leaves me with little option but to go with 2012 R2 which still gives me control over which patches an individual server will install. Yes, I could put individual servers into individual groups in WSUS and try to work around it that way, but that's just messy as hell and my OCD won't let me do it. ;^)

[lando_uk]

There's no difference with 2012R2 and 2016 when both are in WSUS, We have difference OS's in the same WSUS group, when allowing updates you just sort by date and tick all updates to April18 for all OS's, then all your OS's are patched to the same level. Of course they will have their own specific OS related issues, but the Rollup dates can be the same across all your servers.
Until recently I would have chosen 2012R2 for new Veeam servers too, but they seem to have sorted out ReFS now and from the misery of patching 2008R2 recently, even though microsoft say will will support an OS to a certain date, the older they get the worse the update quality testing seem to be.

[Frosty]

I don't agree. The difference is in the user experience in Server 2016 vs Server 2012 R2. Our practice until now:
1. approve new patches via WSUS to our servers (2008 R2 and 2012 R2)
2. once a month, logon to servers (choosing less critical ones first, later more critical ones) and review outstanding patches, choose which ones to install
-- e.g. when a .NET Framework update is available, we would allow this to install initially on a Development app server, then test for compatibility, then later allow install to Production server
-- e.g. when a .NET Framework update is available, for an Exchange/SharePoint/Skype server, we review compatibility, then if service packs or rollups are required for Exchange/SharePoint/Skype we do those first, and only later come back to allow .NET Framework to be installed

I am trying to avoid the problem of all-or-nothing which is what Microsoft seem to have forced me to adopt. If you know a way around that, I'm very open to the idea. But what I don't want to do is have dozens of different server groups in WSUS and have to decide at that level which/when to Approve on a per-server basis. That would be too messy.

[nezzer]

Hi nezzer,
Am intending this physical server to be a one-stop-shop for all things backup. It will run VBR but also perform some other useful functions for us. Am definitely leaning now to Server 2016 instead of 2012 R2. It is going to be a very simple setup though, nothing fancy. As it is physical, yes, network backup instead of Hot Add (which we used with our previous VM of VBR). We have 10GbE as well, so it should be very quick to move the data. It'll be the disk storage which will be slowish, but as our environment is small it isn't a big concern.

Hey!

I would recommend running the backup server Virtual because of the following reasons:

1TB virtual disks is NOT supported in NBD mode and will eventually fail. You should deploy atleast one Hotadd proxy for those VM's.
If VBR server (the backup/ schduler etc) crashes you will have quite a few hours to get it up and running again, much easier.

Regarding 10GBE and NBD - you will get far from those 10GBit becasue of the nature of NBD - consider attaching another 10GBE and Team them with windows Teaming. We will do that when we get some more holes to put fibers in.

Just my 2 cents (;

[lando_uk]

I've been using NBD for years and we have several VMs with over 1TB VD and they are fine. Where did you read it's not supported?

[Frosty]

Yes, I understood that backing up via network was now the recommended method.
Our new environment has 100% SSD for the storage, so I don't want to "waste" that on bulk storage for backups.

[nezzer]

I've been using NBD for years and we have several VMs with over 1TB VD and they are fine. Where did you read it's not supported?

Hey lando!

As you say it "should" be fine ,I've also done quite a few backups of vDisks over 1 TB however the speed is not so good and vmware documentation does not recommend it.
Please tell me if I've got this wrong. https://pubs.vmware.com/vsphere-50/inde ... t.5.5.html

There is a great improvement running hotadd for these large disks.

[Frosty]

Am wondering whether that doesn't apply to me, as it is a vSphere 5 reference?
I'd looked at Veeam's recommendations:
https://bp.veeam.expert/resource_planni ... modes.html
https://bp.veeam.expert/resource_planni ... _mode.html

[oliverL]

Thanks all for the feedback so far.
In answer to the questions asked:

* initial plan was to place all backup components on the one physical server, however I am prepared to deploy some proxies as VMs if that becomes necessary for performance reasons (we only have about 50 VMs to back up, so pretty small environment).

Proxys ain't bad. If you aren't using Proxys which can use hotadd mode you need to use NBDSSL.
In NBD/NBDSSL Mode you have a hardcoded Limit of NFC-Tickets based on the Host which might give you Backup Problems (if the NFC-Ticket Queue is running full and Veeam won't stop the data flow).

Also - since vSphere 6.0 NBD is enforcing SSL which results in slower Speed by design:
https://kb.vmware.com/s/article/2147768

[nezzer]

Am wondering whether that doesn't apply to me, as it is a vSphere 5 reference?
I'd looked at Veeam's recommendations:
https://bp.veeam.expert/resource_planni ... modes.html
https://bp.veeam.expert/resource_planni ... _mode.html

I can tell you that I'm not sure about if this applies to 6.X aswell.

The Veeam guides does not specify anything about VD sizes from what I can see?
What is certain is that reading single mega drives over NDBSSL (which is enforced in 6.X) is super slow even in a all flash vSAN.
I'm getting numbers of around 40 - 80 MB/s.

[lando_uk]

I still have both 5 and 6 hosts.

In ESXI 5.x, Single VM backup and restores using NBD will not go much over 80MB's but when running in parallel like you would in normal production, 10Gbe management nics will go 300MB's +

In v6.x I see slower backups, typically 200MB's

[Frosty]

Ahh, well then I may need to run some proxies.
How would you normally approach this?
e.g. one per physical host?
(I've never had to bother before, as we've always used a VBR VM with HotAdd ability)

[Frosty]

Would appreciate any tips anyone can offer, to help me extract a little more performance out of our new environment.

I was getting pretty decent throughput (so I thought) even without a Proxy in place --> 200MB/sec

I've now added a Windows Server 2012 VM as a Veeam Proxy so we can take advantage of HotAdd --> 4 vCPUs, 4GB RAM, limited to max 3 jobs at any one time.

Configured up a new backup job and am now getting --> 500MB/sec throughput.

The new backup job shows that the performance limitation is the Proxy. I checked the VM Monitor tab for the Proxy and it shows two main things indicating the limits:
--> Network throughput peaked/flatlined at 500MBps
--> CPU usage peaked/flatlined at 100%

We're running everything on 10GbE so there would appear to be some room for improvement in network throughput. But I am guessing that the Proxy CPU is the actual limitation. Does that sound right?

The backup job only had 2 disks using [hotadd] so I am a little surprised that the CPU hit 100% on a 4 vCPU VM. Should I add another 2-4 vCPUs perhaps?

[Frosty]

Mmmmm ... I left all the Backup Jobs to Auto-select proxy.
The jobs overall seemed to take a fair bit longer last night.
My suspicion is that for most of my smaller VMs, I should just stick with NBD and the default Veeam proxy.
But for the big VMs with one or two large .VMDK it would be best to go with my new Proxy.
Does that sound about right?