Done a hardened repo - opinions needed on backup server and overall security

[cosmik]

I'm trying to figure out where to "place" the veeam backup server. My entire server infrastructure is virtualized, so initially I thought that I could perhaps have everything veeam in VR format except definitely for the repo itself, whereas a Linux hardened one with immutable backups is elected.

Continuing my barrage of VBR-related questions, there go some more:

1) First, I'm wondering if having an immutable design would help somehow against advanced persistence threat actors (ATPs): I can have immutability for say 10 days, but if a malevolent actor obtains access to the backup server, he could simply minimize the immutabily interval to 1 day and then wait out to strike. Is that a reasonable train of thought? Against of course we are covered by the fact (IIRC) that at least GFS backups are always immutable.

2) Regardless of (1) above, the backup server seems like an extremely important node, security-wise: it contains information for the ESXi infrastructure, might contain account data to have application-aware snapshots taken (administrator-class accounts presumably), etc. Is having the backup server itself as a VM a weak link?

3) Building upon my thoughts regarding (2), where does one place the backup server? If I had a lot of hardware running around, I'd probably install this also on a dedicated physical host, leaving a backup proxy running on the ESXi hosts. Unfortunately I do not have any Windows Server licenses free (but please check off-topic question below on the subject). I could find some 10th gen intel-based desktops for the role. Considering these limitations, how would you go about it, considering the need to keep things secure?

<Off-topic>We have had three 16-core servers, two of them in a ESXi cluster. When deciding for the number of Windows Server 2019 Standard licenses to buy I believe we were informed that since this would be a virtualized platform, we'd have to purchase twice the number of 2-core licenses. So, instead of 24 two-core licenses (8 each for each physical 16-core host) we ended up buying 48 2-core licenses (3 systems x 8 two-core licenses minimum per system x 2 due to virtualization). We have created three server 2019 installations, each running as a VM on each of the afore-mentioned ESXi hosts. Can one tell how many more installations could be made? Asking here because WAN accelerators run only on MS Server systems... Wish Microsoft had a system that showed X licenses bought, Z used, X-Z free or something</off-topic>

[HannesK]

Hello,

1)

he could simply minimize the immutabily interval to 1 day and then wait out to strike

The minimum configuration is 7 days. But he could remove the immutability checkbox. So that should be monitored (VeeamONE has an alarm for it). V12 has a dedicated "Hardened Repository" type. That one is always at least 7 days immutable

2) Having it unsecured is a weak link.

3) While I prefer a physical backup server for simplicity in environments without geo-redundant storage, a physical server does not make it more secure per definition. Most customers put the backup server outside the domain, because they expect their domain to be infiltrated sooner or later.

Best regards,
Hannes
PS: no opinion on Microsoft licensing. I never heard that licensing scheme before

[cosmik]

Most customers put the backup server outside the domain, because they expect their domain to be infiltrated sooner or later.

The issue is not having the backup server outside the domain. The issue is having it outside the hypervisor. Our hypervisors are not domain-connected for that purpose. How would a non-domain-connected backup server as a VM on (non-domain-connected) EXSi cluster compare to a non-domain-connected backup server on a physical system?

[HannesK]

the main difference is, if an attacker gets access to the ESXi / VCenter, he can delete the backup server.

[cosmik]

I could live with a deletion, I'd recreate the server. Wouldn't the attacker be able to read VM memory and obtain various passwords though from the running backup server VM? This would be far worse than denying me temporarily (since I can always install another server) the capability of doing restores/backups. Would you find this reasonable?

Then again, I assume that nothing would survive an ATP attack and that whatever we do here is basically for ransomware attacks.

[HannesK]

sure, with access to VCenter / ESXi, an attacker could also boot from a 3rd party media, reset the password and then normally access the backup server. then he does not even need to find data in (ESXi) memory. with access to the backup server, he could also dump credentials from the database (same like with any other software that stores credentials, e.g. monitoring software, deployment software etc.)

physical hardware reduces attack vectors like the ones above. but also introduces new ones via out of band management interfaces for example

[Giacomo_N]

Hi, same thoughs of @cosmik, I want to replace my virtual repository of backup copy jobs with physical hardened repository, GFS backup copy, with idrac (out of band management ) off.
After read this topic seems to be not a great idea, because if my out of domain windows virtual Veeam server will be compromise, the "bad guy" could remove the immutability checkbox!
With V12 and dedicated repo is it possible to "remove" the immutability?

How can we hardware monitoring a server without the dedicated management Off? For example broken disks, etc.
I can only imagine a scheduled manual port activation of the switch point to management NIC to check if everything's good.

[Giacomo_N]

Regarding V12 and immutability I've found an official confirmation, I'll wait the new release before go ahead with physical repo.

https://community.veeam.com/blogs-and-p ... ation-2742

[Erik @ Suez]

I have read a post of someone who does his monitoring of a hardened server via a WebCam
I think it was an HPE server where you can fold-out a small display with all the monitoring/warning led's.
Just point the WebCam at the display and add the video feed to your monitoring overview.

[einhirn]

Other than using a webcam you can only look into push based monitoring, ideally with some way of inserting that into the regular monitoring... First approach would be to configure emails to be sent when something happens. But I can imagine something like mail or outbound connection to a monitoring server for an info-push. Don't know whether e.g. CheckMK or Nagios based stuff can work that way (passive check?)

[cosmik]

How can we hardware monitoring a server without the dedicated management Off? For example broken disks, etc.
I can only imagine a scheduled manual port activation of the switch point to management NIC to check if everything's good.

For disk issues (which is the most probable thing you could encounter and was my biggest concern), look no further than smartmontools, configure it to do daily checks and send an e-mail on error.

EDiT: And I have not removed SSH; I've set an extremely intolerant fail2ban to block forever on a VERY small number of tries. Worth a shot, if your server is LAN-visible only. Worst case, you'll have to go the box and login locally...

[dloseke]

<Off-topic>We have had three 16-core servers, two of them in a ESXi cluster. When deciding for the number of Windows Server 2019 Standard licenses to buy I believe we were informed that since this would be a virtualized platform, we'd have to purchase twice the number of 2-core licenses. So, instead of 24 two-core licenses (8 each for each physical 16-core host) we ended up buying 48 2-core licenses (3 systems x 8 two-core licenses minimum per system x 2 due to virtualization). We have created three server 2019 installations, each running as a VM on each of the afore-mentioned ESXi hosts. Can one tell how many more installations could be made? Asking here because WAN accelerators run only on MS Server systems... Wish Microsoft had a system that showed X licenses bought, Z used, X-Z free or something</off-topic>

Microsoft licensing is confusing, but once you wrap your head around this concept, it's not.....terrible I guess. TL;DR/calculations at the bottom

The licensing concepts
Assuming no DRS/vMotion/HA between hosts, you should have both hosts fully licensed so that it can support hosting all VM's on either any host at any time. To make the numbers easy for the theoretical work, I'll start out with the assumption that you have two hosts and not three. In the case of two 16-core hosts (or less cores due to the 16 core licensing minimum), you'd need to buy a 16-core pack (or 8 2-core packs) for each host for every 2 VM's (or what Microsoft will call a VOSE - Virtual Operating System Environment). It should be noted that either 16 core packs or 2 core packs are acceptable for licensing, or even a combination, but to alleviate confusion, Microsoft intends you to buy 16-core pack for hosts with 16 cores or less (because of that 16 core minimum), and using 2 core packs if you have more than 16 cores per host. So if you had 2 hosts, 16 cores per host, and 4 VM's that needed to run, you'd need to purchase 4 16-core packs in total (or 16 8-core packs) - again, assuming no HA, etc. After you get to around 11 VM's in the cluster, it makes more sense to use Datacenter licensing because you then can support unlimited VM's per host, so you'd need 2 16-core Datacenter licenses. If you have more than 16 cores per host, you'd want to use x number of licenses in 2-core packs to get up to the number of cores you have (for instance, 20 core hosts would require 10 2-core packs per host per 2 VM's assuming Standard, or 10 2-core packs per host for Datacenter). Note that MS does allow the core licenses to be transferred between hosts every 90 days, so if you weren't fully licensed for HA and all of your VM's failed over to one host, they would need to stay there for 90 days before moving back to the original host to stay within licensing guidelines. Although, I think you could theoretically move the VM's that were not failed over to the host that failed over (essentially a swap of the VM's to the opposite host), and then they would have to stay on that host for 90 days, but I don't really recommend this because it's outright confusing.

I'm going to ignore the fact that only two hosts are in a cluster and the third is standalone for now because that configuration sounds to me like you have two hosts in a production cluster at a primary site and the third host is at a DR recovery site but if that is the case, you'd need to pay attention to your licensing counts if your primary site fails and you have to spin up all VM's at a recovery site. If they are all at a primary site and running production, then the calculations below make sense. That will be my assumption. I'm also not going to touch on what Software Assurance (SA) offers for failover rights because in my experience, not many folks have been purchasing SA for this purpose.

TL;DR (the calculations)
Since you purchased 48 2-core licenses (or 96 core licenses in total) and have three hosts, you have 32 cores licensed on each or 16 core's licensed on each host twice. With each host fully licensed 2 times and each full round of licensing allowing 2 VOSE's, you can have up to 4 VM's on each host. Since you already have one VM on each host, you can add three more VM's on each host. Again, this is not including any sort of HA failover or DRS when more than four VM's might get dumped on one or both other hosts in the event of a host failure or maintenance. Note that if this was a three host cluster, you can't split the core licensing in the event of failover/maintenance, so you can't have a total of 12 VM's, 6 on one host, 6 on another host. You'd have to have 8 on one host and 4 on the other. But again, you stated that two of the hosts are in a cluster and one is not.

If you had purchased 24 2-core licenses like you initially planned, you would have had enough licenses to place 2 VM's on each host because 24 2-core packs / 3 hosts = 8 2-core packs per host or 16 cores licensed or each host fully licensed once, allowing for 2 VM's per host.

Notes
You mentioned that one host was not part of the cluster, and you are talking about WAN accelerators, so it DOES sound like you are talking about replicating VM's between sites. If that is the case, make sure you have enough licenses to support that failover of the VM's all running on the recovery host.

Sorry I got a bit long-winded, but is that about as clear as mud?

[cosmik]

Derek seriously, extremely well detailed description mate, I didn't think someone would touch my question in that agonizing detail

While waiting for a response here, I stumbled upon a quite clear explanation at https://www.nakivo.com/blog/the-essenti ... -machines/ which basically states the same things: for 2VMs on a normal (non-HA/clustered) 16core host one needs 8 2-license packs from Microsoft for Windows Server standard. In case one has hosts in a cluster, then licenses are bought for the case all VMs end on a host. The following examples from the afore-mentioned link states it clearly:

Example 2

We have two physical servers. Each server has one 12-core processor. Four VMs must run on each server, and Windows Server 2019 Standard is installed on each physical server. We have to buy 16 double-core licenses or two 16-core licenses for each server. In total, we buy four 16-core licenses for the whole Windows Server environment.
Example 3

The configuration is the same as in Example 2. Each server has 4 VMs and these VMs must be able to migrate between servers. As we have two servers with an equal number of VMs on each server, the number of licenses for each server for VM migration must be doubled (4VMs+4VMs). As a result, we need to buy an additional 16 double-core licenses for each server. Finally, each server has 32 double-core licenses (or four 16-core licenses).

If we would have 4 VMs on the first server and 6 VMs on the second server, each server must be licensed to run 10 VMs in case of a VM migration. As a result, four additional 16-core licenses must be bought for each physical server. Totally the equivalent of 80 cores must be covered on each server and there must be 40 double-core licenses (or 5x16 core licenses) for each server.

Onwards with your post, I'll clarify some things a bit as well:

I'm going to ignore the fact that only two hosts are in a cluster and the third is standalone for now because that configuration sounds to me like you have two hosts in a production cluster at a primary site and the third host is at a DR recovery site...

Both are production sites. The difference is that the main site has around 100 employes, whereas the branch around 10. I needed a cluster on the main branch to handle things, due to our budget having a cluster on the remote site was out of the question.

I think I've got it now: please let me know if not. Specifics:
(a) on the main site/cluster I've got one Windows VM running on each ESXi host.
(b) on the branch host I've got also one Windows VM running (but no cluster)

The sites on (a) due to HA require double the number of licenses. Therefore, the machines should be licensed as if all Windows VMs were running either on the first host or the second. Since 2VMs is the maximum number of VMs running on the entire cluster (1+1), and 8 2-license packs cover 2VMs on a 16-core host, I'd need 8 packs for the 1st host and 8 for the 2nd. All in all 16 2-license packs (for 32 cores). If I configure another 2 Windows VMs on the cluster, I can use my 32 (out of 48) 2-license packs. I've already made one to host the WAN accelerator (which will be used basically for my offsite copy backup jobs to compensate for the pitiful low-bandwidth of the branch site). I do not have any DR plan in place. I'm just backing up stuff off-site with Veeam. And of course I'm having everyone in the same domain.

That leaves 16 2-license packs, which is too many. I needed only 8 of them to cover the single host at the branch site for 2VMs (only have one atm).

As for the 8 2-license packs remaining: can they be used on the cluster (I don't think so, as per your post and what I've read)? Or can they only be used at the branch site to host another 2VMs?

And another, most likely stupid question:

Note that MS does allow the core licenses to be transferred between hosts every 90 days, so if you weren't fully licensed for HA and all of your VM's failed over to one host, they would need to stay there for 90 days before moving back to the original host to stay within licensing guidelines. Although, I think you could theoretically move the VM's that were not failed over to the host that failed over (essentially a swap of the VM's to the opposite host), and then they would have to stay on that host for 90 days, but I don't really recommend this because it's outright confusing.

Errrr, how do I "associate" a Windows VM with a specific host? Should I do something special on the MS licensing site? I just activated Windows Server, nothing more...

[dloseke]

The sites on (a) due to HA require double the number of licenses. Therefore, the machines should be licensed as if all Windows VMs were running either on the first host or the second. Since 2VMs is the maximum number of VMs running on the entire cluster (1+1), and 8 2-license packs cover 2VMs on a 16-core host, I'd need 8 packs for the 1st host and 8 for the 2nd. All in all 16 2-license packs (for 32 cores). If I configure another 2 Windows VMs on the cluster, I can use my 32 (out of 48) 2-license packs. I've already made one to host the WAN accelerator (which will be used basically for my offsite copy backup jobs to compensate for the pitiful low-bandwidth of the branch site). I do not have any DR plan in place. I'm just backing up stuff off-site with Veeam. And of course I'm having everyone in the same domain.

Yes, you are correct in that if you have two 16 core hosts in a cluster, and 2 VM's running on that cluster, you'd need 16 2-core licenses, one set of 8 each for each host to support both VM's running on either host with HA failover capabilities. The branch site will use 8 of the 2-core packs for it's current 1, up to 2 VM's. That leaves you 24 2-packs to do whatever you need. Need to add two more VM's on the cluster? That'll use up 16 more of your 2-core packs (16 2-packs per host in total). That would still leave 8 2-packs unused that you could use on the standalone host to support up to 4 VM's as well. You just wouldn't be able to use that last 8 licenses on the cluster unless you tied the 2 VM's it would support to one of those hosts in the cluster with no failover capabilities.

Errrr, how do I "associate" a Windows VM with a specific host? Should I do something special on the MS licensing site? I just activated Windows Server, nothing more...

This is all theoretical assignment and based on the honor system. You just know where the licenses are "assigned" and what your capacity is. I recommend documenting what licenses you have and how they are assigned and keeping that in your documentation repository so that you know what is where. There's no actual system for documenting this. For me, I used IT Glue to I have a licensing section and would keep the licensing something like this:

Code: Select all

Manufacturer: Microsoft
Product:  Windows Server
Version: 2019 Standard
Seats: 48
License Key:  xxxxx-xxxxx-xxxxx-xxxxx-xxxxx
Notes:
  Total Licenses: 48 2-core packs Purchased
  Cluster:
    Host1: 8 - 2-Core Packs (To support up to 2 VM's)
    Host2: 8 - 2-Core Packs (To support up to 2 VM's)
  Branch Office:
    Host1: 8 - 2-Core Packs (To support up to 2 VM's)
  Remaining Available Licenses: 24 2-Core Packs to be assigned as needed

And then under my Configurations, I'd associate each of the host assets with that licensing asset and could add notes to each host that would be something along the lines of:
Host is licensed with 8 Windows Server 2019 2-Core Packs supporting up to 2 VM's

Whatever makes sense to you really, but that's more or less how I do it. If you don't have IT Glue or Hudu or some sort of documentation system, keeps those notes somewhere. OneNote, Word, a TXT documents, Sharepoint...whatever makes sense and is where you want to keep all of your documentation if you don't already have something in place.

[cosmik]

Derek you're awesome mate. Thanks a zillion!

[LickABrick]

Make sure your hardened repo does not use a NTP and DNS server which can be compromised when they get in your environment. The immutability date is based upon the time on the hardened repository, if an attacker gets access to the NTP server he can set the time to somewhere in the future where the immutability has expired. And if and attacker gets access to your DNS server he can simply create a DNS record to redirect to a NTP server within their control and basically do the same thing. And if they get in your router/firewall they can do all sorts of thing with routing/NAT so even using a public NTP/DNS server might not be completely safe.

I would suggest disabling time sync. There's some interesting talk about NTP in the comments of this post: https://community.veeam.com/blogs-and-p ... ehaves-276

I have seen people forget the above so just wanted to make sure you don't

[dloseke]

I've never considered this to be a possibility. Interesting thought though.