Please critique my design

[dhayes16]

Hello All.
I was wondering if you could review this project linked below. This is a rather large project and I want to make sure we are working within the best practices for Veeam. Please tell me if I am off my rocker and/or if you have other potential ideas/solutions. Basically, they want to backup the servers in Seattle and Dallas and also have immutable storage. The server in Boston is just a B&R server but that site has a 500/500 fibre pipe.

I am concerned about the different hyper-V versions for disaster recovery and replication if needed.

Be gentle..lol

https://imgur.com/gallery/g0BUDpI

[joachimeberhard]

Out of curiosity: It is not possible to bring the HyperV hosts to the same level (2022)?
I had the same design decision 2 months ago, and I chose to stick all full with SVR2019 despite having 2022 licenses, because one host did not support uefi/tpm.

[dhayes16]

Thanks for the response. This is a good point. Both of these servers are Dell R740's and they both do support Windows 2022 so this is an option. I just need to deal with the licensing requirements and such. I suspect I could get the requisite number of server licenses and keep the VM's at whatever they are now.

Thanks for the idea. I am just trying to get feedback is my design is a little outlandish for Veeam. They might decide to get rid of the Linux hardened repos and go with VCC with insider protection or object with immutability. But they still need a landing zone for Dallas (performance tier).

Again thanks!

[joachimeberhard]

From what I know from Veeam, using the same HyperV versions is necessary for certain features.
I am not entirely sure which those are, but i suspect things like auto-failover and live migration and such.

It is really not necessary to touch the vms in the first process, but I guess its a good idea to get HyperV to the same level.

[joachimeberhard]

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

Note: Versions of a Microsoft Hyper-V host and off-host backup proxy must coincide. For example, if you use a Microsoft Windows Server 2012 machine with the Hyper-V role enabled as a Microsoft Hyper-V host, you must deploy the off-host backup proxy on a Microsoft Windows Server 2012 machine with the Hyper-V role enabled.


This is what I could find quickly, I think I read a few times something about something like this.

[dhayes16]

Thanks...Much appreciated...I have read this in many places as well that this was an initial design decision by Veeam to make sure there no issues but this is quite a limitation of using replication and such. At the very least I could bring the Seattle host up to 2019 and get the BDR's with 2022 licenses but downgrade to 2019. But we will see what they want to do at this point.

[joachimeberhard]

Personally, I think 2019 is a very proven release at this point.
I made the decision to stay with 2019 for the moment because I did not want to mix it up.
But if they have only few VMs, licensing might not be that much of an issue.

Upgrading 2 very remote hosts though, might be.

Edit: If you do feel safe about upgrading both hosts to 2022, I would consider this definitely.

[Mildur]

I wouldn‘t use Off Host Proxies. It‘s recommended to use On Host proxy when possible. Makes everything alot easier to manage.

The issue will be in replication. For example, you can‘t replicate vms from a HyperV 2019 to a HyperV 2016 version. Target OS must be equal or higher than the source OS.

And the same limitation for the restore process. You can‘t restore 2019er vms on your hyperv 2016 server, if Dallas is offline or have other issues.

The version of the target host on which the VM is restored must be the same or later than the version of the source host where the original VM was registered.

—————————

I would use in Boston a Linux hardened repo instead of a windows vbr server. Or you can place the main vbr server in Boston. Let the OnHost Proxies on the hyperv server do the backup to the linux hardened repo at the same location and create a copy job to Boston.

I never had a hot stand by Backup server. A new backup server doesn‘t take to long to deploy and restore the configuration backup. If you have to use one, then your design works of course. Just make sure, that you are patching the backup server in Seattle and the one in Boston together. They are sharing components in your design.

[joachimeberhard]

Yes, I did not suggest to make an off host proxy, it was just an example if incompatibilities I could find quickly.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

Your example is much better, I was just philosophizing about the best way to get version consistency.

[Mildur]

Thanks Joachim.

The main issue is in replication and restore. Hosts should have the same version. Makes everything easier

[joachimeberhard]

There is a few things to consider when upgrading a HyperV host though.

First I recommend updating firmware and drivers before you upgrade your OS.

Second, in case of Windows 2022 you should check if your existing hosts are already installed in UEFI mode and secure boot / TPM is enabled.

Ideally, you do this after a working backup of course

[dhayes16]

The issue will be in replication. For example, you can‘t replicate vms from a HyperV 2019 to a HyperV 2016 version. Target OS must be equal or higher than the source OS.

And the same limitation for the restore process. You can‘t restore 2019er vms on your hyperv 2016 server, if Dallas is offline or have other issues.

Thank you for your detailed response!! So I should have been more clear on the Windows 2022 servers at each location. The plan there was to install the Hyper-V role on both of those servers and the replicated VM's would live on those servers. For instance, the Dallas 2019 Hyper-V VM's would replicate to V-1 and the Seattle Hyper-V VM's would replicate to V-4 in Boston. Would the issue of OS version issues still hold true here? We would load the B&R server in a VM on each host.

Basically the plan was to build V-1 and V-4 to be able to handle the DR duties and run VM's either in a DR situation at the local site or the remote site. For instance, if Dallas (S-2) gets nuked the replicated VM's sitting on V-1 can failover. Also, if the Seattle server (S-1) drive array crashes we can spin up VM's on V-1 to continue to operate (Dual purpose).

And if Seattle gets nuked we can failover to the V-4 server sitting in Boston. As I type this the only issue I see would be if the Dallas server (S-2) crashes we will not have a BDR to run their VM's since it is sitting on a Linux Hardened Repo. Should we flip flop those servers? Perhaps move V-3 to Boston and V-4 to Dallas and change the replication scheme?

Again many thanks for the help and feedback. This is a rather large one and we want to do it right.

[dhayes16]

There is a few things to consider when upgrading a HyperV host though.

First I recommend updating firmware and drivers before you upgrade your OS.

Second, in case of Windows 2022 you should check if your existing hosts are already installed in UEFI mode and secure boot / TPM is enabled.

Ideally, you do this after a working backup of course

Thanks very much...I agree that server 2019 is very stable and I might just take the road of getting the 2016 Hyper-V upgraded and leave the 2019 alone at the moment. The 2019 sever is very remote for me where the 2016-V box is in town so it would be easier to upgrade.

Thanks again!!

[joachimeberhard]

If the Windows 2022 servers are not HyperV hosts, there will not be a problem at all.

Else I agree, I personally would update the 2016 HV host to 2019, far less headache.

I have also installed Veeam inside a VM, its a far superiour design and outweighs the small performance impact it has.

Benefit is, I can replicate the VBR servers too, and those together with the DCs, make a restore possible in desaster case.

[dhayes16]

Thanks...But I am confused by your response. We had planned to have the BDR servers running Windows Server 2022 WITH the Hyper-V role installed. So on these BDR servers I would have the Hyper-V role and have Veeam B&R running in a VM. And these servers would be replication targets for each location as well as the backup target for the local servers.

So I am thinking that I could upgrade the 2016 Hyper-V host to 2019 so both production servers are running Windows 2019. But the BDR's are running Hyper-V 2022. Will this be an issue?

Again, thx

[joachimeberhard]

Yes, it will be an issue.

It is better to go all 2019 then.

From a performance perspektive, I found Veeam in a VM runs best with 8 vCPUs and 32GB RAM for my scenario.
Giving too much vCPU or RAM makes it worse.
It is better to install another server instead.

[dhayes16]

Thanks again!. The only thing I need to decide on is if I want to move the Windows BDR server from Boston to Dallas and "possibly" the Linux repo from Dallas to Boston. In reality since there are no servers in Boston it might make sense to move a Linux repo over there.

The primary reason I think the windows BDR should be in the same location as the workload it is protecting is to be able to quickly spin up VM's in that location in the event of a server crash (Raid card, etc). I could rely on the replicas but I think it would be easier to keep it in the same locaiton.

What do you think?

[fcardarelli]

Hi. If you don' thave production servers in boston maybe its good to take it out of the bug picture.
I totally agree having on Hyper-V nodes on the same version 2019.
Use RefS on the windows repos.

What I would recommend is creating a backup network in S-1 or S-2 and host the VBR there.
Use On-Host proxy.
Replicate the VBR between S1 <-> S2
Have perf tier on both sites and archive tiers from the remote site.

S1:
- Local Perf Tier
- Remote Arch Tier (S2)

S2:
- Local Perf Tier
- Remote Arch Tier (S1)

In case of a failover, you can publish the backup network (static/dynamic routing) on the other site and keep doing backups. If you install the VBR on the same network as the Hyper-V nodes you can't do that.
To protect from a ransomware, purchase a DraaS Cloud Provider with insider protection on and host a 3rd copy off-site with inmutability.

I have a couple of questions that might help to have a better design:
- Whats the wire speed between Dallas and Seattle?
- What services are offered from each site? Offices? Or production site?
- The Dallas site is a DRP from Seattle?
- Application Services are balanced between sites or they just offer services locally?
- RPO and RTO desired?
- Internet links on both sites?
- IT team in any location?
- Static or dynamic routing?

Regards
Francisco

[c.haydock]

Veeam not even being a part of the equation... just looking at Hyper-V capabilities alone... If you want to move VMs from one Hyper-V host to another using native Hyper-V tools, you'll have to make sure that the VMs are running a Hyper-V Configuration Version that the two hosts understand. So, Server 2016 can understand up to a maximum of configuration version 8. Server 2019 can understand a maximum of configuration version 9. And, Server 2022 understands a maximum of configuration version 10. (There are other sub-versions available if the VMs were made on Windows 10 or using one of the servicing channel versions of Server.) You can view the config version in Hyper-V manager, or run this from a command prompt to see what version they are running....

Get-VM * | Format-Table Name, Version

All that said, you could for instance move/replicate a VM on Server 2016 over to 2022 just fine because 2022 will understand the VM's config version. But, the same would not be true going the other way around. HOWEVER (there's always a catch)... you can create new VMs on newer versions of Hyper-V and force a lower config version to be used. So, if you are mindful of it when you stand up a new VM on 2022... you can create the new VM using config version 8 instead of 10 so that you can move the VM between the two hosts. The config version can be specified one-off upon VM creation, or as a general Hyper-V setting. Also, once you upgrade all your Hyper-V hosts, you can upgrade all the VMs config versions if you like. It's fairly straight forward though the GUI, but the VM does need to be off for the upgrade. Downgrading to my knowledge is not possible. To downgrade you would have to make a new VM with the lower version and then transfer over the VMs VHDX files to it.

But, as mentioned by a few others... in general there tends to be a lot less headache if all your hypervisors are on the same version... even if they aren't clustered.

[dhayes16]

Hi. If you don' thave production servers in boston maybe its good to take it out of the bug picture.
I totally agree having on Hyper-V nodes on the same version 2019.
Use RefS on the windows repos.

What I would recommend is creating a backup network in S-1 or S-2 and host the VBR there.
Use On-Host proxy.
Replicate the VBR between S1 <-> S2
Have perf tier on both sites and archive tiers from the remote site.

S1:
- Local Perf Tier
- Remote Arch Tier (S2)

S2:
- Local Perf Tier
- Remote Arch Tier (S1)

In case of a failover, you can publish the backup network (static/dynamic routing) on the other site and keep doing backups. If you install the VBR on the same network as the Hyper-V nodes you can't do that.
To protect from a ransomware, purchase a DraaS Cloud Provider with insider protection on and host a 3rd copy off-site with inmutability.

I have a couple of questions that might help to have a better design:
- Whats the wire speed between Dallas and Seattle?
- What services are offered from each site? Offices? Or production site?
- The Dallas site is a DRP from Seattle?
- Application Services are balanced between sites or they just offer services locally?
- RPO and RTO desired?
- Internet links on both sites?
- IT team in any location?
- Static or dynamic routing?

Regards
Francisco

THANK YOU for the very detailed explanation and critique. Yes we are going to change this to move the VBR server from Boston to Dallas and make it the perf tier for Dallas and Archive tier for Seattle. Also, we are going to make sure all Hyper-V hosts are 2019.

I have never even thought of replicating the VBR server to the other location and just using one location (Seattle) to manage everything. I do like that idea. So if Seattle goes down (Fire, etc) then we would be able to spin up the VBR at that location and bring up the replicas? I always read where the VBR server should be at the DR site but in our case both sites are DR sites for the other. I like that idea a lot.

When you mention to create a "backup network" do you mean on V-1 and V-4 on my diagram? You mentioned S-1 or S-2 which are the production Hyper-V boxes. Just confirming on what you said.

To answer your questions:
- Whats the wire speed between Dallas and Seattle? - This is a VPN connection where Seattle has 150/150 fibre but Dallas has 200/10 asymmetric cable)
- What services are offered from each site? Offices? Or production site? This is a windows domain so domain controller replication between sites. But primarily file and print as well as an ERP system. Email and such is in the cloud via O365.
- The Dallas site is a DRP from Seattle? Yes...And Seattle is DRP from Dallas. Each site is DR for the other.
- Application Services are balanced between sites or they just offer services locally? Just offer services locally...No load balancing.
- RPO and RTO desired? We were hoping for RTO for 2 hours and RPO of 4.
- Internet links on both sites? Yes Internet links listed above.
- IT team in any location? There is an IT team in Seattle but not really in Dallas.
- Static or dynamic routing? Please explain this a little more. Each site has static IP's via their Internet provider externally.

Again thanks for taking the time to review this design.

[dhayes16]

Veeam not even being a part of the equation... just looking at Hyper-V capabilities alone... If you want to move VMs from one Hyper-V host to another using native Hyper-V tools, you'll have to make sure that the VMs are running a Hyper-V Configuration Version that the two hosts understand. So, Server 2016 can understand up to a maximum of configuration version 8. Server 2019 can understand a maximum of configuration version 9. And, Server 2022 understands a maximum of configuration version 10. (There are other sub-versions available if the VMs were made on Windows 10 or using one of the servicing channel versions of Server.) You can view the config version in Hyper-V manager, or run this from a command prompt to see what version they are running....

Get-VM * | Format-Table Name, Version

All that said, you could for instance move/replicate a VM on Server 2016 over to 2022 just fine because 2022 will understand the VM's config version. But, the same would not be true going the other way around. HOWEVER (there's always a catch)... you can create new VMs on newer versions of Hyper-V and force a lower config version to be used. So, if you are mindful of it when you stand up a new VM on 2022... you can create the new VM using config version 8 instead of 10 so that you can move the VM between the two hosts. The config version can be specified one-off upon VM creation, or as a general Hyper-V setting. Also, once you upgrade all your Hyper-V hosts, you can upgrade all the VMs config versions if you like. It's fairly straight forward though the GUI, but the VM does need to be off for the upgrade. Downgrading to my knowledge is not possible. To downgrade you would have to make a new VM with the lower version and then transfer over the VMs VHDX files to it.

But, as mentioned by a few others... in general there tends to be a lot less headache if all your hypervisors are on the same version... even if they aren't clustered.

Thanks for the detailed explanation. We are going to go 100% Server 2019 in this case. 2019 has proven to be solid and we want to Keep in simple especially in a DR situation. So the Seattle location will need to be upgraded to Server 2019 and the BDR's will just be loaded with 2019.

I am just trying to figure out if we want to use Linux hardened Repos or cloud based solutions for immutability.

Thanks again