New Veeam Essentials user seeking advice on setup

[cosmik]

Hello,

just bought 20 Veeam essentials universal licenses and are struggling to find out how we should go with setting up the product. The image depicts our setup


Some details:

• Geographically, we have a main office and a branch one, connected over a "company" WAN over 100Mbps and 2 (!) Mbps respectively. The latter will be upgraded to 10 in the coming months.
• Two esx hosts, 1 and 2, will become a cluster with the Dell FC SAN providing the storage (waiting for the upgrade to the VMWare essentials plus package)
• One synology unit on each site with lots of storage (Seagate 14Tb enterprise disks). Not using them at the moment (on the main site storage is provided by the Dell SAN, whereas on the branch office the ESXi server local datastore is more than sufficient for the 2 VMs) but we do plan to use them to host the backup themselves.
• Both esx1 and 2 have a low memory utilization. VMs running on each machine are not very taxing. More than half of the host RAM is unclaimed.
• All 3 esx servers support the needs of an office network, running workdays (Mon-Fri) and office hours (7am to 5pm). With the exception of the mail server which should be network accessible 24/7, there is nothing else that runs all the time. In fact, one could stop everything albeit the mail servers and the AD VMs (one Server 2019 VM on each esxi host)
• Wan accelerator is a no-no here, if I understand correctly, since we do not have the Veeam enterprise license.
• Would feel better if I could have each site have a remote backup to the other one (3-2-1 rule)
• I've read the stuff written about using general purpose NAS. Unfortunately, we've already bought these Synology units and disks...
• Total newbie here :p

In previous correspondence here I've been instructed to have a look at Veeam requirements. I'm sorry, but I got lost there. There's a ton of detail and a multitude of options to cover for all cases, but I need people to actually step in and point to directions for my specific setup.

[soncscy]

Heya Michael,

https://www.veeam.com/smb-vmware-hyper- ... tials.html

As far as I know, Essentials is only sold with VUL (universal licenses) so you get Enterprise Plus (ENT+) by default. Maybe a Veeam Employee can correct me.

A few things that you'll want to do:

1. Use iscsi for the repositories and spin up a dedicated LUN on the Synologies for your Backups; connect a repository server to the iscsi target for your Synology.
2. WAN Accelerators will mitigate some of the damage of that slow link to site 2, but is there any way you can increase it even more? 10 Mbps is workable with Backup Copy Seeding, just means that someone needs to toss some drives in their car and go from main to branch: https://helpcenter.veeam.com/docs/backu ... ml?ver=110
3. With your host resources, I don't anticipate hot-backups causing you an issue; I'm not sure if you have a physical box for your VBR server/backup proxies or if your Dell storage for the datastore also lets you simultaneously export via iscsi, but you could do Backup from Storage snapshot for nice and tidy backups that leave the ESXi hosts alone; else you can keep it very simple and just use hotadd proxies (just any virtual machine in the same VMware Datacenter (logical datacenter) ought work almost immediately; search linux hotadd on this forum and you'll find some great tutorials for Ubuntu and Debian. Even if you're not a strong linux shop, it's a very low-knowledge investment project and will save you gobs on Windows licenses, plus far less of a pain to just spin up another if something goes ass-up with the proxy)
4. Long-term, you might look at Capacity Tier for additional off-sites if tape isn't an option for you: https://helpcenter.veeam.com/docs/backu ... ml?ver=110 I prefer this for my clients as opposed to rotated drives as individual rotated drives still have too low of a ceiling to be reasonable in most cases. They're fine for an 'oh snap' moment and last resort, but I wouldn't ever rely on them. S3 gets tricky with pricing though, so shop around on vendors. AWS/Azure have some pretty outrageous API charges so be aware. Backblaze and Wasabi have been doing fairly well and both support immutable out of the box, which helps a lot of my clients sleep at night

Hope some of this advice helps. Your biggest issue is that branch connection and there are workarounds; I'm honestly not confident you're going to get high returns on WAN Accelerators, but it depends entirely on your workload and how "dedupable" it is.

[cosmik]

Heya matey!

A ton of useful pointers here, thank you!

As far as I know, Essentials is only sold with VUL (universal licenses) so you get Enterprise Plus (ENT+) by default. Maybe a Veeam Employee can correct me.

Well I really wish you'd translate this into English for me My angst is that I've possibly bought the wrong product here :/

1. Use iscsi for the repositories and spin up a dedicated LUN on the Synologies for your Backups; connect a repository server to the iscsi target for your Synology.

Ok detail time. Suppose you're starting from scratch here. What are you going to setup from Veeam and where? A, say, "full" install as a VM on one of our Site A servers? Another full install on site "B" (on host "B"? These are the start points of my journey here, any detail provided will be appreciated.

Any intro designed for total idiots to the terminology used by Veeam, so I can get a grip?

2. WAN Accelerators will mitigate some of the damage of that slow link to site 2, but is there any way you can increase it even more? 10 Mbps is workable with Backup Copy Seeding, just means that someone needs to toss some drives in their car and go from main to branch: https://helpcenter.veeam.com/docs/backu ... ml?ver=110

No, unfortunately nothing can be done here... I'll study what you've posted.

3. With your host resources, I don't anticipate hot-backups causing you an issue; I'm not sure if you have a physical box for your VBR server/backup proxies or if your Dell storage for the datastore also lets you simultaneously export via iscsi...

The Dell is FC-connected to the hosts. Problem is that I want to avoid even touching the thing

... else you can keep it very simple and just use hotadd proxies (just any virtual machine in the same VMware Datacenter (logical datacenter) ought work almost immediately; search linux hotadd on this forum and you'll find some great tutorials for Ubuntu and Debian. Even if you're not a strong linux shop, it's a very low-knowledge investment project and will save you gobs on Windows licenses, plus far less of a pain to just spin up another if something goes ass-up with the proxy)

Not that bad with Linux, will read about hotadd proxies (even though I don't understand what they're proxying).

4. Long-term, you might look at Capacity Tier for additional off-sites if tape isn't an option for you: https://helpcenter.veeam.com/docs/backu ... ml?ver=110 I prefer this for my clients as opposed to rotated drives as individual rotated drives still have too low of a ceiling to be reasonable in most cases. They're fine for an 'oh snap' moment and last resort, but I wouldn't ever rely on them. S3 gets tricky with pricing though, so shop around on vendors. AWS/Azure have some pretty outrageous API charges so be aware. Backblaze and Wasabi have been doing fairly well and both support immutable out of the box, which helps a lot of my clients sleep at night

There will be some organization-bought Azure space, so perhaps I could transfer stuff there for cold storage. Not in my direct plans.

Your biggest issue is that branch connection and there are workarounds

I'd say that my biggest issue is my total lack of competence in the server/storage/backup area. We don't have that many servers, so I don't have that hard-earned knowledge that comes for doing the same thing again and again... Proper way would be perhaps to hire someone else do this, but we lack the capability at this point to do something about that. Oh well...

[cosmik]

Took me a while (less than a year) to get started, but I've finally reached that magical "install" moment (doh).

Considering my setup, would it be advisable to do a physical or a VM installation? And where would I need to start reading/preparing for configuring Veeam?

[Mildur]

Hi Michael

You can start with our Quick Start Guide:
https://helpcenter.veeam.com/docs/backu ... ml?ver=110

I don't see any physical Host except the ESXI Hosts in your graphic. So I assume it will be a virtual installation. Just make sure, that you use a VM on each site for the iscsi connection to the synology.
Don't mount the iscsi LUN over the 2 Mbit Link. Not even try it
This 2 Mbit will be a bottleneck, even for backup copy jobs. You should check if you can upgrade this to a higher bandwidth.

And what I'm missing is an airgapped or immutable backup copy. With your design, a attacker can delete backups on both synology's. Consider to use a SOBR with immutable Object Storage. You are not protected against ransomware attacks with the current design.

[cosmik]

Hi Mildur

Hi Michael

You can start with our Quick Start Guide:
https://helpcenter.veeam.com/docs/backu ... ml?ver=110

I'll be taking a look at the quick start guide, thank you. The extensive guide on https://helpcenter.veeam.com/docs/backu ... rview.html got me drowned in information: it overwhelmed me!

I don't see any physical Host except the ESXI Hosts in your graphic. So I assume it will be a virtual installation.

Basically I was asking which way to go for the installation. We could place a physical Windows 10 system (core i5/8th gen with 8Gb of RAM) if a physical system was needed but my basic question was what was preferable: have the server as a VM (on our ESXi cluster) or the Windows 10 physical box.

Regardless of whether we'll go physical or virtual, the Windows platform will be on Windows 10, since we lack any other Server 2019 licenses... And I must add that we are not doing 24/7 operation. System load is "high" (even then no significant CPU utilization takes place) during office days/office hours. So there would be no issue doing our work during off-hours and on weekends.

Just make sure, that you use a VM on each site for the iscsi connection to the synology.

Which roles should this VM have? On the first site that has two hosts forming our cluster, would it be better to have this VM on each host as well?

Don't mount the iscsi LUN over the 2 Mbit Link. Not even try it
This 2 Mbit will be a bottleneck, even for backup copy jobs. You should check if you can upgrade this to a higher bandwidth.

I'd wait till this link was upgraded to 10Mbps, but as @soncscy above advised it would still be too low to do the job. Harvey was kind enough to suggest using seed backups.

What I want to do is to have local backups running and build from there, that is:
* make backups faster (perhaps utilizing snapshotting on the Dell SAN if possible)
* optimize network traffic
* implement "cross" backups
* and have things secured somehow

And what I'm missing is an airgapped or immutable backup copy. With your design, a attacker can delete backups on both synology's. Consider to use a SOBR with immutable Object Storage. You are not protected against ransomware attacks with the current design.

... which brings me to your point. Right now I'm scared shitless considering the simplifications I'm making in order to keep implementation a manageable beast in my mind. So I'm going easy for the time. Help is here is excellent; really hope that I'll be able to do things properly with it.

[cosmik]

With your design, a attacker can delete backups on both synology's.

iSCSI volumes offered from the synology are on top of btrfs volumes. For which daily snapshots will be taken on the synology. Would that mitigate this threat?

[Mildur]

iSCSI volumes offered from the synology are on top of btrfs volumes. For which daily snapshots will be taken on the synology. Would that mitigate this threat?

It will help, until an attacker gets access to the synology admin interface The admin interface must be protected. Put it in another subnet, protect it with a firewall and MFA for the admin user.
A SOBR with immutable Object Storage is still the easiest solution. Have a look at wasabi. Just make sure that your internet upload is good enough.

Basically I was asking which way to go for the installation. We could place a physical Windows 10 system (core i5/8th gen with 8Gb of RAM) if a physical system was needed but my basic question was what was preferable: have the server as a VM (on our ESXi cluster) or the Windows 10 physical box.

Microsoft doesn't like it, if someone use a windows client OS to run a server services. It's against their license terms.

And to use the Synology storage the most effective way, you need a server OS from Microsoft or a Linux OS. In both locations. You can't leverage our FastClone technology with Windows 10, expect for windows 10 Workstation.

I recommend to get in contact with a local Veeam Partner. He can help you to design your backup environment in the best way possible. There are to many things to consider:
- SOBR
- bandwidth
- correct OS version
- placement of veeam components
- immutable backup storage
- security

https://www.veeam.com/find-a-veeam-accr ... rtner.html

[cosmik]

Off-topic but for some reasons your posts do not offer me the option to quote them.

It will help, until an attacker gets access to the synology admin interface The admin interface must be protected. Put it in another subnet, protect it with a firewall and MFA for the admin user.

Thank you for the details provided. No "admin" admin user, combined with a policy to ban SSH/web admin login attempts permanently after 3 tries do the trick nicely for the time being.

A SOBR with immutable Object Storage is still the easiest solution. Have a look at wasabi. Just make sure that your internet upload is good enough.

Day-in/day-out I'm rapidly expanding my vocabulary here, didn't know what "wasabi" is... There's a chance we might be offered free Azure cold backups for 3 years. But that's like trying to get from Earth to Mars for me, when I've barely managed to outline schematics to just get me a couple of meters from the ground :p

Microsoft doesn't like it, if someone use a windows client OS to run a server services. It's against their license terms.

In that case I'll have to do as is for the time and order some Server 2022 license (which I'll have in a year from now...) and hopefully do a migrate to the new server. In the meantime, it's either Windows 10 for me, or some Linux.

(I was under the impression though though that according to Veeam, Windows 10 was a viable installation option...)

And to use the Synology storage the most effective way, you need a server OS from Microsoft or a Linux OS. In both locations. You can't leverage our FastClone technology with Windows 10, expect for windows 10 Workstation.

Got some Windows 10 enterprise licenses, don't know if they could be in-place alternatives to Windows 10 Workstation. I could implement Linux (possibly Debian 10.x) perhaps for certain roles, to avoid

I recommend to get in contact with a local Veeam Partner. He can help you to design your backup environment in the best way possible. There are to many things to consider:
- SOBR
- bandwidth
- correct OS version
- placement of veeam components
- immutable backup storage
- security

https://www.veeam.com/find-a-veeam-accr ... rtner.html

Indeed: I should have done this when I bought the licenses, was under the impression at the time that I could pull this off on my own (not the first, nor the last time I designed something rather complicated and succesfully implemented it by simply asking around/posting/chatting). I failed miserably in this case...

Now, I would have to wait for a tendering procedure to complete if I went about contracting a Veeam partner to help me out.

[Mildur]

Hi Michael

You cannot use the quote button for the last comment in a topic

I was under the impression though though that according to Veeam, Windows 10 was a viable installation option.

It's supported from our side. There are no technical obstacles.

Windows 10 enterprise doesn't support reFS and Fastcloning. FastClone allows you to have really fast and space less synthetic full backups. It works with reFS (Windows Server 2016 or later, Windows Workstation) or XFS (Linux)
If you don't have the correct windows license, deploy two ubuntu vms (Debian is also supported) and connect the synology iscsi LUN's to the vms at their location. Format the LUN with XFS as descripted in our Fast Clone article.

Code: Select all

mkfs.xfs -b size=4096 -m reflink=1,crc=1 /dev/sda1

nor the last time I designed something rather complicated and succesfully implemented it by simply asking around/posting/chatting)

It's always ok to ask The main problem I see is the bandwidth between both locations. The rest should work.
Local Backup to the Linux VM with the connected ISCSI Synology LUN. Backup Copy Job to the other Linux VM in the other datacenter.

[cosmik]

Thanks again for your information! My intention was to proceed with Linux for backup repositories. Possibly for backup proxies. With any luck by the end of next week I'll have 90% of the functionality in place, with immutable backups coming next.

[cosmik]

Some more questions: assuming one wants to go with Linux servers for the role of backup repository and this server has to attach to iscsi, is a gui needed for the system? That is, would it make iSCSI management easier, or is shell access sufficient?

On a similar context, is a GUI required/recommended for immutable backup configuration on the Linux system?

Finally, which would be the most heavily tested/recommended distribution that provides also XFS and fast clone?

I've pushed things to make an initial installation today and proceed from there, so I'm crossing my fingers here.

[Mildur]

That is, would it make iSCSI management easier, or is shell access sufficient?

You don't need a GUI to manage a linux system There are many guides out there on how to connect a iSCSI LUN with a linux machine.

is a GUI required/recommended for immutable backup configuration on the Linux system

No, A linux hardened Repository cannot have any access at all over network. No GUI, no Shell, No VM Console on the Hypervisor. That's why I haven't mentioned to use this 2 vms as Linux hardened repositories. If you use a VM with connected iSCSI LUNs as a linux hardened repository, an attacker will delete all your backups within minutes if he has access to the production hypervisor (Login to vCenter, Reboot the linux vm in Rescue Mode, Reset root User PW, Boot the Linux VM, Logon as Root, Remove Immutability Flag of all Backup Files, Delete all Backup Files).

Finally, which would be the most heavily tested/recommended distribution that provides also XFS and fast clone?

I recommend Ubuntu. But you can use any of them: CentOS 8.2 and 8.3, Debian 10.x, RHEL 8.2 or later, SLES 15 SP2, Ubuntu 18.04 LTS and 20.04 LTS.

[cosmik]

Thanks Mildur for all your clear and detailed pieces of information!

[cosmik]

Coming back to this thread, I must say that preliminary tests look fine. I'm trying to familiarize myself with the (too many) aspects of Veeam, focusing solely on my main site for the time being (left part of the figure at the OP). So far I have implemented a prototype all in one Windows 10 VM, as well as a Debian 11 VM with fastclone support as a hardened repository connected to the main site synology. Since I do not have any faster-than-1G connections, I was also testing multipathing on the Debian. Which looks good on the prototype, but will not be used on the final setup: as Fabian noted, can't have a hardened backup repo with immutable backups as a VM... So I'll spin a physical system for the purpose, losing unfortunately the twin/multipath'ed connections to the Synology LUN...

I am most impressed with XFS as well: the savings seem huge! Will probably implement the backup proxy as a Linux hotadd VM as well, I hope it won't be too difficult to setup. Got some more questions for whoever would have the time to address:

1) I want to slowly bring to play the branch site. If I understand correctly, I'll need a physical machine on that side as well to build the backup repo over there. To which I'll spin two LUNs: one to keep backups from that side and another to keep the backups from the main site. So I'll need a proxy VM over there to take care of the branch host backups, correct?

2) What else might I need to install at that side? A wan accelerator?

3) Which backup option should I use to send backups taken from main site hosts to the branch repo, a copy backup job?

4) If I didn't have the constraint to have immutable backups, I'd simply have a combined proxy/repo at each site: the combination of proxy/repo would eliminate the bandwidth-cost of transferring data from proxy to repo and I could always allocate 2 or even 4 network ports on the iscsi vlan-side of the network to talk to the synology at speeds ranging from to 2G to 4G (due to iscsi multipathing).

Enter reality though and now I have a pc to do the repo role, keeping its embedded nic to talk to the proxy and adding 1-2 GbE NICs to talk to the synology on the storage vlan (via multipathing). While possibly good for doing the synthetic backups (which stress only the repo and not the proxy I think) the proxy <-> backup repo will suffer. Any idea on what I could do here to mitigate the lack of ample bandwidth?