Are Veeam Servers "aware" of each other?

[smartsys]

I am in a discussion with some collegues about the non-existence of awareness between Veeam Servers and how much complexity this adds to a backup design when using multiple Veeam Servers in one network.

For Example:
I have two sites with 3 Hyper-V hypervisors on each location and I configure one Veeam Server for each site.
There is a limited bandwidth VPN (100Mbps max) between these locations. Production needs at least 50Mbps.

100% bandwidth usage will cause production issues, so I want to limit the bandwidth used by Veeam.
Backup Storage is low-cost non-high performance, so I want to limit the maximum number of jobs running on a repository to 2.
Proxies are on-host and we don't want backup te cripple performance of the host, so we limit the maximum number of jobs running on a proxy to 2.

As far as I know the Veeam Servers are not aware of each other.
So if I limit the bandwidth on server A to 50Mbps and on server B to 50Mbps, the end result will be 100% bandwidth usage.
So I have to limit server A to 25% and server B to 25% to leave 50Mbps for normal production traffic.
While this works, I don't use the available bandwidth very efficient.
Jobs from site A to site B take 4 hours to complete and vice versa 1 hour. After 1 hour backups from site B to A are finished, but Server A isn't aware of this so the bandwidth is still limited to 25Mbps while it could have been 50Mbps from the moment site B is finished using the WAN link. Duration of the jobs is thus longer than necessary.

Same is for Proxies and Repositories.
Server A isn't aware that Server B is already using a proxy or repository and will trigger the backup although the repository is limited to max 2 jobs to prevent storage overload. The proxy or repository is now running 4 jobs, consuming resources of production VM's on the Hypervisor and trashing the storage to a grinding hold, causing the backup storage to become the bottleneck and causing the backups jobs to take 3 times longer than allowed.

In this situation one has to micro manage all schedules to mis-match then as much as possible to prevent simultaneous use of the WAN link, proxies and repositories or limit everything way more than required and not efficiently using available resources.

Is my thinking of multiple Veeam Servers and "awareness" correct?

[foggy]

Hi Jeroen, yes, your understanding is correct and I would recommend going with a single Veeam B&R instance in this case.

[smartsys]

Thank you for this answer.

The single server solution is what I am recommending for some time, but there is a difference in understanding between myself and collegues about this subject.
I only use a second server on a remote site when we use replication and let this host manage all replication tasks. This way I can allways use Veeam to manage failover and failbacks if required.
It must also be possible to replicatie the Veeam Server itself to the remote location and manually do a failover for this VM, but this is more complex and difficult.

[foggy]

I only use a second server on a remote site when we use replication and let this host manage all replication tasks.

This is a valid approach, but in your case (bandwidth-wise) you should decide what is more reasonable.

[smartsys]

It would be a nice feature if Veaam Servers can "share" information so that the usage of resources of proxies, wan accelerators, network connections and repositories are known by all Veeam Servers within a network.

I think this information doesn't have to be distributed real-time and on a continues base. I think that Veeam Servers should share information about what server will act as the "master" or "slave" of a particular component. Settings of components are shared between all linked Veeam Servers, so you only have to configure components once on a single Veeam Server.
As as soon as a job starts a Veeam Server checks to see what Veeam Server is the master for a particular component (proxy, repository, data mover, etc...) and will gather information about the state of this component from the master. The master will continue to control the component, so if throttling of a data stream is required the master will manage this.
Is a slave server cannot reach a master it will negotiate for mastership with other slaves and one will become the new manager. This way the infrastructure will remain available and fully managed.

[Gostev]

What is the ultimate goal though, is it making sure that the particular component is not overloaded? If yes, then there are much easier ways to achieve this (comparing to building this type of a complex distributed scheduling system).

[smartsys]

Hello Gostev,

Great to see you respond to my post!

The ultimate goal is to create a redundant backup solution!

In this particular example two companies share the same infrastructure and AD domain, but both have their own building a few miles apart with a VPN connection in between.
A demand/wish from both companies was that they could manage their own backups and they asked our company to come up with a solution.

The first design had two Veeam Servers with WAN accelerators and shared repositories and data movers.
My colleagues were instructed to design two build this backup solutions and then came to me with lots of questions.

My initial thought was: There is ONE network, so there must be ONE backup server.
Then more information came available and is seems the customer demands that local backups should be independent of the VPN connection? This is not possible with one Veeam Server! Hmmmm….Back to the drawing board, or not???
The more I thought about it, the more this really made no sense to me, because if the VPN is down, a lot of mission critical functionalities fail as well. Exchange is located in the primary location, all the FSMO roles of the AD are in the primary location, some licensing servers are in the primary location, application servers are in the primary location. So if the VPN is down, some things continue to work, but lots of others will fail. Is the loss of backup functionality really the biggest concern at this moment?
An other point was when you create two backup solutions in one network and share all components like proxies, data movers, repositories and wan accelerators there is no sharing of usage information. So there is no "respect" of limitation rules! To avoid this you have to half the limits and thus waste potential resources or scheduling things so they do not conflict and this can be a real pain in a place where the sun doesn't shine.
Both are two reasons to simple use one backup server. If the VPN is down there are much more bigger issues than backups not running on the secondary location, but I am an Engineer and NO or IMPOSSIBLE is not always the definitive answer I like to give, so this is where the thought of a "redundant" Veeam Solution came from.

If Veeam Servers are able to share usage information or simply make one server the master, the configured limits are respected by all other Veeam Servers and scheduling can be arranged locally by a specific Veeam Server. If connection is lost, scheduling will remain available, local repositories remain available and only remote components are offline and jobs that require these components will fail. Everything not depending on these remote components will continue to function and if communications are restored the other jobs will start working just as before.

[billcouper]

Perhaps an overview of your design would help?
You have two sites that have production servers, and both are backing up across the VPN? How then, do they both end up on the same storage to conflict with each other? And how are you sharing proxys if they are in different sites?
I'm confused what your design looks like and how you are sharing repositorys/proxys but still consuming VPN in both directions during a backup.
Also, the proxy should have 3 vCPU and 2.2GB memory to process 2 concurrent tasks - how does this consume resources of production VM's?

[smartsys]

Site A:
3 Hyper-V hosts (Windows Server 2016) with shared storage (HPE MSA2052)
1 physical backup server (Windows Server 2016) with one Raid set for backup data (Raid 50, 10K disks)

Sita B:
1 Hyper-V host (Windows Server 2016)
1 physical backup server (Windows 2016) with one Raid set for backup data (Raid 50, 10K disks)

At first there was the idea of using an off-host proxy at site A. This caused numerous problems so we changed this to three proxies on all three Hyper-V hosts.
The Hyper-V host at Site B will also function as a proxy.
During backup resources on the Hyper-V servers are consumed by backup functions and are not available to production VMs.

The Veeam Server will be a virtual machine in the latest design, running on the Hyper-V cluster at Site A.
The physical backup server at Site A will function as WAN Accelerator and Repository for A the local backup of Site A and B the remote copy of site B.
The physical backup server at Site B wil function as WAN Accelerator and Repository for A the local backup of Site B and B the remote copy of site A.

The shared components in this configuration are data movers on the repository servers, the WAN Accelerators and the WAN link.

Now because we only use one Veeam Server. We can limit the usage of the repositories, use bandwidth throttling on the WAN link and limit the usage of the WAN accelerators to 1 job at the time.
Because all backup traffic and WAN accelerator IO are performed on the same disk array and the CPU and RAM is also shared, we don't want to "overload" this storage and make jobs take forever to complete.

If we would use two backup Servers from the original plan it would be much more difficult to limit the usage of the WAN link and the repositories and I don't know for the WAN accelerator wat happens if jobs are sending data cross sites at the same time? So it must act as a source and target WAN Accelerator at the same time. Now it is much easier to schedule to prevent this.

[ddayton]

One other thought to through into the ring is use 2 servers and let the network guru's throttle the max WAN bandwidth with QOS settings.

[foggy]

If we would use two backup Servers from the original plan it would be much more difficult to limit the usage of the WAN link and the repositories and I don't know for the WAN accelerator wat happens if jobs are sending data cross sites at the same time? So it must act as a source and target WAN Accelerator at the same time. Now it is much easier to schedule to prevent this.

A single WAN accelerator can act both as a source and a target one and a single server should be able to act as WAN accelerator for two different backup servers. There's a limitation though that a source WAN accelerator can only process one task at a time, other tasks will queue.