Scaling SureBackup to test many sites?

[John.Akemann]

Is anyone using SureBackup to test backups for many sites?
I have 25+ sites, all managed from a central management instance running out of my DR site. Each site has local components – proxies, gateways, immutable repositories, etc. All sites have copy jobs that copy to the central DR site. With that in mind, I’m struggling to engineer around the need to have the SureBackup Virtual Lab appliance live at the management server site, while also having to test the source jobs as testing copy jobs is not an option.
One path forward that I see is breaking each site off into its own separate local management server to pair with a local SureBackup Virtual Lab appliance.
I’m not concerned about managing 25 management servers as enterprise manager is able to do that part just fine.
I am concerned about updating 25 management servers, because I can’t justify having 25 separate immutable repository servers in order to have a shared-none environment at my DR site. That means having multiple management servers managing their own repositories on shared repository servers must be an accepted reality. I’ve dealt with multiple management servers sharing components in the past, after you update the first management server, it can’t use the components in the DR site until it pushes software updates to it. But that then breaks the ability of any other management server to use those shared components until they too are updated. Now I’m looking at coordinating the update of 25 management servers in a single 4-hour window when the backup environment is idle?
One idea that has come up is using service provider console to manage pushing software updates – a feature that enterprise console is lacking.
In total, all of this feels like an incredible amount of complexity to overcome the limitations of SureBackup. Has anyone come up with a different way to make this work that doesn’t involve the “routing rules should be added in the production network” suggestion that would get me laughed out of a meeting with my enterprise networking team?
The only other path forward I see might be to create replication jobs in the DR site that keeps a rehydrated copy of the VMs in the DR site and uses the copy job in the DR site as it's source. I could then test that "replication job" in the DR site, provided that is a supported configuration. It also means I have to allocate SAN space for every VM, including non-prod VMs that I might or might not need to recover, but must be tested.

Thanks in advance for any thoughts and insights,

John Akemann

[Andreas Neufert]

Hi John,

let´s start by the SureBackup issue and then go to the other things you mentioned.

You can have the Backup server central and still use the Virtual Labs on the site with a little trick on your routers setting some manual routes.
Let me explain:

I guess you can connect from VBR to the Virtual Lab Appliance and back by regular routing (you have set the VBR default route (Gateway) and the Virtual Lab Appliance (Gateway in the virtual lab wizard where you set the virtual lab appliance IP).
So you can ping from both sides each other.
In order to ensure that VBR finds the route to the Masquerade IP subnet, you need to set on the router from the VBR servers Gateway IP the following route:
Masquerade Subent + Subnet Mask => Virtual Lab IP

Then you can let surebackup run locally on the different subnets.

Did you tried this?

[John.Akemann]

Hi Andreas,

The first time I ran the idea of "a little trick on our routers" past our enterprise network architect, the idea was dismissed as a non-starter.

But after reviewing my alternative options, I proposed it once again and will have talks with both our network and security architects on Monday.

They may be open to the idea of limited routing of these masquerading subnets between just our backups VLAN in our DR site (where Veeam Management resides) and our compute sites where the virtual labs reside, possibly in their own segmented VLANs. One challenge I foresee is I'm going to need to get buy-in from both our security architect/team and our networking architect/team. These "tricks" need to be added to the firewalls at each source and destination site so the traffic is allowed to traverse those firewalls at the edge of the security zones. Additionally, the networking team needs add these "tricks" to the data center fabrics, core routers, and WAN routers.

This would be so much easier if the testing of the jobs could be done by the local VBR proxy at each site, or baring that, if the masquerade traffic between the management server and the virtual labs was encapsulated in a point-to-point tunnel, which would also support multiple sites with no need to get enterprise network and security on-board with implementing "routing tricks"

I'll let you know if my fellow architects get onboard with "routing tricks" or not after our meeting on Monday.

[Andreas Neufert]

Yes I agree, but for now the technology is that way.

This is not a security issue to set these routes.
I agree it is a bit of a complex topic to discuss and understand and security/network people are just too afraid of what they do not fully understand.

[Andreas Neufert]

I think you only need to set the route in one directon.

[John.Akemann]

Unfortunately, making the masquerading subnets routable is unworkable in an enterprise our size. In order for a packet to get between the management server and a virtual lab in another site, that packet is going to take 8+ hops as it traverses server switches, firewalls, core switches, WAN appliances on both ends. Our security team has concerns about routing this traffic without also setting up default to deny rules for it elsewhere to guarantee there's no leakage into production. If we add a single summary route per site that can save on effort, but it will also potentially waste more IP space - something we already accept with production vlans - but now we're doubling the summary allocations and doubling the potential waste. Then that all needs to be replicated (and if doing summary routes - appropriately sized for current plus future growth) for each additional site - 25 in all. I'm sorry, but this is not a scalable way to do basic backup job validation in a medium to large enterprise!

[Andreas Neufert]

I suggest to reach out to our field system engineer team in your country. They can connect you with a field solution architect that can help you with the design and choosing the right strategy and architecture. This is something that need a direct discussion and would be too complex to discuss here.

There are many ways to address the concerns you raised. We used in other areas IP ranges of public IPs to not waste private IP ranges for the labs. As well you could place the virutal lab in a dedicated vlan and use VPN to let the backup server be part of that vlan to not have to mess with routing. There are additional options available please discuss directly with the solutions architect.

In general it really depends as well on the verification level. SureBackup is a unique technology and yes, it needs a bit of a complex network setup, but the technology is unique in what it can detect that can prevent restore that is rooted in your environment outside of Veeam. There is as well the option to not use the network and application testing and just let the VMs boot there and detect that the VMware Tools are started within the VM (that the OS just booted).

[John.Akemann]

Thanks Andreas. I am also working with my account team in parallel - just trying to evaluate all avenues to gain as much insight as I can.