Sure Backup On Backup Copy Jobs Query

[bjc3000]

Hi Team,

Working for a fairly large MSP servicing multiple Clients with Veeam cloud connect, and our off-site repositories for backup copy jobs. We were looking into configuring a sure backup server that exists only to check our backup copies on the cloud repository.

I understand that best practice is be to verify before offloading to the cloud repo, unfortunately for a lot of our clients, especially smaller businesses, they do not have the server resources to service the Sure Backup Requirements. We have attempted on quite a few to reduce start up configuration as much as possible to make it all fit, but it's just not possible for a quite many of our customers.

The plan was to use a few of our HCI Cluster nodes as targets, have out B&R server in our datacenter connected to our repo, and setup All of the VLabs required on those devices. Add all of the imported Backup Jobs to Named Application groups per client. And assign them to the correct VLAB

Our thought process behind this is that if the backup copy is verified, the on-prem must be good. However, this would take all of the load off our customers Host's (our bigger clients are not an issue but our smaller clients do have limited resources) We wish to service everyone equally. This would not mean that we would stop running Surebackup on-prem where the clients Hardware can support it.

I did not realise this was not possible until trying to configure this. I noted that it used to be possible with a workaround, however those commands have been deprecated for some time.

Our secondary reason is that we have a massive high performance storage array with multiple 10gb links to the DC VBR server. With this configuration we would be able to validate every customer backup, every single day, without affecting their Hosts at all.

I'm wondering if there are any methods of accomplishing this, or if this is or will ever be on the roadmap. I like the idea of being able to verify our offsite backups for multiple reasons.

One of our larger clients, who refused a hardened repo got cripto'd and luckily all of our offsite backups were fine, Their on-site repo got dusted as well. I'd be fearful of any transmission corruption etc after the fact causing them to be non-restorable.

I guess my thought process is like an airplane check, The plane's are checked before takeoff (Surebackup on-prem image), and never checked on arrival (when the backup copy job completes), and then they are cleared to takeoff from the next site without checking (Restoring from Backup Copy Job). If anything happened during that flight that affected the planes ability to takeoff, you wouldn't know until it was time to take off again.

I'm aware that we can manually run up the backup copies and that is best practice, and we do. But I could have a Dedicated host server ready to run surebackup on every single backup copy, every single day.
All on our infrastructure, with no issues about hardware provisioning requirements.

[david.domask]

Hi Brendan, welcome to the forums and thank you for the detailed post.

Surebackup for Backup Copies is a known request, but nothing to share on that at this time, though your use case indeed is a common and understandable reason for wanting it.

I would comment on a few points in your post though, as while overall I agree with your position, there are a few things I'd like for you to consider:

> Our thought process behind this is that if the backup copy is verified, the on-prem must be good.

This is a fallacious statement, because remember, SureBackup and other verification checks (Health Check) can only tell you that everything was fine _at the time of the test_. It cannot tell you if bitrot occurred on the source side for example and prevents the restore from fully working or if there is an issue on production side which would make functional restored VMs not work (e.g., consider a network restructuring that took place after the backups ran or a security patch that requires reconfiguring the servers on the prod side, just examples)

So having checks going in to the Provider environment and while inside is ideal; Surebackup/HealthCheck are never "one and done" events, they are continued assurances that the backups are usable.

> I understand that best practice is be to verify before offloading to the cloud repo, unfortunately for a lot of our clients, especially smaller businesses, they do not have the server resources to service the Sure Backup Requirements.

Out of curiosity, what limits them on this? Hypervisor resources? Repository can't handle the workload to complete in a reasonable amount of time? Basically, there are ways around most of the most resource intensive parts of SureBackup (including integrity check), so while it should work out of the box for most environments, those that require additional consideration have options as well.

It's great how you're thinking on this, and agreed that the Feature Request is reasonable. In the meantime, maybe it's worth reviewing those elements that are used as reasons to not use SureBackup on the client side, as there should be ways of handling this.

[bjc3000]

Hi David,

Thank you for getting back to me.

In regards to the on-prem being ok. We run nightly backup copies. Our thought process is that if we have a decent server completely dedicated to running sure backup tests on a daily basis, we can confirm that the backup copies of the previous nights backups are ok, and therefore you could assume that the on-premise copy job should be ok. Regardless, you are assured that your failover backup copy is in restorable condition.

I agree with the sentiment of testing backups on-site and for those that would support it, we would not be removing the sure-backup jobs we have in place, just adding a secondary validation on our side to ensure we have consistent and workable backups.

In terms of the clients who can't support it, we have clients that have 64GB ram servers and maybe hosting 1-3 servers.
In one instance, the servers on-premise are taking up 52Gb ram on the server as we speak. Our typical solution is to ensure we have application groups as their servers require active directory. and in some instances, the VM's do not have enough resources to run correctly, or boot at all. And we also ensure that the Host always has enough Free memory to continue it's operations.

This is unfortunately for a lot of smaller clients quite common. A lot of them are adverse to upgrading their on-prem infrastructure with the mantra "if it isn't broke, don't spend thousands on a new server"

As a business we have come to appreciate what Veeam has given us as a form of Backup Solution we can provide across the gamut of our clients, Being able to Sure backup test our off-site cloud connect backup copy jobs would be the next step in insuring that all our clients have useable backups at any given time, and the automation would assist in bring our testing workload down.

Our repo is more than capable of handling multiple tests at once, and we'd be able to deploy a Server as a single-pane for all of our Sure Backup Tasks.

[david.domask]

Hi Brendan,

You're welcome for the answers, and thank you for elaborating on the limitations for your clients.

https://helpcenter.veeam.com/docs/backu ... 20#startup
https://helpcenter.veeam.com/docs/backu ... 20#startup

Can I ask, aren't the start up options for SureBackup which allow you to configure the VM memory in the Virtual Lab sufficient for this? Since the machines don't need to actually run operations during SureBackup, you just check that it boots correctly and any other verification tests you wish to run, would this help with the resource issues on those clients? Or have you tried these options and that is when the machines couldn't even boot?

Your request on SureBackup from Backup Copy is noted and it's a request that is being tracked, but nothing to share on that front at this time, so wanted to ensure that the Startup Options weren't missed as it sounds like this would help in those resource constrained environments.

[bjc3000]

Hi David,

Yeah we have experimented with the Startup options, but we struggle with the process timing out and the scripts etc not being run.

They work perfectly with other clients that have ample resources, just struggle a bit with the clients that do not have it.

I've managed to throw a solution together that works for us in the interim, and thats new chain backups to the repo.

Sure backup is finicky about the adapters. Unfortunately if you do not have a Virtual Switch on your test host with the exact same name as the host the backup originated from. You are in a bind, Luckily pretty much all of our clients have the same naming conventions across all the hyper-visors so we have no issues there, and the ones that do, I have re-named their V-Switches and the next backups work correctly.

Ideally I'd like to utilize the Backup Copy for this task, But i can get away with Full backups to the repo.
I've got a PowerShell script Re-Scanning the Repo every morning at 7:00 am and then a bunch of V Labs spinning up after that testing the backups

I'm hoping the Backup Copy Sure Backups come into play eventually, however I would assume based on my own findings that you will have issues with Virtual Switch Naming across different Clients / Hosts

This isn't the most eloquent approach, but now I have a single server testing all of the offsite backups for the Clients that can not support it, and It's all running off the same server and Managed by our SPC.

[bjc3000]

It appears that this method has been patched out as of 12.1, I do not understand why this would be the case...

[bjc3000]

So previous to upgrading.

If the Backup was "On Disk" I was able to add it to an application group and from there spin up Sure Backup. As of 12.1.1, I no longer see the On Disk Backups in the selection menu, Can anyone confirm for me if this was actually changed and if so, why?