Leveraging multiple VBO Proxies

[snom]

Opened case ID 04765391 for this -

We're seeing slow performance on our VBO backups which is being investigated separately.

Core questions:

• Can you use multiple Backup Proxies for a single organization or job?

• Can you change the Backup Proxy being used for a Backup Job?

• Can you change the Backup Proxy being used for a Backup/Object Repository?

• Can you permanently or temporarily move the proxy being used for existing backup jobs?

We'd like to leverage an offsite proxy with a faster connection to speed up our initial backup - I am under the impression you should be able to use multiple proxies to backup. What isn't clear is how to actually utilize the proxies. I have already connected an offsite proxy with more bandwidth.

Despite the documentation or lack thereof, it looks like the VBO interface implies a Backup Job can only be connected to one Backup Proxy and one Repository/Object Storage at a time. It also doesn't seem to be possible to change which Backup Proxy is used for each job, or each repository.

Can anyone offer guidance on the best way to handle this situation?

[nielsengelen]

Which slow performance issues are you facing? Could you tell us what the bottleneck is and a bit more insight in your current setup (on-premises server, storage used for repository, sizing for components,...)?

On your core questions;
A proxy is dedicated per job. VBO works differently in this way compared to Veeam Backup & Replication and a proxy and repository are seen as 1 within a backup job. You can however create multiple repositories (by creating different folders) and use 1 proxy with multiple repositories. For example:

• c:\veeam\repo01

• c:\veeam\repo02

You are correct that you can't use multiple proxies per job but you can use multiple proxies in an organization.

This is also listed in the user guide (see https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=50).

• Each backup proxy server can process one or several organizations.

• An organization can be processed by one or several backup proxies.

You can achieve this by having multiple proxies/repositories and per job assign one to it, for example

• Job 1 with Exchange data goes to REPOSITORY01 (proxy01)

• Job 2 with OneDrive data goes to REPOSITORY02 (proxy02)

• Job 3 with SharePoint data goes to REPOSITORY03 (proxy03)

• Job 4 with Teams data goes to REPOSITORY04 (could be stored on the same server as REPOSITORY01 and we can re-use proxy01)

We have a best practice guide for VBO available at https://bp.veeam.com/vbo where you may find additional information for extra questions that may arise.

[JRRW]

I'll take a stab at answering some of this in real world background:

• Can you use multiple Backup Proxies for a single organization or job?
  Yes, for a single Org but not for a single job
• Can you change the Backup Proxy being used for a Backup Job?
  Yes - with certain understanding of how that migration/change occurs
• Can you change the Backup Proxy being used for a Backup/Object Repository?
  Again yes, with understanding of connectivity changes
• Can you permanently or temporarily move the proxy being used for existing backup jobs?
  Yes, but again there are some understandings to take into consideration

Here's a real world example I've used:

1. Backups started on one server using one proxy for one org - but multiple backup jobs - with 4 repositories all off one proxy
2. Migrated the repositories into two proxies: one for sharepoint/onedrive one for Exchange/mailboxes
3. Updated the jobs to use the new proxies/repositories - THIS is where a 'gotya' occurs as it has to do a 'sync' on the data - this isn't a full download but it is very I/O intensive (not as much bandwidth I/O but on the storage system)
4. While troubleshooting why things were taking a while we created a new job for data we hadn't yet backed up off site on another proxy that had connectivity to the control server - and then moved just the folder structure over to an existing proxy where it still had to do the resync but not re-download

Same should in theory occur for the S3/obj stor - however that's a bit harder to identify as it's not officially supported to migrate object storage, from what I've read. Now if that ObjStor is say a VM and you just move the VM.... It shouldn't care, and I've done this without issue (obviously ensuring that DNS is accurate)

In THEORY if you CLONE an obj stor it also shouldn't - i've not tested this however.

[snom]

nielsengelen -

We are not 100% sure where the bottleneck is but I have a strong suspicion it is O365 throttling. I reduced our threads to 10 this morning and was seeing some promise but it has gone back to averaging 4-10Mbps.

VBO is running on a shared server with VBR but VBR is not currently in production or being used for active jobs. 6vCPU, 16GB RAM (11 used). It is on-prem downloading from O365 straight to Backblaze B2/S3. We expect our total backup will be about 4TB when all is said and done.

[snom]

[*]While troubleshooting why things were taking a while we created a new job for data we hadn't yet backed up off site on another proxy that had connectivity to the control server - and then moved just the folder structure over to an existing proxy where it still had to do the resync but not re-download[/list]

This is basically what I want to do and I am not clear on the correct way to do it.

I want to do the initial backup on an offsite proxy, and then run the same backups from the onsite proxy. It would be the same repository, it doesn't need to run at the same time.

In theory, it sounds like I should just connect the offsite proxy to the existing repository and re-create the backup jobs. Then hopefully I could just disconnect the object repository, connect it to the onsite proxy and sync it to resume backups from there.

Is this a supported use case? Is this any different than if my VBO proxy failed and I spun up a new one to resume where it left off?

[nielsengelen]

@snom Could you clarify a bit more about what setup you plan then? Where would your backup repository be located? If you still need to do the initial backup, I don't see much advantage of using a remote proxy to store data locally as it still needs to be transferred to the local backup repo in that case.

[JRRW]

@Nielsengelen my assumption based on their description is sneaker net. Download to Server/NAS/etc off-site with faster download speed and physically move storage back to the 'slow' location.

We didn't even recreate our jobs - literally just changed repositories (thus proxies) from the 'offsite' one to the 'onsite' one once we moved data.

When doing it I'd highly recommend using something like robocopy /mir if you're copying between servers - but the 'best' way is if it's in some form of virtual HDD (VHDX/VMDK) that you just move the entire 'drive' that held the repository and OS - and if you can't move the OS (i.e. if it's on a different domain or don't want to re-IP) you can at least re-attach the repository file structure to another proxy, and 'add' the repository via the 'onsite' proxy.

Not sure if it's supported per se, but the software didn't really care, it just had to do the resync. Again, that's I/O (disk) intensive but not as intensive on bandwidth.

On the topic of bandwidth: one thing I did notice is that sharepoint/onedrive seems to use more upload than download when doing a lot of work; when running on a cable connection it maxed the 10Mb upload pretty consistently when the jobs would start off.

[snom]

Ok, I now realize I should have put more emphasis on this -

I am backing up to a cloud based S3 object repository (specifically Backblaze). I found what I needed to understand in another old thread - that I can use a remote offsite proxy to do the initial seed of the S3 repository and then move/reconnect that repository back to the onsite Veeam proxy later, eliminating the remote proxy. The only annoyance is that I'll need to re-create the backup jobs, but not the data backing them.

I wish this information was clearer in the actual User Guide for VBO - it would have saved me a lot of time...

veeam-backup-for-office-365-f47/vbo365- ... 68766.html
https://www.veeam.com/kb2649

There are two key reasons for doing this -

1. The temporary remote proxies have no bandwidth caps, while onsite we have to cap at 25Mbps during daytime hours and 150Mbps off hours
2. The temporary remote proxies can be allocated significantly greater resources to help with initial backup.

I now have two remote proxies, one handling mailboxes and one handling SharePoint/OneDrive. Seems to be doing a bit better with 8 cores, 32GB RAM each. They also have 1Gbps fiber uplinks. Using 20 backup accounts on mixed modern/legacy. So at least there is probably no limitation on my end.

The upload is still going a bit slower than I hoped but it is doing better than the onsite performance I was seeing before. Getting about 60Mbps overall. I passed the newest logs to support (Case #04761017) to see if there is anything else I can do.