[Enhancement Request] Set up multiple backup apps DURING org setup, and reuse data later

[AlexHeylin]

I thought I'd created this already, but I can't find it - if this is a duplicate, please merge and accept apologies from a tired person.

There are two related requests here

1. When adding an organisation, prompt for how many backup applications to create. This avoids the need to immediately add more backup apps and reenter all the info that just got entered in adding the organisation in the first place.

In case anyone is going to query the need for additional backup apps, I suggest you read the other threads (which I've contributed to heavily) about what happens with large SharePoint Online lists, and / or high change rates.

2. When adding additional backup apps to an existing organisation, look up the information already in use rather than prompting for it all over again. The certificate for example - should already be known to VBO.

Thanks

Alex

[Mildur]

Hi Alex

While I understand the value of your request, I personally do not think that request will ever make it into the product. The reason is simple. Using multiple apps to download Sharepoint data violates Microsoft's policy and may be deprecated by us in a future version of VB365. If we decide to deprecate that option (because we are not allowed to use it by Microsofts policies), we have to remove every GUI element about additional backup applications instead adding new elements.

Thanks
Fabian

[AlexHeylin]

@Mildur
Thanks for your reply. However I'm shocked by it. Last year Veeam support told us very clearly that using multiple backup apps was the only current mitigation for a number of issues, and that it was specifically put there to enable backups to be completed in (something like) a reasonable time because of the limitations MS put on O365. If it violates MS policy now, did it always violate the policy - or did they change the policy just to mess you up because they realised what you were doing?

Some, but not all, of those issues were made worse by suboptimal design in VBO. Are you saying that all those design issues either have been addressed, or will be addressed very soon?

I suggest Veeam need to have "an honest discussion" with Microsoft about the challenges, and work with them to come up with a workable solution to O365 backup in the future. I think I won't be the only person here that's getting rather bored and frustrated with this cat and mouse scenario with MS and VBO that's been going on for some time and the resulting fallout for customers and anyone running VBO at significant scale. If we're hitting these issues at our relatively small scale, the product must be virtually inoperable for larger organisations, and incredibly frustrating to manage at full SP scale even if the tenants aren't large enough to cause major problems.

Thanks
Alex

[Mildur]

Since the release of VB365 V6.0 in March 2022, we have a statement in our help centre (many applications may affect performance). If you use too many applications at once, Microsoft will start throttling you. I assume that the policies on Microsoft's part changed just before V6 was released. I can not say exactly when.
Microsoft does not technically ban it yet. They write in their own docs page that using too many applications can lead to rate limiting for all API calls to the entire M365 tenant.

Some, but not all, of those issues were made worse by suboptimal design in VBO. Are you saying that all those design issues either have been addressed, or will be addressed very soon?

We are not sitting still. Developing for the cloud means, that you have to adjust your client (backup software) on a regular basis.
Optimizing the performance and adjusting to new rules from Microsoft is done by us with each release.

Thanks
Fabian

[AlexHeylin]

Thanks Fabian.

Can I request that these changes like these need to be MUCH easier for existing customers to find? Perhaps a delta / difference view of the docs would be a good thing?

I've just been told on Case # 05613184 to add additional backup apps to two tenants so it seems Veeam's own staff aren't clear on this change either.
Right now we're being told both to stop using additional backup accounts, and to add more additional backup accounts to resolve throttling for a couple of tenants (which isn't the issue the ticket is about anyway).

We just want it to work reliably, and for Veeam to give clear guidance on how to make that happen so we don't have to have it keep stalling / grinding to a halt, throttling (avoidably), or having to spend hours on support cases.

Thanks

Alex

[Polina]

Hi Alex,

First, to clarify, there're no issues with adding extra backup accounts (for organizations using basic authentication). The new recommendations apply to modern app-only auth and additional applications. In this case, it's highly recommended to use a single app, but a few more apps can still help in reducing the 429 errors. In larger tenants, this could be 3-4 apps - normally this number shouldn't impact the production environment. So far there were just a couple of cases when we saw a conflict caused by additional applications, and these were tens and hundreds of apps with a simultaneous temporary turning off throttling on the M365 side. This caused a high load of requests to SharePoint with disabled native protecting mechanisms.

As Fabian said, in the future, we consider deprecating the auxiliary apps feature (and when it comes we'll make sure to announce it properly not to cause any issues to our customers). And at the same time, we're working with Microsoft to optimize the usage of Graph APIs in our code to achieve the best backup performance with just a single app.

Makes sense?

Thanks

[AlexHeylin]

Hi Polina,

As only modern auth is now "supported" (not deprecated), we're only discussing backup apps not users or mixed mode.

After querying this and pointing to this thread, support confirmed that their current recommendation is 6-8 apps per tenant (though I'd imagine that depends on the size / complexity of the tenant).

I suspect this may be a matter of timing. If you're going to have a working "single app" solution in 3-12 months, then please ignore this request. If it's going to be 12+ months, please consider this request as standing because until there's a single app solution in place - this continues to be a small, but irritating pain point.

Thanks

Alex

[Polina]

Trust me, I prefer to see happy customers and get new nice feature requests instead of fixing issues. And given that backup performance is the basis, this improvement is taken very seriously.

Thanks!