Almost giving up.

[staun]

We have been using O365 backup for about 1½ year by now and have never been really satisfied with it as we are with VB Console&Replication, so we are about to give up on the product.
Why.. I don't recall we had a fully succesful job completed. The jobs keep running for weeks sometimes, which mean an uncompleted job cannot be restored...

A few examples of our jobs.
OneDrive status. Low read/ write transfers, I have never seen it above 1-2 MB/s (this goes for all four O365 jobs we have)
We have tried the solutions with several application-accounts in office 365, but didn't change much.
https://ibb.co/nLH7tXv

Status of each OneDrive account - Duration is between 14-xxx hours, Some OneDrive accounts never complete.
https://ibb.co/qyysfyq

Status of Sharepoint job - the quite known error I see many others have, "Already processed by another job"
https://ibb.co/MfhSZGF

License error on people that are resigned. - we used to be able to use powershell to delete the personal site, but the error still occurs for some strange reason.
https://ibb.co/dcg5mpV

Restore mailbox - Unable to select any restore points as the job have NEVER finished. (we have one in current month, but this date is due to we stopped the job due to maintenance)
https://ibb.co/88d4m4K

As it is right now after more than a year, I am still not able to say.. We have full backup of our O365 solution, we can restore anything we want...

We are using a Service provider for our O365 backup and running version. 6.0.0.379. We have contacted their support and it usually ends up with - That is how the application works. Microsoft set throttling etc....

Are others seeing similar or is just our setup that is completely wrong.

[Polina]

Hi Jørgen,

First, thank you for bringing this feedback to us.
To be honest, I'm pretty sure that many of your issues can be fixed with a few tweaks to the current setup and upgrading the product to the latest version.
But if I understood it correctly, you're working with a service provider, which probably means that you cannot open a support case with Veeam directly. Is that correct?

[staun]

Correct. We cannot make any changes our self.

[Polina]

Please DM me and we'll figure out together how to resolve the situation.

[hunterisageek]

We have a very similar outcome.

We have ~8k used. We use dynamic groups to lessen the qty of users per job and we split our jobs by sending domain.

We have 2 large jobs with a majority of the company using 2 sending domains.

Our email jobs finished after a few months, but the OneDrive jobs have not actually finished in a year and a half. There is ~2500 users in one job and ~3000 in the other. About a year ago we ended up splitting them into separate jobs. One for just Email and another for just the OneDrive job.


We are running an N-1 version todate. The issue is every time we update or reboot or restart the job. It appears to not pickup where it left off and basically never end up finishing.

We are not running service provider version. Iv opened a couple tickets over the past couple years or so but with little help of tweaking things to atleast try and finish things.

For the smaller jobs with 500 users or less. It’s awesome. Works great. Seems to run into issues with the larger jobs.

[Butha]

Hi @staun - seems similar to our experience - almost 3 years (and 3 versions) and I have never seen fast performance or successful runs in that time - until around a week ago. I see you are working with a service provider, so slightly different to us who manage it ourselves - but I'll post the solution for the performance portion at least. Some of the "reruns/warnings" are valid, has to do with anything from no owners, sharepoint sites that have been removed etc - and the lastest patch available did resolve one last warning/rerun issue with sharepoint sites.

I'm not sure if it's a commonly known - almost like it should be a "Step 3" in any new O365 deployment - but I remember mentioning to the engineer that in 3 years it was never mentioned or highlighted to me - with many calls around performance issues. Also unsure how long it's existed as an option or if it's only since V6 but I think it's a simple configuration/deployment option that's existed for a while already. No idea why I've never heard about it - maybe because 3 verions ago it wasn't available and since upgrades it was never thought of or highlighted.

Your authentication method determines slightly different procedure - but the cause and resolution was the same for us. When the last engineer (Thanks @George Duimovici) had a look at another issue - he mentioned he saw a lot of "throttling - waiting for 60 seconds" in the logs - multiply that by hundreds or thousands of onedrive accounts - you are talking days of runtime. The solution is to simply add more "backup applications" - the suggestion for our use case was to add 10 - maybe with you it might be a lot more - but ask support to recommend. To me this is similar to adding more proxies - as it allows many more parallel runs.

The link that explains the process - but note this is for when using Oauth with a certain config (we dont' use the new API option for Advanced MS Teams backups - just the "standard" MS Teams).

https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=60

The documentation is not very clear - but basically you follow it to the point of adding - but don't look or select " existing ones" - just add NEW ones - giving it a name - selecting 10x quantity (or whatever amount you want to try) - call it anything you want, it will re-use the same name for all instances - following the OAuth prompt (opens that little popup page with Azure - copy/paste the link/code) and let it configure what it needs automatically in the background. It almost seemed too simple - but made a massive difference in throughput.

Job runtime came down from 15+ hours to under 2, sometimes quicker.

Since then, I've had mostly "Green- successful" runs - (Never seen in the previous 3 years) - if the solution of multiple backup applications is already in place your side, then I hope you find a solution - but perhaps ask the engineers to look at logs for throttling - that might point you in the right direction.

[micoolpaul]

Morning!

I don’t want this to be taken as dismissive of the issues in this topic, because these issues are clearly real and providing a serious impact to the recoverability of your organisations.

I did however want to state that the application itself does scale well, I’ve deployed it to customer bases of 20k+ users.

I feel for you @staun that your service provider, isn’t providing an acceptable service. Hopefully Polina can provide a way forward on this

[infused]

Run it in azure on blobs. It's very easy to do

[robnicholsonmalt]

This is a very important post as we're in a similar situation. We're hosting with iLand and I've never seen so many system problems. We get "problem" emails from them several times a week. Whenever we come to restore anything (which is fortunately rare), the VEX clients never work. Usually because they've been updated without our knowledge. I know this is a communication problem from iLand but it still reflects badly on Veeam.

I'm also aware that the interface to Microsoft 365 is horrible. It's pretty dire when we're trying to write our own scripts to do things like check unique sharing of documents within document libraries. The throttling is a well know core problem.

Whenever I'm asked to survey "How are we doing" from Microsoft, I always mention poor backup options.

But given that's the lay of the land, some chewing the cud over how Veeam M365 works would be helpful. I've got a sketchy script that walks through every M365 group (over 2,500 of them) and carries out some checks. I've had to take the approach that the API/Graph/MgGraph/whatever calls will fail and add them to a queue to try again later. If it fails, then it's put to sleep for a while and tries again. Only if it fails a certain number of times (days) do I email an error report.

I get the impression that Veeam just tries to do "For Each Group, do stuff".

[robnicholsonmalt]

And yes, we're looking at hosting ourselves so we're in total control AND looking at alternatives. No problem with managing a server (Windows or Linux) but the cost of storage and the underlying architectural problems of Veeam make us wary.

Microsoft - implement a MUCH better backup API
Veeam - reflect on the core architecture and maybe do it differently

BTW - the VEX SharePoint client when restoring is still HORRIBLE.

[robnicholsonmalt]

Yet another email just landed in my inbox. I always see "Status running", never "Status complete". Yes, we're tried multiple accounts, splitting jobs etc.

[Mike Resseler]

@robnicholsonmalt

Did you already contacted your service provider? (I see you are running this with one of our VCSP's?

[dotdk]

Hi @Staun, it must be frustrating seeing your backups not completing and can nothing about it yourself. I just wanna add that it is possible to get it to work, but in some cases it requires some effort both from the customer and the service provider to reach your common goal.

I have been working with O365 backup since version 3 at a Danish service Provider, and I just want to add, that we do have customers with 10.000+ users, and we are getting daily backups.

One customer even have more than 20.000 users and more than 1.6 PB data, and we run daily backups of every user mailbox. It took some trial and error to get it all up and running, and we had to finetune with Veeam to find the sweet spot for number of proxies and backup applications. Due to the sheer amount of number of Teams and SharePoint sites of this customer these are done on a weekly basis, but we do get backups through. Our biggest problem is the throttling we are facing from MS, but we have managed to overcome this to an acceptable level, working alongside Veeam and finetuning the larger environments we have.

On the technical side we are using object storage as repositories, and depending on the customer, we split up jobs where it makes sense to split them, but in the end, it really comes down to knowing your customers O365 environment and align your backup environment to fit your customer. Most of our backup customers can fit in a standard 1 server solution, but some requires multiple proxies. Largest one of our customer needs 11 proxies to handle the load.

We are on the latest version 6.1.0.423

[unsichtbarre]

Staun,

My company is a service provider and we have a successful implementation for a very large company. I can tell you that the product does work as advertised. It is not always easy to overcome some of the limitations and permissions, however, and our internal MSFT team, along with Veeam excellent support has helped us to our client's success!

First off - why doesn't your MSP grant you access to your own solution? Investigate a new MSP that will take the challenges seriously (and give you access if/when requested), maybe you will find my company in the list of Veeam partners/VCSP's!
Second, don't run o365 backup on Azure, that's mostly self-defeating - even if it might work.
Thirdly, break your o365 backup jobs out (current, archive, etc) as much as possible
Fourthly, you will need to investigate lifting outbound traffic limitations for a period while the data seeds.

Veeam backup for o365 is not a perfect product, but it works as advertised and it is the best/only way to protect your own o365 data.

[k00laid]

This is a very important post as we're in a similar situation. We're hosting with iLand and I've never seen so many system problems. We get "problem" emails from them several times a week. Whenever we come to restore anything (which is fortunately rare), the VEX clients never work. Usually because they've been updated without our knowledge. I know this is a communication problem from iLand but it still reflects badly on Veeam.

I'm also aware that the interface to Microsoft 365 is horrible. It's pretty dire when we're trying to write our own scripts to do things like check unique sharing of documents within document libraries. The throttling is a well know core problem.

Whenever I'm asked to survey "How are we doing" from Microsoft, I always mention poor backup options.

But given that's the lay of the land, some chewing the cud over how Veeam M365 works would be helpful. I've got a sketchy script that walks through every M365 group (over 2,500 of them) and carries out some checks. I've had to take the approach that the API/Graph/MgGraph/whatever calls will fail and add them to a queue to try again later. If it fails, then it's put to sleep for a while and tries again. Only if it fails a certain number of times (days) do I email an error report.

I get the impression that Veeam just tries to do "For Each Group, do stuff".

Hi Rob,

This is Jim Jones, I work in the product group here at iland/11:11. We for sure are upgrading our environments to VB365 6a right now but you should have recieved an email or 3 from us letting you know a) your environment was being upgraded and on which date and time, b) that the VEX applications would need to be upgraded for compatibility and c) where to download them from. Did you receive these?

[robnicholsonmalt]

@robnicholsonmalt

Did you already contacted your service provider? (I see you are running this with one of our VCSP's?

Ohh many times.

[robnicholsonmalt]

Hi Rob,

This is Jim Jones, I work in the product group here at iland/11:11. We for sure are upgrading our environments to VB365 6a right now but you should have recieved an email or 3 from us letting you know a) your environment was being upgraded and on which date and time, b) that the VEX applications would need to be upgraded for compatibility and c) where to download them from. Did you receive these?

No, despite each time I raise a support ticket asking whether there is a mailing list that I could be added to.

[Mike Resseler]

All,

For those with issues in this thread, please send your case numbers to me in PM (not to overload this thread) so I can add those for review with the correct RnD teams.

Many thanks
Mike

[robnicholsonmalt]

>but in some cases it requires some effort both from the customer and the service provider to reach your common goal.

Due to the (poor) nature of the interface to M365, this is probably the case. I wonder what the relationship is like between Microsoft, the service providers and Veeam. I would imagine the service provider to Veeam relationship is pretty good as Veeam are getting £££ in licenses and support fees. With Microsoft? I could imagine that's less happy.

>I have been working with O365 backup since version 3 at a Danish service Provider, and I just want to add, that we do have customers with 10.000+ users, and we are getting daily backups.

At first I read that I thought, wow - we're moving to you but then I read that you only do SharePoint once a week and went "Ahh". Whilst email is important, for use SharePoint is at the top of the list. If we lost a week's email - annoying but company would still keep going. Loose a week of SharePoint documents and that would hurt.

>Our biggest problem is the throttling we are facing from MS, but we have managed to overcome this to an acceptable level, working alongside Veeam and finetuning the larger environments we have.

This kind of backs up my comment about relationship with Microsoft not been too good. I guess all we can do is apply pressure here both from us as clients and service providers. Until Microsoft have their own viable backup solution covering all bases, they must let us back up our data.

Like most, we backup our data for three main reasons: 1) accidental deletions/changes outside the recycle bin windows of SharePoint/OneDrive & Exchange, 2) recovery from ransomware and 3) compliance, i.e. keep historical backup for a number of years. Veeam is okay for #1 (although the VEX interface in SharePoint is horrible) and #3 as we'd be cherry picking data for restore. For #2 is nigh on useless but fortunately, the ability to restore a document library to a previous point in time in M365 itself is what we'd use first, not Veeam. But what happen if this M365 instant restore was compromised? Certainly possible - and Microsoft couldn't restore it? We'd have to fall back on Veeam but restoring the data would take MONTHS with the current interfaces.

So I'd like to see some discussion around how we could plan for a DR recovery? Basically, we'd need to triage the restore. We'd need additional metadata somewhere saying that in case of disaster, recovery these document libraries first. Of course, we'd have to manage our 2,500+ document libraries to supply the importance information.

I'm wandering off topic a little here, maybe a different thread on disaster recovery processes is needed?

>it really comes down to knowing your customers O365 environment and align your backup environment to fit your customer. Most of our backup customers can fit in a standard 1 server solution, but some requires multiple proxies. Largest one of our customer needs 11 proxies to handle the load.

If the iLand rep is still listening, I'll reiterate that our relationship with you could be better.

[dotdk]

"At first I read that I thought, wow - we're moving to you but then I read that you only do SharePoint once a week and went "Ahh". Whilst email is important, for use SharePoint is at the top of the list. If we lost a week's email - annoying but company would still keep going. Loose a week of SharePoint documents and that would hurt."

The reason for this is the amount of SharePoint objects for this particular customer which is 89706 (last backup) and just the enumeration of the objects alone takes 3-4 days, and then 10-15 TB of data pr incremental run.
Personal sites are included in the daily backup jobs.

[robnicholsonmalt]

>The reason for this is the amount of SharePoint objects for this particular customer which is 89706 (last backup)

I'll have to check the jobs but my client isn't far behind capacity wise @ 7.5TB. This is very much where I think Veeam could make improvements in the management/job interface. For example, whilst we have 2,500 groups/document libraries in there, I'm guessing 2,000 of them are archived and therefore don't need backing up every day. For some of them, once a month would be fine. Sometimes even less.