Slow backup performance

[gvo91]

Dear,
Since the 30th of december we're noticing we're having very slow backup performance on all the tenants we're processing for our customers.
Multiple jobs that succeeded whitin minutes or seconds now ran +65 hours, this morning 23 jobs we're still running and on normal days this number is 0. At that moment our Veeam O365 backup version was 4C (build 4.0.1.531). Because we upgraded our staging environment to v5 in the beginning of December we choose to do the same with our production environment today so now we're at 5A (build 5.0.0.1070).
Unfortunately, after restarting the jobs we don't see any improvements and it looks like we going to have a lot of running jobs in the morning.

Examples:
Exchange job customer A (52 objects):
on 31/12 ran 1min 55sec. On 31/12 this job took 4min 44sec. On 1/01 this job ran for 36hours (184 MB processed)

Exchange job customer B (451 objects):
on 31/12 ran 2h 37min 31sec. On 31/12 this job took 2h 39min 37sec. On 1/01 this job ran for 1h 42 min 12sec but the run that started on 2/01 we had to force stop today to perform our upgrade to v5. It was running for 59h 27min and 54 sec (1.1GB processed)

As you can see the time the job takes exploded without any cause and this not only for Exchange Online. Also SharePoint Online and OneDrive for Business jobs show the same issue. No changes were made to our backend infrastructure or the network in our datacenter.
The version used for the jobs in my exaples was 4C but now we upgraded our infrastructure to 5A we already have jobs with 2 objects in them running for 2h 23min that have processed 1MB up to now. Another job is also running for 2h 30min and processed 80MB up to now.
I can also see some job objects failing with 401 unauthorized while other objects do succeed. So it looks like the upgrade did not resolved the issue, all the statistics were much much better before the 30th of December.

I have a ticket open for this issue 04572267 but i was hoping to get some more information/insights here.

Thanks in advance,
kind regards.

[heinrichloewen]

Hello gvo91,

we've the same problem, but our problems starts on 31th of december and we are on 5A (build 5.0.0.1070).
Our case is also open and we got this answer:
"This appears to be a known issue that our QA team is currently investigating."

kind regards

[gvo91]

Hi heinrichloewen,

This morning i received an email from Veeam support where the engineer, as you said, confirms it's a known issue and they already have a hotfix for it.
The engineer wrote 7 simple steps to implement the fix but the first step is 'Ensure you are running version 10.0.2.1070 which I hope is a typo because as far as I know the latest version is 5.0.0.1070.
I'm waiting on confirmation.

Kind regards

[Mike Resseler]

Correct all,

There is indeed an issue with the switch between the old and new year (so glad that one is over ). If you have this issue, please contact support so they can investigate and see if the hotfix applies to you.

And @gvo91 You should be right. The latest version is indeed 5.0.0.1070

Thanks
Mike

[gvo91]

Hi Mike,

Thanks for jumping in. I applied the hotfix provided by support this morning but this did not resolve the issue for us. Backup performance is still very low and a lot of jobs fail with '401 Unauthorized'
A fresh bundle of logs was uploaded to the case so i'm expecting to hear from the engineer soon.

Kind regards.

[Mike Resseler]

Thanks for informing me.

I will follow the case also from my side.

[dasfliege]

Same problem here. We don't have any backups for our customers since 31th december.
Just applied the hotfix and will report tomorrow if that fixed the problem for us.

[mrjack007]

Is the hotfix only for v5 or v4 also? We have multiple servers here with the same thing since December 31st. Rebooted all servers, but still a few jobs running for 20-30-40 hours.

[dasfliege]

Seems to work a little better with the fix in place. At least the jobs started to do anything. But we still have several jobs stuck at some point which are now running since hours with zero activity. I even can’t export logs for the proxy. I receive the following error when I try:

------------------
This request operation sent to net.tcp://127.0.0.1:9191/Proxy did not receive a reply within the configured timeout (00:01:00). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.
------------------

I asked veeam support to raise the priority of this case as we do have SLA with our customers who pay for the backup service and for some of them, we don’t have backups since 6 days, which isn’t acceptable.

[gvo91]

Hi all,

After applying the fix our performance stayed extremely low. I started one or our OneDrive for Business jobs yesterday evening and disabled all other jobs just to make sure there is no server resource throtling.
The job should process the 335 OneDrive for Business objects in it in about 2 hours. Now it's running for 11 hours 15 minutes and it processed 91 objects.

Also when I start an Exchange Online job it fails within minutes saying all objects failed with '401 Unauthorized'. Support tels me both problems cannot be linked to each other but they popped up on exact the same time so one is at least caused by the other.
Bottom line is, just like dasfliege says, we don't have backups for six days in a row now for all of our customers.
We really need a fix asap!

Kind regards,

[dasfliege]

By the way @Mike Resseler
It may would make sense if VBO would be able to perform restores from incomplete or failed backupjobs. Right now if a job is able to save 99.9% of all elements but then fails, i won't be able to restore the succesfull saved elements as i can't select the appropriate restore point. If there is any possibility codewise to implement that function, that would be a nice feature for v6.

[tommy.lidal]

Hi.
Just info from oure veeam installation, we're on V 4.0.1.545
i can see that we also have slow performance on Onedriv backups ( email and sharepoint are stil ok)
started 1/1/21 : 426 objects took 103h to backup 10.4GB, ended sucessfully
the Job that started 31/12/20 did 426 objects , 6GB in 24:50min

no supportcase started, i'll just wait for a working fix
maybe som index job in bacground ? will be exited to see tomorrow 0:-)

Kind regards

[gvo91]

Hi all,

Together with support I got our SharePoint and OneDrive backups up and running again. With up and running i mean that they are processing objects rather than sitting idle for hours and hours. About the performance compared between December and now I cannot jump to conclusions at this moment.

The other issue we're having and support is investigating the Fiddle trace at this momentis the fact that every mailbox (shared, M365 mailbox, user, archive,..) fails with the error 401 Unauthorized. This is also started the 1st of January and for every customer.
A quick look on the Fiddler trace before we concluded the call learned us that Veeam could not connect to Exchange Online Powershell. The strange part is when we connect to Exchange Online Powershell directly from the backup environment using the same username&password it works without a problem. So the problem is definitely situated within the backup software.

Kind regards,

[tvveen]

Same problems here unfortunately
Created case (#04574572) yesterday requesting the hotfix. I hope I can apply this soon to test if this works on our environment
I'll keep you posted

[gvo91]

Dear all,

We found the issue regarding the 401 Unauthorized errors for Exchange Online backups.
When I tried to 're-add' the organization to Veeam I received the following error when Veeam checked the service account permissions:
Connect to Microsoft Graph: unable to connect to the resource. The specified Azure AD application is missing required application permissions (Microsoft API: Exchange, permissions:full_access_as_app)

When I checked the API permissions in Azure AD the Exchange API permissions disappeared and after searching the net I found the following post:
According to this Microsoft Exchange Team blogpost they decomissioned the Outlook REST API beta and Outlook REST API v2.0 on November 30th and retired the Oauth Sandbox on December 31.
Although Requirements for Modern Authentication with Legacy Protocols Allowed and also "Requirements for Modern App-Only Authentication do require permissions from the Exchange API. Although you cannot find it anymore where it used be next to the Azure Active Directory Graph.


You have to go to 'APIs my organization uses' as explained in this blogpost and re-assign the Exchange permissions in order to create a succesfull user, shared, O365 group mailbox backup.
We can switch all organization to Modern App-Only Authentication but as described in the required permissions in the v5 documentation Veeam still needs this permission set in order to function.

@Mike Resseler, is this something we missed or is it also something Veeam wasn't aware of?

Thanks in advance,

[dasfliege]

@gvo91
This is something you have to do after v5 update. It has already been discussed in several posts. f.e.: veeam-backup-for-microsoft-office-365-f ... 70883.html

It actually seems like no one knows why everything was working fine without the additional permissions in v4.
With modern only auth, you don't have to set any permissions. The wizard will do that for you. The documentation just states which permission do have to be in place, but you don't have to manually add them in this case.

[tommy.lidal]

Morning.
regarding time usage it looks like our installation is back on track..
Onedrive backup, 430 objects, 31GB 1:50h
06.01.2021 22.50.01 :: Transferred: 31.0 GB (7449 items) at 4.8 MB/s (1 items/s)

i would gues that my veeam V 4.0.1.545 did a fresh full backup of our onedrive 1/1/21 (based on time usage)
my onedrive repo is 15TB (12,7 used) snapshotbased , virual machine on Nimble hf20

[burrowsr1]

I have the same problems

I built a brand new server as I had another issue that unfortunately Veeam Support could not resolve so I bit the bullet. Needless to say performance is extremely slow running on the latest v5 build for SharePoint and OneDrive Organisation backups. Exchange only job's seem ok however there's a lot less data to deal with.

I have 64 hour and 62 hour jobs and counting with no end in sight at the moment. I would say up until about October of last year my environment on v4 was singing. I've then battled ever since and with a new server in play it hasn't improved. I do see my proxy getting hammered on CPU and RAM mind.

Does anyone get any better performance using Auxiliary Backup Accounts. I have now implemented them for the two jobs mentioned however not sure they will kick in until another start of the job.

I have a call open with Veeam and they have the logs, waiting for a response which should be today.

[dasfliege]

@burrowsr1
Same problems here. I'm struggling since months. We of course have some more customers and more data now, but it seems to me like performance is much much worse since a while. Im still working on it with veeam support and do provide logs almost every day. Will let you know if there is any progress. We also have the CPU and RAM issues on the proxy.

[Mike Resseler]

All,

Please keep working with support. I know I have been silent but in the background the teams are working hard on it.

[HenrikS.]

We also see extreme degrade in performance of OneDrive For Business jobs after new year.
1 incremental (only 3 objects) job taking ~10 minutes in 2020 to complete, took over 4 hours in 2021 for the first incremental, but then got back to just 10 minutes on the next incremental run. There are almost no new items fetched, but very high CPU consumption and constant smaller disk read access.
It seems like "Sync time" for each thread has risen significantly, over 30+ minutes, making me wonder if there are some kind of index that is no longer valid or item hashes beeing recalculated?

[Mike Resseler]

@HenrikS
Log a call, there is a hotfix available, not sure if it is exactly what you encounter, but there is at least one already that solves the performance issue.

[heinrichloewen]

We have applied the hotfix and now backups runs as before.

So, if you have these problems, contact support and let analyse them your logfiles and give you the hotfix.

[Mike Resseler]

Thanks for letting us know @heinrichloewen

[gvo91]

Hi all,

I can also confirm the hotfix took care of the performance issues. Everything is back up and running

Kind regards,

[tommy.lidal]

Hi again.
Looks like my v4 instalation is back to normal speed without any hotfix ( compared history)..
@Mike Resseler : will there bee an updated v5 installation set soon ?

[Mike Resseler]

Tommy,

Later this month. But if it is back to full-speed, it means the first backup after 2021 is completed, and the next ones should be at normal speed anyway. This bug only happens the first backup of the new year. After that it is automatically solved. (Well, until next year but that means you still have some time to apply the fix )

[aich365]

Not sure if it's connected to the issues above but we run version 5.0.0.1070 and it advises last backup as "never" in job details
A bit strange as it is running every 2 hours....
Under "organisations" it has the correct last run time but under last 24 hours it advises "never".

[Mike Resseler]

@aich365
This looks strange. Can you create a support call please? Let us know the ID after you have created it.

Thanks

[aich365]

Case #04583852