Office 365 backup performance. Good/Bad ?

[rdrost]

These are my results for the first Full Backup :

On Premis:Exchange backup is 3 minutes for 30 MB/775 items data. In Azure it took 6 minutes for the same to the Azure blob.
On Premis:Onedrive backup took 8:28 minutes for 10,28 GB data. In Azure it took 9:30 minutes for 11,5 GB to the Azure blob.
On Premis:The sites took 54 minutes for 73.7 GB/42644 items . In Azure it took 1:37 hours for 74.4 GB/54456

Furthermore I noticed that most of the time the Source was the bottleneck. Also the cpu was close to 100 % sometimes during a 5/10 minutes period but not all the time.

Is this good or bad ?
Anyone else with better figures /

[Mike Resseler]

Hey @rdrost

How did you run these jobs? What was the timing in between? If the source is the bottleneck, it probably means you were throttled.

[rdrost]

First i had only one backup account but then added 8 backup auxilary backup accounts. Veeam support says (based on the logs) there's no throttling anymore after this addition.
Then i ran the Onedrive Full backup job for the organization and let it finish.
Sometime after that i ran the SharePoint Full backup.
Then i ran the Exchange Full backup but since the organision has mail on premis there's not much to backup.

I must add that the cpu was near to 100 % during periods of 5/10 minutes only with the SharePoint backup. And in the beginning of the job the target is bottleneck and after that it altered between source and target but most of the time it was the source. Now with the SharePoint incrementals also cpu nearing 100 % but with the Incrementals the source is not shown as a bottleneck and they are finished within a few minutes.

But i just wonder what backup performance one can achieve with VBO365.

[Mike Resseler]

The reason I ask is because if you have run the tests (on-prem / public) close after each other, one of them might not have been throttled, but the second run might get throttled.

[rdrost]

OK, yes i did run them rather close after each other. I will test it another time with SharePoint and give it a day or so between. I will share results.

[nielsengelen]

Regarding the 100% CPU, do you have more info on the server configuration?

[rdrost]

On premis : vcpu : 6 Mem: 8 Gb repository : local disk
In Azure : vcpu : 8 Mem: 16 GB repository : azure blob storage

[rdrost]

New results for the SharePoint backup , this time they where performed with 24 hours apart.
Azure: 74.,4 GB / 59489 items. 151 objects Time 1:36 minutes Proces rate : 13.1 MB/s 10 items/s Bottleneck : Source
On premis: 74,8 GB / 59366 items 151 objects Time : 56 minutes Proces rate : 22.5 MB/s 17 items/s Bottleneck : Source.

My general conclusion is that at this point the on premis backup of SharePoint sites are faster then in Azure. The source seems bottleneck in both cases.
The Azure blob storage repository is on Standard disks so that could me improved. Although i did not seem to have another option in Azure when i created the Azure Blob.

But what i realy want to know what perfrormance other users of VBO 365 Experience. Can it be improved much ?
It's not much data and incremental is finished within minutes.

It's a POC. Just want to know what to expect from VBO365 performance.

[Mike Resseler]

@rdrost
Hopefully someone else will also speak, but from my perspective these are good numbers. How many service accounts have you used for the SharePoint backup?

[rdrost]

8 backup accounts.

[Mike Resseler]

You might gain some more speed (depending on your SharePoint setup) with 16 accounts, but again, as said, from my perspective, these are good numbers.

[rdrost]

Thanks for your opinion Mike.
What suprises is that on premis VBO365 backup is faster then Azure (at least for SharePoint Sites it is ).

[Mike Resseler]

To be honest, it surprises me also. I kind of feel that the Azure one might be connecting to the wrong endpoint or so?

[rdrost]

The Azure VBO365 VM is from the Marketplace and is the only VM we have in Azure. Nothing special.
The repository is on Azure blob storage.

Not sure what you mean by wrong endpoint. Do you mean the network ?

[Mike Resseler]

@rdrost Yes. If you are not using an internet proxy theoretically the service should connect to the closest O365 endpoint and there should not be any issues. Unless it is making a tour. For example, when you use a VPN or internet proxy or direct route or so

[agrob]

i can share my numbers:

Exchange Online Backup to On Prem: 84min / 973 Objects / 8.2GB
Sharepoint Online Backup to On Prem: 82min / 316 Objects /3.2GB

For exchange, bottleneck always Source (only one proxy and one account is used at the moment)
For Sharepoint, bottleneck mostly Source, sometimes Target

Performance depends also on items processed

[rdrost]

Mike: No internetproxy or VPN is used at this poc. I am in a proces with Veeam to add more backup accounts from 8 to 16 to 24 and increase the threads from 64 to 96. Raising to 16 backup accounts did already ipmprove a bit (10 minutes faster).

agrob: Thanks for your numbers.

Unfortunatly i cannot test mail backup at this point.

[shafeeq]

Hi @Mike,
We are also doing a POC for sharepoint online backup and it seems to be really slow. Here are the details

Infra in Azure:

• Both VBO and proxy running on same VM4 vcpus, 16 GiB memory

• Backup is pushed to azure blob storage(StorageV2, Standard Tier)

• Local Cache disk is premium SSD

• Number of Aux accounts 3

• Full Backup Size 14 GB, Time taken 9 Hrs

• Last incremental size 1 GB, Time taken 1.15 hrs

• also,tried the same with local repos and VM with high specs but no significant difference

Is this the expeted performance?

The size of the site will go upto some hunderd GB once it is live and we have multiple tenants like this.

[Polina]

Hi @shafeeq,

First, which Azure template are you using? Next, 3 accounts might be not enough. Normally we recommend starting with at least 8 accounts.
Did your full backup complete with a 'Success' or 'Warning' status?

[shafeeq]

Template is Standared B4ms, VBO version is 4.0.1.545
Full backup was completed with warning

[rdrost]

I have some new results for SharePoint Full backup after adding the backup accounts (now 24) and increasing the threads (now 96).

Azure: 78.,2 GB / 63372 items. 154 objects Time 1:26 minutes Proces rate : 15.1 MB/s 12 items/s Bottleneck : Source
On premis: 78,2 GB / 63437 items 154 objects Time : 51 minutes Proces rate : 26.2 MB/s 20 items/s Bottleneck : Source.

So i have still a bit improvement of performance both in Azure or on Premis. On Premis is still a bit faster which still suprises me.
Still working on it with Veeam Support to pin down further.

[Polina]

Consider changing your VM template to compute optimized - VBO underperforms on B-series.
Also, you can check 'Accounts throttling statistics' in proxy logs to get an idea if throttling still affects processing or not.

Overall, I must say that your processing rates look quite fair.

[rdrost]

Polina, i am using the E series (compute optimized) template.
Will check throttling again with Veeam Support.

But isn't it odd that backup on premis perform better then in Azure ?

[Polina]

Well, based on what we see in our tests and learn from the field I'd also expect that backup to Azure would have been faster compared to Jet.

[shafeeq]

Consider changing your VM template to compute optimized - VBO underperforms on B-series.
Also, you can check 'Accounts throttling statistics' in proxy logs to get an idea if throttling still affects processing or not.

Overall, I must say that your processing rates look quite fair.

Hi Polina,
I have now 8 auxilary users and VM is compute optimized FSv2. Last incremental job took 45 min for 784 MB.
Bottleneck is shown as Target. Is there any sugestion for this?
I will run a full backup tonight to see if any improvemnt there!

[shafeeq]

Hi,
Most of the time bottleneck is Target, process rate is around 400 KB/s and average writing rate is around 500-650 KB/s. Is this the expected one?
Any recommended configuration for the azure storage account?