Veeam M365 backups taking hundreds of hours after version 8 upgrade

[jokester6633]

Hello

I have had an open case for over a month now, escalated, etc, with no resolution still.

Case #07591384

Basically after upgrading to version 8, our M365 backups take hundreds of hours to complete. This is to on-prem and object storage repositories. Support has done some basic things, editing some proxy file, checking disk speed, etc but nothing has been resolved and I am not sure they are capable of resolving this.

Is this anything your team has seen after upgrading? Typically our M365 backups would take a 2-4 hours at most for the largest (eg Sharepoint) and even just a few minutes on something smaller like Teams.

[Polina]

Hi Paul,

Have you already upgraded to the latest available patch?
Do you have an all-in-one installation or dedicated hosts for the VB365 server, NATS, and the database?

Btw, if you are unhappy with the turnaround on your case or the quality of support you receive, you may consider escalating your case via the "Talk to Manager" function.

[jokester6633]

Hi Polina,

I am on 8.1.0.305. Updated to 8.1 from 8 as that was recommended first before I entered the ticket.

It is an all-in-one installation on the same server for the backup server, nats and the postgre database. It is shared with VBR as well.

I think I have hit the manager button at least twice now.

[mjr.epicfail]

Hi Paul,

What are the specs of the VBM365 server? I've noticed a big increase of memory usage of the NATS service (albeit that mine is on a linux server) after upgrading to 8.1.
Maybe the OS is swapping memory to disk?

[karsten123]

update to the latest version and migrate your vb365 server to a dedicated machine.
sharing resources with vbr is almost never a very good idea.

[jokester6633]

Hi Paul,

What are the specs of the VBM365 server? I've noticed a big increase of memory usage of the NATS service (albeit that mine is on a linux server) after upgrading to 8.1.
Maybe the OS is swapping memory to disk?

16 core x 2.3Ghz on VM
48GB RAM

[jokester6633]

update to the latest version and migrate your vb365 server to a dedicated machine.
sharing resources with vbr is almost never a very good idea.

Sure, could go down that path eventually.

I assume our deployment is not very atypical and wondering if there isnt something fundamentally broken or misconfigured for it to behave like this after the update from 7 to 8, otherwise I would imagine this being more common if version 8 is way heavier on resources.

[mjr.epicfail]

you say its a VM with 16 cores, what's the CPU, are you sure this isn't NUMA related? Are you seeing CPU ready issues?

[karsten123]

is the vbr server also all-in-one?
do the vb365 jobs run at the same time as your vbr jobs? what is your monitoring saying about performance metrics?

[jokester6633]

you say its a VM with 16 cores, what's the CPU, are you sure this isn't NUMA related? Are you seeing CPU ready issues?

Intel Xeon Platinum 8380, I am not seeing abnormal with CPU usage, for instance right now 365 backups are running and CPU is at 4% for Veeam, 9-12% total with other processes.

is the vbr server also all-in-one?
do the vb365 jobs run at the same time as your vbr jobs? what is your monitoring saying about performance metrics?

For VBR also all in one, and there are 3 additional backup proxies for our main site, and one proxy an a seperate site.
No, the VBR backups run middle of the night, and M365 were typically every few hours throughout the day.

So far support _thinks_ it may be a possible disk issue. The repositories are located on a SAN. The VBR backups also backup to the same SAN with no issues, so it does not look like say a corruption issue. Using an AJA tool the speeds were good to the SAN. Using MS DISKSPD, speeds varied depending on my command options, but best total I/O per s was 8705. That was with command "diskspd.exe -t12 -o32 -b4k -r4k -w50 -d120 -Sh -D -L -c5G H:\temp\io.dat"

Again, no issues prior to the update to version 8 for M365. No issues with our VBR.

[mjr.epicfail]

Intel Xeon Platinum 8380, I am not seeing abnormal with CPU usage, for instance right now 365 backups are running and CPU is at 4% for Veeam, 9-12% total with other processes.

I meant on the ESXi level (if its VMware). Maybe its a good thing to look on the ESXi layer to.

If I understand correctly, support thinks its a disk issue for the metadata cache? or a disk issue where the Postgress DB just doesn't want to perform on the same disk?

Is VBR also running on Postgres instead of MSSQL? In that case the Postgress instance is shared between VBR and VBM365.
Maybe some tuning on Postgres is needed.

Keep in mind that VBM365 v8 is very different from v7 and earlier versions as the communication between the components now go through NATS and there is a centralized DB instead of a JETdb per proxy.

[jokester6633]

Any tips on checking performance of Postgres or NATS server?

[DaStivi]

is the job still running (counting objects) or is it stalled at a particular item??

i had issues in v7, where a job just hangs on one particular mailbox for example... i stopped the job
created a new one and just put that single mailbox in it and pointed again to the same repo as in the original job. then run this job... it triggers a "full sync" for the single item...
after that, i was able to run the main job again successfully without hanging again.

[mjr.epicfail]

Hi Paul,

I do remember some extra configuration steps needed if you install VBR first and VBM365 second on the same instance. But I'm not sure if that was only for SSPI errors or also performance.
For the NATS, its memory intensive so you should see restriction on that part right away.

Stephan pointed out a valid point, it could be a stuck object.
Also, as this is not a support forum, we won't ask logfiles as this is a public forum, its best for now to ask for escalation (again) and ask for next tier support.

[hans.ovli]

Hi, we have experienced a lot of problems due to performance issues after upgrading from v7 to v8.
We now run v.8.1.0.4068 and after upgrading ALL disks on the console server and the 2 proxies we have from 500 to 5000 IOPS, it now seems to run ok.
So disk performance is way more important in v8 from our perspective.

[evgenich]

Hi,
Where di dyou get version 8.1.0.4068? Dont see it in release notes.
veeam-backup-for-microsoft-365-f47/curr ... 39185.html

[hans.ovli]

Latest patch i received directly from the support after troubleshooting "Failed to download block in time" errors.

[PwrBank]

Not that this is helpful for troubleshooting, but maybe some context.

We have been using M365 backup for about a year now and we've NEVER had it have good performance, ever. The best we've ever seen it download our environment is 15MB/s. We've created multiple tickets to have support help get it tuned, and that's the best we ever got.

We went with a different alternative and it backs up our M365 as fast as the onsite network can handle, and somehow it's compressed it much better, taking up less than half the space the Veeam backup took.

Veeam doesn't seem to have the M365 tool worked out just yet.

[mjr.epicfail]

PSQL as centralized DB requires a lot more IOPS than the JetDB's on the proxies that were in use in v7.
That could account for the better performance seen on higher IOPs disks.