Is Veeam still willing to optimize performance issues on MSSQL?

[mkretzer]

Hello,

in the past weeks we found some performance issues which most likely are connected with the Veeam database (MSSQL in our case):

- The backup browser is extremly slow / hangs for minutes the more backups we have in one job
- Synthetic fulls could be much faster on our new flash based backup storage on XFS but seem to be limited by how fast Veeam server can handle the different tasks (even if repo is not loaded at all)

Does it still make sense to open a support case or are such optimisations no longer relevant to Veeam as everything will be on Postgresql in the future?

Markus

[Gostev]

Hi, Markus. Actually, the operations you mentioned require very little (if any) configuration database interaction. Their slowness is typically caused by backup storage performance, never by database engine type.

But answering your question - yes, it does make sense. While as you can imagine, in the past almost 20 years our interaction with Microsoft SQL Server has been heavily optimized and polished already, it is unlikely you will encounter many bugs in this area. However, if you happen to find one, we will definitely fix it as in our experience such bugs impact both database engines equally anyway, as most of the code is shared.

The only minor difference is that due to our support matrix for SQL Server versions, we must use the "lowest common denominator" SQL functionality level when we're talking to SQL Server. While for PostgreSQL we used its latest version at launch, which allows us to leverage most modern SQL capabilities with it, making it perform a little better under heavy load in certain scenarios.

However, this difference will not have a material impact unless you're operating your VBR deployments close to our recommended maximums.

[mkretzer]

Gostev,

- the backup browser does not really do anything on backup storage.
- our new backup storage is extremely overpowered. Latency is always at 0,2 ms. Still, Veeam hangs between tasks - as before when there was not much optimization for many VM in one job.

What we see is not really a bug, just not perfectly optimized. Everything works if you are patient.

We are definitely at the recommended maximums but these look somewhat like they were designed for us after we did our last optimization with dev

Markus

[Gostev]

This statement is not technically correct:
"the backup browser does not really do anything on backup storage".

The mount process is quite I/O intensive as the OS must read the entire MFT content from a disk image in backup.
If you know anything about NTFS MFT design, you know it can potentially take a bigger part of the disk.
And during this whole time, which can take minutes, there's literally zero configuration database interaction.

[mkretzer]

Sorry not the backup browser i meant
I mean selecting a restore point - i search for a point and then the whole interface hangs just before/after i right click on the point. As soon as i am in the restore dialog everything is fast again.

An example: We need 10 Minutes to get to the Exchange restore dialog but as soon as the dialog is finished and the backup browser is started we see the exchange databases and can even select a Mail in 1-2 Minutes (because the backup storage is SO FAST).

[Gostev]

Then something is terrible wrong, some corner case UI bug or perhaps some 3rd party tech is impacting Veeam or something. As there's nothing that needs to happen at this stage that needs 10 minutes.

[bct44]

Hello,

Without considering the right sizing for the VBR and its DB, I have seen this kind of behaviour in customer environments due to AV/EDR on the jump host with console and/or on Veeam Components. After polishing the policies of some security tools, it was much better.

[mkretzer]

We have set all exclusions as needed/documented.
It happens also directly in the Veeam Server.

I am quite sure that both are DB/Process optimization issues as they "feel" exactly like such issues in the past.

We tried on our test system with the same EDR without any exclusions and had no issues whatsoever (much less data but also much slower server).

[mkretzer]

After talking to Hannes Kasparick i have opened case # 07635924 for the synthetic issue.

[iDeNt_5]

Hi mkretzer,

I had a similar experience with a specific Client.

Just out of curiosity, how much RAM does the configuration DB server have? Is it dedicated only to this VBR?

[mkretzer]

1 TB, Veeam is on this server alone with nothing else. 192 CPU cores.

Support now tells me that is because of 1900 VM in one job. But since 2000 is supported a told them i want to help them optimize this

[Gostev]

Thanks, we will work with you as clearly there are no compute constraints and the performance QA team got really interested in the issue.

[mkretzer]

Nice, we will provide everything they need