Backup are super slow after upgrading to Version 11.

[tinu]

We had a similar Situation after updating to V11 and get rid of slow backup performance by enabling caching on the RAID Controller of our bare-metal Veeam Backupserver...

[tgagnon]

I started having this issue immediately after updating to B&R 11. Turning on align data blocks did not resolve it. I'm backing up to a Linux hardware RAID 60.

I've had case 04914657 open for over a month now and it's gone nowhere. I've uploaded logs like 5 times now and they just sit there until I'm told to upload some more. I'm definitely not pleased with the quality or speed of support I'm receiving. Not how Veeam support used to be.

[PetrM]

Hello,

Please do not hesitate to escalate the support case if you think that more profound technical examination is needed to determine the root cause.

Thanks!

[Gostev]

Veeam support has always been fast for simple issues (like obvious bugs in our code) and slow for the complex ones (like environment-specific performance problems) where even our devs have to guess for weeks what might be wrong with the particular environment by collecting different sets of logs with performance debug mode enabled. So the only real change here is that you've been lucky thus far, and never really faced a single complex issue until now - which left you with the impression that Veeam support is always super fast.

[tgagnon]

Please do not hesitate to escalate the support case if you think that more profound technical examination is needed to determine the root cause.

It's already been escalated to tier 3 and it's just sitting there. Not a single case update from the tier 3 engineer. It sits, I upload some logs, it sits some more, I get asked to upload more logs, it sits some more, etc. I would be more apt to believe the slowness was due to the technical complexity if I got the slightest sense that someone was even looking at my case.

Also, I'm not the only one to notice the dip in support quality lately. I work around a lot of Veeam users. We can agree to disagree with that one. I've already raised the issue with my VAR and I will express my displeasure with my wallet if I have to.

[Gostev]

Yes, this for sure means the case is with R&D. T3s are basically gateways to R&D. It is strange you say there are no updates when you are getting asked about more logs periodically. Of course R&D cannot tell T3 anything meaningful to update you with, until they do find something meaningful. But the fact that they keep asking for logs means they cannot reproduce the same issue internally.

As for support quality, there are both "local impressions" after a few unlucky support cases, and there's a bigger picture across the entire customer base. If you work with Veeam for a long time, you know we send a CSAT survey after every single support case. And this data does not indicate a dip in support quality lately comparing to 5 years ago for example. Last I looked a few weeks ago, 97% of customers answered they were satisficed with the level of support received (which is above average of the past 10 years). Not saying there are no issues because there are still the other 3%, but only that there's for sure no dip in the overall support quality if you compare with some years ago. Which I think is quite an accomplishment considering we're growing 25% YoY and cannot hire fast enough to follow our customer base growth, especially now that IT job markets are going absolutely crazy in some key regions.

And you can help us to further reduce the remaining 3% by using "Talk to a Manager" functionality in a support portal when you think your support engineer is not doing a good job. Although normally you will only run into inexperienced engineer at T1, once you passed that almost all people will have many years of experience with both the product and their role.

[ElmerAcme]

Hello,
we have had a similar problem directly after upgrading from V10 to V11 at about 15 installations.
Backup performance was fine, but replication performance went from 110 MB/s down to 35 MB/s.
We found out that the upgrade changed the standard setting of the proxy transport mode from "automatic" to "direct storage access".
Regards

[PetrM]

Hello,

If the reported "bottleneck" is Target and the replicated VM has thick lazy zeroed or thin disks, then it's worth trying HotAdd mode on the target proxy. Thin and thick lazy disks can be slower on write in Direct SAN mode due to allocation of new blocks which is carried out by means of proxy-Vcenter round trips through the disk manager APIs. Also, I suggest to open a support case as it looks like a technical issue. Please don't forget to paste a support case ID over here for our reference.

Thanks!

[jossmi]

Only if this does not help, try disabling unbuffered access next. In theory, it can only possibly help on very low-end hardware without proper enterprise-grade RAID controllers (for example, software RAID) or some "special" storage devices. By the way, do you use software RAID by chance?

Our clients do use low-end hardware, NAS from BuffaloTech, QNAP and Synology. We use iSCSI and MPIO to provide the best possible performance for these backup repositories. We noticed that performance had degraded significantly, but we thought it was the result of NAS firmware updates which were performed on the same day that we migrated to v11. We changed 2 variables at once and chased the wrong one as the cause of the problem. We just added the "UseUnbufferedAccess"=dword:00000000 registry key yesterday and throughput has returned to normal. We actually did not notice the degradation until we changed a 3rd variable over a month later when we started using SentinelOne, S1, as our AntiVirus solution. We also had to disable S1s VSS snapshot feature on the servers to which the iSCSI volumes were attached.

On a related note, we were wondering what others are using as the maximum concurrent tasks limits on these low end devices. We currently use 2 when the device has 2 MPIO paths and 4 when the device has 4 MPIO paths.

[santunez_cl]

Good evening

I have installed Veeam 11 and when backing up a remote site it takes 30 hours to do an incremental

In the remote site, a Veeam proxy was installed on Windows and the repository is a shared repository that is in a Microsoft Windows Server 2019 cluster, mounted by SMB from a Storage that is in the same remote site.

The backup is made to this repository that is also configured as Scale-Out so that after 20 days it moves the backup to a space in Azure.

Before installing the proxy, the backup took 3 days. What is being backed up are volumes of a file server cluster.

We have made several configurations, reconfigured Jobs and the problem persists in that it does not improve the transfer rates and the backup is getting slower and slower.

Someone will have an idea what may be happening.

Thanks for your help.

Regards

Sebastian

[Natalia Lupacheva]

Hi Sebastian,

Moved your post to the existing thread to keep similar issues together.
Kindly open a Support ticket and share its ID with us so we could monitor your case.
You can also check if there is a bottleneck reported and take a look at this thread - might give you a hint about the configuration.

Thanks!

[foggy]

In case the bottleneck is the target, please review the Issue #9 here - might be worth upgrading to v11a.

[veehexx]

Issue #9 on KB has no fix for us. currently on 11.0.1.1261. IO Alignment setting was off, tried on with no change (i've left this setting enabled now)
Applauing performance upgrading from v10 - around 20-50MB/s since upgrading to v11 which we only did last week.

Our 2 repo's are Centos7, i think raid6 hw raid with LVM+ext4 on top. 'iotop -o' was reporting 99% IO time during these 20-50MB/s speeds.
Hosts are HyperV

using the following regkey has put us back upto 700MB/s+ (peaks just over 1GB/s) according to veeam console, and repos are now <25% IO during these speeds according to iotop.
[HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication]
"UseUnbufferedAccess"=dword:00000000

Without this key, that means its implied "UseUnbufferedAccess"=dword:00000001. unbuffered writes seem a bad default, as clearly evident in this thread.
I didn't try DisableHtAsyncIo=1 as mentioned at post424051.html#p424051. disabling async IO doesnt seem a great idea either for performance.