Backup are super slow after upgrading to Version 11.

[Supraman]

All my backup jobs including replications are super slow after the upgrade from V10 to V11 latest release. Currently build version for Veeam is 11.0.0.837 P20210401. I have open a support case and uploaded the logs.

Some of my jobs that normally takes 1-2 hours went to 16+ hours without completion and are stuck at 0KB. If anyone have encounter this issue and has a solution please let me know.

Support tech did inform me that after the update the jobs are suppose to be slow for its initial backup job after the upgrade which I am not sure if its true or not.

[veremin]

It looks a bit similar to the issue#2, but the build you're running should have it fixed.

The response from support team sounds a bit strange, so can you provide a ticket number and we will check the case internally?

Thanks!

[Supraman]

Hello Vladimir,

The case number is as follows Case #04752163 and I have conducted the test in which the initial backup did completed, but did not improve the speed of the same job ran the second time.

[Supraman]

Support was able to look at the logs I had provided and this was the responds,

"I have had a chance to review the logs and there does not appear to be any other major issues or reasons for slowdowns. I was able to get clarification that each backup job will need to read the entire disk, even if the VM is part of other jobs that have completed. Let me know once the jobs have had more time to preform the initial runs since the v11 upgrade, I suspect after this is done speeds will resume too rates similar then before the upgrade."

This is severely affecting us as of now we are unable to run replication jobs along with our daily backups.

[PetrM]

Hello,

As far as I see some additional logs have been collected by our support engineer, let's wait for the result of analysis.

Thanks!

[foggy]

I'm not sure why reading the entire disk might be needed after the upgrade, so I'd double-check regarding that with the engineer. Especially since you've mentioned that the jobs 'stuck at 0KB', which might be the actual cause of the problem, not the amount of data read.

[Supraman]

I have uploaded the new logs, as requested and I hope someone would have an answers for us as prior to upgrading to V11 all of our backup were running very fast ranging in a couple of 100mbs, but now it going at in the high 70s and low 10s.

[Supraman]

For anyone going through the same problem I had, the solution was a Registry Fix which apparently address the issue with using Linux base repositories and Raid 6, and shout out to the Sydney, Australia support team.

Add the following registry key to the following Path.

[HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication]
"UseUnbufferedAccess"=dword:00000000

[foggy]

Hi Vinh, thanks for getting back with the update. Turned out you essentially experienced the issue discussed in this thread > StoreOnce backups slow on V11.

[Gostev]

address the issue with using Linux base repositories and Raid 6

By the way, according to our own testing last week, there's no generic issue in V11 with Linux-based repositories and RAID 6... so, it appears to be specific to some storage hardware we don't have. At least in our labs, the default V11 install performs well on such configurations, while disabling unbuffered access reduces performance as expected.

Instead of disabling unbuffered access (which is generally a bad idea for any decent storage hardware), everyone should start from enabling the Align data blocks checkbox in the advanced backup repository settings, since the V11 backup engine is tuned for this setting selected. As the What's New in V11 document explains, this is now the default setting for all newly created backup repositories, however most of the existing repositories will not have this checkbox selected - and having this disabled may result in some performance impact.

Only if this does not help, try disabling unbuffered access next. In theory, it can only possibly help on very low-end hardware without proper enterprise-grade RAID controllers (for example, software RAID) or some "special" storage devices. By the way, do you use software RAID by chance?

[Supraman]

Hello Gostev,

In our environment we use both Synology and Nimble storages, the comment with regards to linux and raid 6 was mention to me by Veeam's tier 2 support. After 4 days of backup and replication issue that was preventing our backup and replication from completing, He manage to get us going by adding the unbuffered access registry. Now since you made mention of Align data blocks and enabling it, I will go ahead and test to see if it actually makes any difference.

On the other hand, may I ask during your lab test may I asked if you guys notice anything weird where the backup/replication jobs hang at 0kb at the start or somewhere during the task and randomly resuming? I have another case with regards to it and I could provide screenshots. I notice it suppose to be address in the latest ISO release.

[Gostev]

Indeed, this was one of the first V11 bugs reported, and it was the first we fixed.

[Supraman]

The reason why I asked was because I had that issue occurring at this moment and I have an active case with regards to it. The currently suggestion by the support tech was to add the following registry key DisableHtAsyncIo on our repository server.

[gwbuckley]

I am experiencing a similar situation. After upgrading to Version 11, backup times are much slower, especially in the case of Synthetic full backup jobs. A job that took 3 hours in the past can take over 15 hours after the upgrade to 11. I see very little pressure on CPU or RAM on both the Veeam backup server or storage repository. My backup repository is a Linux based system running Openmediavault on a Supermicro chassis with about 125TB of storage configured with RAID6 via and a hardware Areca-1882i raid controller. The storage repository is serving the volumes up to a Windows-based Veeam server as an iSCSI device.

My case Case #04725802 — Case #04685952

[Supraman]

Hey Greg,

Have you tried enabling align backup file data block? or the above registry key mention?

The registry key that was added did boost my performance; however I was still having issues with the backup sometime just drops off and goes into 0kbs for hours and then resume randomly. In order to have it apply you would have to reboot all your Veeam server and proxies according to the support rep I had worked with.

The Align backup file data block option suggested by Gostev have also worked and it is one that I am using now and it does require removing the registry key that tier 2 suggests and reboot the Veeam machine and proxies as well.

The location to where it can be found and enable is as follows:

Under the Backup Infrastructure tab on the bottom left of the Veeam console> under Backup Proxies> Backup Repositories> right click on the Repository and select Properties> Repository > Advdance > Check Align Backup file data block options.

I hope it helps you somewhat, as it has help me but has not completely resolve all my performance issue.

[Robvil]

I also do see some slowdown periodicly on both my REFS backup storage (Windows 2016 plus a DELL DAS) but mostly on our Synology NAS which is used as a backup copy repository.
I have not looked deeply into the issue, but i do see both active backup copy jobs and merging process not finished hours after it normal is finished with only a small subset of data changes to be backup.

I have Align backup file data block option turned on, on the Synology, and we are using the newest pacth level.

/Robert

[gwbuckley]

Supraman, yes thank you, I enabled align block yesterday after reading this thread. I'll give it a go and see how it does.

[pber]

I have a similar issue but only with my Backup Copy job between my HPe StoreOnces using a Catalyst store over Fiber Channel. v10 was working perfectly and fast. v11 the backup copy is doing 1 file at a time with big delays between each file.

[PetrM]

Hi Patrick,

I suggest to raise a support request and let our engineers to have a closer look at the issue. Please don't forget to share a support case ID with us.

Thanks!

[nunciate]

Looks like I am having this issue. Upgraded to v11 from the latest v10 version yesterday. Logged in this morning and almost all of my backup jobs were still running. Many extremely slow.

I use an HP Apollo high-density storage unit, not Linux repositories and not REFS. Would the reg key below help in any way?

[HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication]
"UseUnbufferedAccess"=dword:00000000

[pber]

Thanks I'll try that reg setting. My case id is: 04768394

[PetrM]

I recommend to review the suggestions above, in particular this one and to see this thread. Also, it makes sense to confirm correctness of the idea to test the registry key above with our support engineers.

Thanks!

[Gostev]

Instead of disabling unbuffered access (which is generally a bad idea for any decent storage hardware), everyone should start from enabling the Align data blocks checkbox in the advanced backup repository settings, since the V11 backup engine is tuned for this setting selected. As the What's New in V11 document explains, this is now the default setting for all newly created backup repositories, however most of the existing repositories will not have this checkbox selected - and having this disabled may result in some performance impact.

Only if this does not help, try disabling unbuffered access next. In theory, it can only possibly help on very low-end hardware without proper enterprise-grade RAID controllers (for example, software RAID) or some "special" storage devices. By the way, do you use software RAID by chance?

Yes, please don't ignore my message guys

I use an HP Apollo high-density storage unit, not Linux repositories and not REFS. Would the reg key below help in any way?

This is a bad idea, as by doing this you would be disabling the functionality implemented in tight functionality with HPE in particular, to make V11 perform BETTER on storage units like the one you have. On top of that, V11 has been thoroughly tested with HPE Apollo specifically, setting some backup performance records

[nunciate]

For my setup I have the Align data blocks checked. It was set that way when I first created my repositories years ago.

My hardware is 52 - 6Tb drives with 6 - 800Gb SSD dives in front of that for the write cache. All on a P840 Smart Array Controller. The performance of the hardware seems to be good. I have had no real issues other than space over the last few years as we store a lot of data on the volumes.

[gwbuckley]

Enabling the Align data blocks checkbox in the advanced backup repository settings made a big difference. I am curious to see how well the synthetic jobs go this weekend.

[Supraman]

Yes, please don't ignore my message guys


This is a bad idea, as by doing this you would be disabling the functionality implemented in tight functionality with HPE in particular, to make V11 perform BETTER on storage units like the one you have. On top of that, V11 has been thoroughly tested with HPE Apollo specifically, setting some backup performance records

Hello Gostev,

My question for you is basically, I have enabled Align block on my Synology repositories without any issues; however, when I tried that on my nimble repositories I get the following pop up " The file system on the specified volume does not support fast cloning. We recommending using ReFS volume fomratted with 64kb cluster size. Proceed anyways?

As far as what I do know we are not using ReFS volume and we are still in a mix environment with 2012R2 servers and 2016. Therefore, should I select proceed anyways or leave it alone?

[Mildur]

It has nothing todo with the „align“ option.
It looks like you have here an NTFS Partition and veeam wants to let you know, that reFS would be the better choice.
If you have backup data on it, format the volume will mostly be none option in this case.
Simply ignore the warning and proceed in your case

If you can format the volume with a refs partition, then you can use FastClone. You will save massive storage space and will have fast syntentic full, if you choose to format it with refs. But you will loose your existing restore points on this backup repo of course.

[b.vanderhorst]

For anyone going through the same problem I had, the solution was a Registry Fix which apparently address the issue with using Linux base repositories and Raid 6, and shout out to the Sydney, Australia support team.

Add the following registry key to the following Path.

[HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication]
"UseUnbufferedAccess"=dword:00000000

Thanks, saved me some downgrades.

[Supraman]

Just so everyone know I just recently installed the latest hotfix that was release and it has some what resolve my backup hanging up issue where the job would idol at 0kb for a while before starting.

[villapla]

For anyone going through the same problem I had, the solution was a Registry Fix which apparently address the issue with using Linux base repositories and Raid 6, and shout out to the Sydney, Australia support team.

Add the following registry key to the following Path.

[HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication]
"UseUnbufferedAccess"=dword:00000000

Same issue here, Backup Repository on a CentOS 8 server with software RAID 6, worked fine on Veeam 10 but bandwidht limited to ~90 MB/s with Veeam 11.

Adding the registry key fixed it.