StoreOnce backups slow on V11

[KT752]

I just wanted to share what we are doing as a solution. We used SMB shares on StoreOnce only because when we moved to Veeam and StoreOnce, agent backups to a Catalyst store wasn't supported. Whatever the reason when moving to Veeam 11 the biggest problem was that agent backups and tape backups that backed up the agent backups were extremely slow as they were using the SMB shares on StoreOnce. StoreOnce now supports agent backups to Catalyst stores. We created a new Catalyst for the agent jobs and much improvement has been seen. Tape backups are running fast again. Agent backups are running quicker and jobs are not stacking up as they were before. I hope this helps others.

[foggy]

Our support engineers report that in some cases they see much better performance with block alignment enabled on the repository. Could those who have it disabled please check this assumption?

[m.aschenazi]

We have the same issue here, case #04734098. Been engaged with support since 2nd April. Copy jobs to SMB StoreOnce repo are very slow (~4 - 5 MB/s), NFS repo to the same StoreOnce works at the normal speed we would be getting before the upgrade from 9.5 to 11 (~ 80 - 100 MB/s).
Did some tests by simply copying a file from the the local disk repository on the Veeam server to \\storeonce\share using Windows File Explorer, so still using SMB, that copies at the same speeds we used to see on Veeam 9.5 or speeds we see on the NFS repo.
Our problem is actually how to move the existing backups from the SMB to the NFS repo, is a lot of data. We are also looking to get a Catalyst license now.

[Andreas Neufert]

Can you try the following registry key on the Backup Server (only when you are on v11)

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
UseUnbufferedAccess
• Type: REG_DWORD
• Set it to 0
Restart Veeam Backup Service

[foggy]

Actually, both enabling block alignment or disabling unbuffered access should help (either of those). In v11 we've forced unbuffered access by default, which doesn't work well with the disabled block alignment setting on the repository. The issue doesn't show up on the new repositories created in v11 as block alignment is enabled by default for those, while the upgrade procedure doesn't change the alignment setting for existing repos.

So, in short:

Performance degradation:
- block alignment = disabled
- unbuffered access = enabled

Normal performance:
- block alignment = enabled
- unbuffered access = enabled
or
- block alignment = disabled (recommended for StoreOnce and other dedupe devices)
- unbuffered access = disabled

This is going to be addressed in one of the next updates for v11.

[darryl]

Recap for me:
StoreOnce CIFS share, backup copy performance went down to 1-4 MB/sec after the upgrade to v11. Existing StoreOnce repositories have alignment disabled.

As suggested tried backup copy to a new StoreOnce NFS share. Alignment disabled. Also turned on per-machine backup files at this time. Performance is great, but overall leaves us with an issue as we're storing a bunch of data to a different de-dupe domain on the StoreOnce, and end up using up a lot of space.

Today added the UseUnbufferedAccess reg key and re-tried my old CIFS Backup Copy jobs. Performance is back to what it was. Yay!

I'll do more testing but have a couple of questions:
Will that UseUnbufferedAccess cause performance issues with my other backup repositories - SAN and USB attached? No way/need to target only the StoreOnce CIFS repositories?

As a result of this exercise, I understand that I probably should have had per-machine backup files enabled on my existing StoreOnce CIFS repositories. Is it a problem if I enable that setting in Veeam on an existing backup repository? Would it just trigger for new backup chains?

Thanks to everyone for the help so far.

[Andreas Neufert]

The reason why we introduced the changes are performance and processing stability. We had some pretty rough support cases with storage system loosing data on caches (induced by downtimes/environment conditions usually). Here we have a change that affects the performance of existing repository on a specific storage system that we will address in the next patch as shared by Alexander.

In the meantime you can use the registry key shared above or enable block allignment. Block allignment on StoreOnce is not optimal as it affects the deduplication engine, so for now it is better to create the registry key until the patch is out and automate everything.

[darryl]

Ok - so the patch will address the StoreOnce issue specifically, and the registry key should be backed out at that time?

[foggy]

Will that UseUnbufferedAccess cause performance issues with my other backup repositories - SAN and USB attached? No way/need to target only the StoreOnce CIFS repositories?

As a result of this exercise, I understand that I probably should have had per-machine backup files enabled on my existing StoreOnce CIFS repositories. Is it a problem if I enable that setting in Veeam on an existing backup repository? Would it just trigger for new backup chains?

Per-VM backup chains setting will take effect after the next active full. As for the other repositories affected by UseUnbufferedAccess, we will share our findings a bit later.

[m.mohan@al-dhow.com]

All, My issue of slow backup has been resolved ( Thanks to Veeam Support Case # 04715674) by making below changes .

1 : On backup repository checked the option "Align backup File data block" - https://helpcenter.veeam.com/docs/backu ... ml?ver=110

2 : HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
UseUnbufferedAccess
Type: REG_DWORD
Set it to 0
Restart Veeam Backup Service

[darryl]

Recap for me:
StoreOnce CIFS share, backup copy performance went down to 1-4 MB/sec after the upgrade to v11. Existing StoreOnce repositories have alignment disabled.

As suggested tried backup copy to a new StoreOnce NFS share. Alignment disabled. Also turned on per-machine backup files at this time. Performance is great, but overall leaves us with an issue as we're storing a bunch of data to a different de-dupe domain on the StoreOnce, and end up using up a lot of space.

Today added the UseUnbufferedAccess reg key and re-tried my old CIFS Backup Copy jobs. Performance is back to what it was. Yay!

I'll do more testing but have a couple of questions:
Will that UseUnbufferedAccess cause performance issues with my other backup repositories - SAN and USB attached? No way/need to target only the StoreOnce CIFS repositories?

As a result of this exercise, I understand that I probably should have had per-machine backup files enabled on my existing StoreOnce CIFS repositories. Is it a problem if I enable that setting in Veeam on an existing backup repository? Would it just trigger for new backup chains?

Thanks to everyone for the help so far.

So since I've added the registry setting, performance of a backup copy to a USB repository has tanked badly. I don't know if this is expected or not. We had been testing a USB repository on v9.5 and saw poor performance (20 MB/sec). Re-tested on v11 and it was 400+ MB/sec. But after adding UseUnbufferedAccess to work around the StoreOnce issue, we're back to 20 MB/sec on USB. When I get a chance I'll reverse the registry setting and re-test USB, as a confirmation.

Hopefully after the patch, the StoreOnce workaround can be done in such a way that we can apply it ONLY to the StoreOnce repositories.

[darryl]

...Data block alignment wasn't enabled on USB. Enabled, it's ok afterwards. UseUnbufferedAccess=1 mitigates that? Anyway, wasn't trying to derail the StoreOnce theme of the thread.

[foggy]

Hi Darryl, interesting findings - I will let QA know but look forward to your confirmation as well. Essentially, as mentioned above, either enabling block alignment or buffered access should help in the case of StoreOnce (with the former being a recommended workaround from the data consistency perspective). So instead of creating the registry value that affects all the repositories, you could enable alignment just for StoreOnce - have you tried that? Btw, how is the USB device presented, is it a Windows repository actually?

[jm@rc]

Hi there. After upgrading from v10, I had the same slow performance problem with the last v11 release and two storeonce repositories.
We didn't have block alignment enabled which led us to the very same behavior you all had.
I went for the registry key "UseUnbufferedAccess" modification on Backup server and performance are now back.
Thanks all

[m.aschenazi]

Can confirm that adding the registry key "UseUnbufferedAccess" has fixed our speed issue with SMB StoreOnce repository.

[PetrN]

Can you try the following registry key on the Backup Server (only when you are on v11)

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
UseUnbufferedAccess
• Type: REG_DWORD
• Set it to 0
Restart Veeam Backup Service

I can confirm that this helped me. Storeonce seems to be working again.

[gonzo99]

Just for info:

We do not use StoreOnce, but since V11 our replication jobs took much more time to complete. V-One throwed a lot of messages telling things like "Duration of job "<servername>" (623.2%) is above the configured threshold (200.0%)".
Usually half an hour is sufficient for handling the replication of 23 servers, but after the upgrade to V11 from V10 they needed more than 2 hours for one pass.

Setting the registry key (Useunbufferedaccess) helped here, too. Now the pass time is half an hour again. Thanks for telling.

[gonzo99]

I opened the champagne too early....slowliness came back this morning at 6am when the daily replications start over. The durations are now above the configured thresholds again, some of them even slower than yesterday. And these are replication jobs, directly from the vCenter to an ESXi host as the replication target.
That's very unexpected...I never realized such a behaviour after an upgrade and I'm with B&R since V4.

[Andreas Neufert]

VM Replication works totaly different and I think not affected at all by this setting. Please open a support ticket for investigation.

[jah2323]

I wonder if the unbuffered registry key will help me with my crappy COHESITY storage platform. It's slow anyway, but after wiping both cohesity units EARLIER this year, speeds were at least bearable. After the v11 upgrade (up at our DR veeam server), the speeds are way slower. Cohesity uses SMB based on SOBR. I was actually about to open a ticket, but decided to check this threat out first. It also sounds like maybe NEW view/repository and SOBR creation from scratch on v11 might help as well? i think i might try that here in a few.

[Andreas Neufert]

Yes, it should help in this case.

[jah2323]

bah. i'm probably going to have to open a ticket. Newly Created SMB SOBR on cohesity is slightly faster than the one that was "pre-existing" before the v11 update. but only slightly faster. The storage is already slow, but it's too slow for me to upgrade my production site to v11. 5tb exchange server was taking around 23 hours for the weekly full (not synthetic as that is horrible on the cohesity compared to my old DataDomain 2500). It is now a bit over 40 hours. I will try the unbuffered registry key (and will test on both the new and the pre-existing SOBR) before i put my ticket in.

[darryl]

So P20210507 is out but I don't see any mention of a fix for this.

We'll stick with UseUnbufferedAccess then.

[foggy]

Right, the latest patch doesn't have any related fixes. We're working on the hotfix at the moment. In the next major update, we will also enable block alignment by default during the upgrade.

[darryl]

Ok, thanks for the update.

[mr_iang]

Thanks everyone for the insight in to this problem. I've been watching and waiting for my upgrade but needed to move ahead. So I am happy to report that I've upgraded to v11 and manually set the UseUnbufferedAccess registry key with successful backups.

[CraigL2112]

Everyone -

Wanted to chime in here and report the above registry key did -not- fix the issue for us. We have a case open (#04935464) and are working through some troubleshooting steps. Our products/versions:

Veeam 11 / P20210525
StoreOnce 4500 running latest 3.18.20 firmware
Align data blocks checkbox was already set on repository

What we are seeing is that any -READS- from CIFS/NAS repositories that live on StoreOnce get the 1-4MB/s performance that others have reported.
Reading/writing to Catalyst stores or even a temporary local repository on our Veeam server yield expected performance.

Setting the UseUnbufferedAccess key to 0 and restarting the service did not fix the performance. Pre-V11, performance was not as good as Catalyst but was dramatically faster than 4MB/s.

Is there anything we can throw at this?

Thanks in advance!

-Craig

[CraigL2112]

UPDATE:

We have a fix -- a different registry key than UseUnbufferedAccess:

Key Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Value Name: DisableHtAsyncIo
Value Type: REG_DWORD
Value Data: 1
Value Data Type: Binary

Performance on our NAS share has returned to what it was prior to upgrading to Veeam 11.

Hopefully this helps someone else out there!

-Craig

[rubles]

bah. i'm probably going to have to open a ticket. Newly Created SMB SOBR on cohesity is slightly faster than the one that was "pre-existing" before the v11 update. but only slightly faster. The storage is already slow, but it's too slow for me to upgrade my production site to v11. 5tb exchange server was taking around 23 hours for the weekly full (not synthetic as that is horrible on the cohesity compared to my old DataDomain 2500). It is now a bit over 40 hours. I will try the unbuffered registry key (and will test on both the new and the pre-existing SOBR) before i put my ticket in.

JAH2323, Did you ever get your cohesity back up to normal speeds in v11? I am having this same issue where my SoBR are either performing slowly (without the UseUnbufferedAccess registry key) or they are backing up at a reasonable speed with the registry key in place but larger VMs and deltas cause our Linux NFS mount servers to fill up on memory and crash repeatedly.

I have a support ticket in but not much has been determined. REALLY hoping this is fixed in some simple way that I have overlooked and wont require me to rollback our entire backup infrastructure.

[Andreas Neufert]

Regarding the out of Memory issue. This happens usually only if you work with unlimited task slots on the Repository configuration. It is a very bad idea to configure it that way.
Can you please check?