Comprehensive data protection for all workloads
KT752
Novice
Posts: 5
Liked: 1 time
Joined: Mar 31, 2021 2:25 pm
Full Name: Kortenay Templeton

Re: StoreOnce backups slow on V11

Post by KT752 » 1 person likes this post

I just wanted to share what we are doing as a solution. We used SMB shares on StoreOnce only because when we moved to Veeam and StoreOnce, agent backups to a Catalyst store wasn't supported. Whatever the reason when moving to Veeam 11 the biggest problem was that agent backups and tape backups that backed up the agent backups were extremely slow as they were using the SMB shares on StoreOnce. StoreOnce now supports agent backups to Catalyst stores. We created a new Catalyst for the agent jobs and much improvement has been seen. Tape backups are running fast again. Agent backups are running quicker and jobs are not stacking up as they were before. I hope this helps others.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: StoreOnce backups slow on V11

Post by foggy »

Our support engineers report that in some cases they see much better performance with block alignment enabled on the repository. Could those who have it disabled please check this assumption?
m.aschenazi
Lurker
Posts: 2
Liked: never
Joined: Apr 12, 2021 3:05 pm
Full Name: MAsch
Contact:

Re: StoreOnce backups slow on V11

Post by m.aschenazi »

We have the same issue here, case #04734098. Been engaged with support since 2nd April. Copy jobs to SMB StoreOnce repo are very slow (~4 - 5 MB/s), NFS repo to the same StoreOnce works at the normal speed we would be getting before the upgrade from 9.5 to 11 (~ 80 - 100 MB/s).
Did some tests by simply copying a file from the the local disk repository on the Veeam server to \\storeonce\share using Windows File Explorer, so still using SMB, that copies at the same speeds we used to see on Veeam 9.5 or speeds we see on the NFS repo.
Our problem is actually how to move the existing backups from the SMB to the NFS repo, is a lot of data. We are also looking to get a Catalyst license now.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: StoreOnce backups slow on V11

Post by Andreas Neufert » 2 people like this post

Can you try the following registry key on the Backup Server (only when you are on v11)

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
UseUnbufferedAccess
• Type: REG_DWORD
• Set it to 0
Restart Veeam Backup Service
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: StoreOnce backups slow on V11

Post by foggy » 4 people like this post

Actually, both enabling block alignment or disabling unbuffered access should help (either of those). In v11 we've forced unbuffered access by default, which doesn't work well with the disabled block alignment setting on the repository. The issue doesn't show up on the new repositories created in v11 as block alignment is enabled by default for those, while the upgrade procedure doesn't change the alignment setting for existing repos.

So, in short:

Performance degradation:
- block alignment = disabled
- unbuffered access = enabled

Normal performance:
- block alignment = enabled
- unbuffered access = enabled
or
- block alignment = disabled (recommended for StoreOnce and other dedupe devices)
- unbuffered access = disabled

This is going to be addressed in one of the next updates for v11.
darryl
Enthusiast
Posts: 35
Liked: 4 times
Joined: May 11, 2011 1:37 pm
Contact:

Re: StoreOnce backups slow on V11

Post by darryl » 1 person likes this post

Recap for me:
StoreOnce CIFS share, backup copy performance went down to 1-4 MB/sec after the upgrade to v11. Existing StoreOnce repositories have alignment disabled.

As suggested tried backup copy to a new StoreOnce NFS share. Alignment disabled. Also turned on per-machine backup files at this time. Performance is great, but overall leaves us with an issue as we're storing a bunch of data to a different de-dupe domain on the StoreOnce, and end up using up a lot of space.

Today added the UseUnbufferedAccess reg key and re-tried my old CIFS Backup Copy jobs. Performance is back to what it was. Yay!

I'll do more testing but have a couple of questions:
Will that UseUnbufferedAccess cause performance issues with my other backup repositories - SAN and USB attached? No way/need to target only the StoreOnce CIFS repositories?

As a result of this exercise, I understand that I probably should have had per-machine backup files enabled on my existing StoreOnce CIFS repositories. Is it a problem if I enable that setting in Veeam on an existing backup repository? Would it just trigger for new backup chains?

Thanks to everyone for the help so far.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: StoreOnce backups slow on V11

Post by Andreas Neufert »

The reason why we introduced the changes are performance and processing stability. We had some pretty rough support cases with storage system loosing data on caches (induced by downtimes/environment conditions usually). Here we have a change that affects the performance of existing repository on a specific storage system that we will address in the next patch as shared by Alexander.

In the meantime you can use the registry key shared above or enable block allignment. Block allignment on StoreOnce is not optimal as it affects the deduplication engine, so for now it is better to create the registry key until the patch is out and automate everything.
darryl
Enthusiast
Posts: 35
Liked: 4 times
Joined: May 11, 2011 1:37 pm
Contact:

Re: StoreOnce backups slow on V11

Post by darryl »

Ok - so the patch will address the StoreOnce issue specifically, and the registry key should be backed out at that time?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: StoreOnce backups slow on V11

Post by foggy » 1 person likes this post

darryl wrote: Apr 13, 2021 3:07 pm Will that UseUnbufferedAccess cause performance issues with my other backup repositories - SAN and USB attached? No way/need to target only the StoreOnce CIFS repositories?

As a result of this exercise, I understand that I probably should have had per-machine backup files enabled on my existing StoreOnce CIFS repositories. Is it a problem if I enable that setting in Veeam on an existing backup repository? Would it just trigger for new backup chains?
Per-VM backup chains setting will take effect after the next active full. As for the other repositories affected by UseUnbufferedAccess, we will share our findings a bit later.
m.mohan@al-dhow.com
Novice
Posts: 3
Liked: never
Joined: Mar 27, 2021 1:39 pm
Full Name: Murali Mohan
Contact:

Re: StoreOnce backups slow on V11

Post by m.mohan@al-dhow.com »

All, My issue of slow backup has been resolved ( Thanks to Veeam Support Case # 04715674) by making below changes .

1 : On backup repository checked the option "Align backup File data block" - https://helpcenter.veeam.com/docs/backu ... ml?ver=110

2 : HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
UseUnbufferedAccess
Type: REG_DWORD
Set it to 0
Restart Veeam Backup Service
darryl
Enthusiast
Posts: 35
Liked: 4 times
Joined: May 11, 2011 1:37 pm
Contact:

Re: StoreOnce backups slow on V11

Post by darryl »

darryl wrote: Apr 13, 2021 3:07 pm Recap for me:
StoreOnce CIFS share, backup copy performance went down to 1-4 MB/sec after the upgrade to v11. Existing StoreOnce repositories have alignment disabled.

As suggested tried backup copy to a new StoreOnce NFS share. Alignment disabled. Also turned on per-machine backup files at this time. Performance is great, but overall leaves us with an issue as we're storing a bunch of data to a different de-dupe domain on the StoreOnce, and end up using up a lot of space.

Today added the UseUnbufferedAccess reg key and re-tried my old CIFS Backup Copy jobs. Performance is back to what it was. Yay!

I'll do more testing but have a couple of questions:
Will that UseUnbufferedAccess cause performance issues with my other backup repositories - SAN and USB attached? No way/need to target only the StoreOnce CIFS repositories?

As a result of this exercise, I understand that I probably should have had per-machine backup files enabled on my existing StoreOnce CIFS repositories. Is it a problem if I enable that setting in Veeam on an existing backup repository? Would it just trigger for new backup chains?

Thanks to everyone for the help so far.
So since I've added the registry setting, performance of a backup copy to a USB repository has tanked badly. I don't know if this is expected or not. We had been testing a USB repository on v9.5 and saw poor performance (20 MB/sec). Re-tested on v11 and it was 400+ MB/sec. But after adding UseUnbufferedAccess to work around the StoreOnce issue, we're back to 20 MB/sec on USB. When I get a chance I'll reverse the registry setting and re-test USB, as a confirmation.

Hopefully after the patch, the StoreOnce workaround can be done in such a way that we can apply it ONLY to the StoreOnce repositories.
darryl
Enthusiast
Posts: 35
Liked: 4 times
Joined: May 11, 2011 1:37 pm
Contact:

Re: StoreOnce backups slow on V11

Post by darryl »

...Data block alignment wasn't enabled on USB. Enabled, it's ok afterwards. UseUnbufferedAccess=1 mitigates that? Anyway, wasn't trying to derail the StoreOnce theme of the thread.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: StoreOnce backups slow on V11

Post by foggy »

Hi Darryl, interesting findings - I will let QA know but look forward to your confirmation as well. Essentially, as mentioned above, either enabling block alignment or buffered access should help in the case of StoreOnce (with the former being a recommended workaround from the data consistency perspective). So instead of creating the registry value that affects all the repositories, you could enable alignment just for StoreOnce - have you tried that? Btw, how is the USB device presented, is it a Windows repository actually?
jm@rc
Lurker
Posts: 1
Liked: never
Joined: Jun 07, 2018 2:26 pm
Contact:

Re: StoreOnce backups slow on V11

Post by jm@rc »

Hi there. After upgrading from v10, I had the same slow performance problem with the last v11 release and two storeonce repositories.
We didn't have block alignment enabled which led us to the very same behavior you all had.
I went for the registry key "UseUnbufferedAccess" modification on Backup server and performance are now back.
Thanks all ;)
m.aschenazi
Lurker
Posts: 2
Liked: never
Joined: Apr 12, 2021 3:05 pm
Full Name: MAsch
Contact:

Re: StoreOnce backups slow on V11

Post by m.aschenazi »

Can confirm that adding the registry key "UseUnbufferedAccess" has fixed our speed issue with SMB StoreOnce repository.
PetrN
Novice
Posts: 3
Liked: never
Joined: Mar 27, 2021 8:37 am
Full Name: Petr Neumann
Contact:

Re: StoreOnce backups slow on V11

Post by PetrN »

Andreas Neufert wrote: Apr 13, 2021 1:32 pm Can you try the following registry key on the Backup Server (only when you are on v11)

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
UseUnbufferedAccess
• Type: REG_DWORD
• Set it to 0
Restart Veeam Backup Service
I can confirm that this helped me. Storeonce seems to be working again.
gonzo99
Enthusiast
Posts: 25
Liked: 5 times
Joined: Nov 04, 2015 11:26 am
Contact:

Re: StoreOnce backups slow on V11

Post by gonzo99 »

Just for info:

We do not use StoreOnce, but since V11 our replication jobs took much more time to complete. V-One throwed a lot of messages telling things like "Duration of job "<servername>" (623.2%) is above the configured threshold (200.0%)".
Usually half an hour is sufficient for handling the replication of 23 servers, but after the upgrade to V11 from V10 they needed more than 2 hours for one pass.

Setting the registry key (Useunbufferedaccess) helped here, too. Now the pass time is half an hour again. Thanks for telling.
gonzo99
Enthusiast
Posts: 25
Liked: 5 times
Joined: Nov 04, 2015 11:26 am
Contact:

Re: StoreOnce backups slow on V11

Post by gonzo99 »

I opened the champagne too early....slowliness came back this morning at 6am when the daily replications start over. The durations are now above the configured thresholds again, some of them even slower than yesterday. And these are replication jobs, directly from the vCenter to an ESXi host as the replication target.
That's very unexpected...I never realized such a behaviour after an upgrade and I'm with B&R since V4.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: StoreOnce backups slow on V11

Post by Andreas Neufert » 1 person likes this post

VM Replication works totaly different and I think not affected at all by this setting. Please open a support ticket for investigation.
jah2323
Novice
Posts: 6
Liked: never
Joined: Sep 27, 2012 7:22 pm
Full Name: Jason A Hassett
Contact:

Re: StoreOnce backups slow on V11

Post by jah2323 »

I wonder if the unbuffered registry key will help me with my crappy COHESITY storage platform. It's slow anyway, but after wiping both cohesity units EARLIER this year, speeds were at least bearable. After the v11 upgrade (up at our DR veeam server), the speeds are way slower. Cohesity uses SMB based on SOBR. I was actually about to open a ticket, but decided to check this threat out first. It also sounds like maybe NEW view/repository and SOBR creation from scratch on v11 might help as well? i think i might try that here in a few.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: StoreOnce backups slow on V11

Post by Andreas Neufert »

Yes, it should help in this case.
jah2323
Novice
Posts: 6
Liked: never
Joined: Sep 27, 2012 7:22 pm
Full Name: Jason A Hassett
Contact:

Re: StoreOnce backups slow on V11

Post by jah2323 »

bah. i'm probably going to have to open a ticket. Newly Created SMB SOBR on cohesity is slightly faster than the one that was "pre-existing" before the v11 update. but only slightly faster. The storage is already slow, but it's too slow for me to upgrade my production site to v11. 5tb exchange server was taking around 23 hours for the weekly full (not synthetic as that is horrible on the cohesity compared to my old DataDomain 2500). It is now a bit over 40 hours. I will try the unbuffered registry key (and will test on both the new and the pre-existing SOBR) before i put my ticket in.
darryl
Enthusiast
Posts: 35
Liked: 4 times
Joined: May 11, 2011 1:37 pm
Contact:

Re: StoreOnce backups slow on V11

Post by darryl »

So P20210507 is out but I don't see any mention of a fix for this.

We'll stick with UseUnbufferedAccess then.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: StoreOnce backups slow on V11

Post by foggy »

Right, the latest patch doesn't have any related fixes. We're working on the hotfix at the moment. In the next major update, we will also enable block alignment by default during the upgrade.
darryl
Enthusiast
Posts: 35
Liked: 4 times
Joined: May 11, 2011 1:37 pm
Contact:

Re: StoreOnce backups slow on V11

Post by darryl »

Ok, thanks for the update.
mr_iang
Novice
Posts: 3
Liked: never
Joined: Dec 12, 2016 8:09 pm
Full Name: Ian Gorman
Contact:

Re: StoreOnce backups slow on V11

Post by mr_iang »

Thanks everyone for the insight in to this problem. I've been watching and waiting for my upgrade but needed to move ahead. So I am happy to report that I've upgraded to v11 and manually set the UseUnbufferedAccess registry key with successful backups.
CraigL2112
Enthusiast
Posts: 27
Liked: 8 times
Joined: May 18, 2015 3:05 pm
Full Name: Craig L.
Contact:

Re: StoreOnce backups slow on V11

Post by CraigL2112 »

Everyone -

Wanted to chime in here and report the above registry key did -not- fix the issue for us. We have a case open (#04935464) and are working through some troubleshooting steps. Our products/versions:

Veeam 11 / P20210525
StoreOnce 4500 running latest 3.18.20 firmware
Align data blocks checkbox was already set on repository

What we are seeing is that any -READS- from CIFS/NAS repositories that live on StoreOnce get the 1-4MB/s performance that others have reported.
Reading/writing to Catalyst stores or even a temporary local repository on our Veeam server yield expected performance.

Setting the UseUnbufferedAccess key to 0 and restarting the service did not fix the performance. Pre-V11, performance was not as good as Catalyst but was dramatically faster than 4MB/s.

Is there anything we can throw at this?

Thanks in advance!

-Craig
CraigL2112
Enthusiast
Posts: 27
Liked: 8 times
Joined: May 18, 2015 3:05 pm
Full Name: Craig L.
Contact:

Re: StoreOnce backups slow on V11

Post by CraigL2112 » 1 person likes this post

UPDATE:

We have a fix -- a different registry key than UseUnbufferedAccess:

Key Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Value Name: DisableHtAsyncIo
Value Type: REG_DWORD
Value Data: 1
Value Data Type: Binary

Performance on our NAS share has returned to what it was prior to upgrading to Veeam 11.

Hopefully this helps someone else out there!

-Craig
rubles
Lurker
Posts: 2
Liked: never
Joined: Nov 26, 2021 4:28 pm
Full Name: Will Hoke
Contact:

Re: StoreOnce backups slow on V11

Post by rubles »

jah2323 wrote: May 11, 2021 2:07 am bah. i'm probably going to have to open a ticket. Newly Created SMB SOBR on cohesity is slightly faster than the one that was "pre-existing" before the v11 update. but only slightly faster. The storage is already slow, but it's too slow for me to upgrade my production site to v11. 5tb exchange server was taking around 23 hours for the weekly full (not synthetic as that is horrible on the cohesity compared to my old DataDomain 2500). It is now a bit over 40 hours. I will try the unbuffered registry key (and will test on both the new and the pre-existing SOBR) before i put my ticket in.
JAH2323, Did you ever get your cohesity back up to normal speeds in v11? I am having this same issue where my SoBR are either performing slowly (without the UseUnbufferedAccess registry key) or they are backing up at a reasonable speed with the registry key in place but larger VMs and deltas cause our Linux NFS mount servers to fill up on memory and crash repeatedly.

I have a support ticket in but not much has been determined. REALLY hoping this is fixed in some simple way that I have overlooked and wont require me to rollback our entire backup infrastructure.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: StoreOnce backups slow on V11

Post by Andreas Neufert » 1 person likes this post

Regarding the out of Memory issue. This happens usually only if you work with unlimited task slots on the Repository configuration. It is a very bad idea to configure it that way.
Can you please check?
Post Reply

Who is online

Users browsing this forum: Amazon [Bot], ENBS, MarioZ, xSOU1 and 198 guests