Comprehensive data protection for all workloads
mr_iang
Novice
Posts: 3
Liked: never
Joined: Dec 12, 2016 8:09 pm
Full Name: Ian Gorman
Contact:

StoreOnce backups slow on V11

Post by mr_iang »

Hi,
I have Case # 04688930 open with support, it is really dragging on unfortunately and I'm reaching out to the community to see if anyone else has run in to this or has tips. I think it is specific to the Storeonce given my troubleshooting so far.

Backstory:
We were on Veeam 9.5U4 with vSphere 6.7, running two Veeam servers, the secondary just as a proxy. The primary server only had access to the backup device via CIFS on a dual 10gig connection -we were not using Catalyst. I'd followed the setup recommendations so that dedupe was handled by the appliance, in general this has been working well for at least 6 months. We had been planning on an upgrade for a while but Veeam just worked, so.. Anyway we did run in to an issue with an instant recovery recently, and during support it was revealed that this issue was addressed in newer versions. So some motivation came in. Given the great track record with Veeam I went all-in and upgraded directly to Version 11.

That night our 40 or so backup jobs for approx 75 VM's started up and it became clear very quickly that there was a problem. The jobs reported the bottleneck as the target. I took a look at the setup and it seemed like maybe they were having an issue because the secondary proxy server didnt have CIFS access to the backup appliance, so I modified the job proxy preferences accordingly. This made no difference. I even opened up access to the CIFS share from the proxy to no avail. Looking through job logs, even a single job running at a time was getting an average of 5MB/s and even then the little traffic graph would simply stop for periods. Our jobs piled up and I had to abort them for the next day.

After giving up on Veeam11 because we were falling behind on backups I installed Veeam 10 on another server and rebuilt the jobs from scratch. That was a pain.. and unfortunately we didnt have a snapshot of the backup server to roll back on, this just seemed like the best way to get backups caught up. Of course then I had to redo many base images as some of the backups had completed and now were a newer version than my V10 server supported.

Once caught up I took a snapshot of the backup server and did the Veeam 11 upgrade then tried backups on a couple jobs. Same issue as before (keep in mind this is a completely new backup server). Jobs which ran 100+ MB/s previously were struggling around 5 megs. I pulled logs out and reverted the snapshot. Backups are running smoothly on Veeam10, but we need a solution here..

Cheers,
Ian
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: StoreOnce backups slow on V11

Post by HannesK »

Hello,
if you say that it works faster with 9.5U4 and V10 than with V11, that sounds strange. I can only recommend to continue with support to find the reason.

75 VMs should be a hand full of jobs. StoreOnce should use Catalyst integration. For dedupe appliance without integration (SMB in your case), active full is the way to go. I also assume that you have configured per-machine backup chains and active full backups https://bp.veeam.com/vbr/VBP/3_Build_st ... appliances

In general, the setup is against the best practices. So having slow speed in general is expected. Having target as bottleneck is no problem in general. Please see Forum FAQ. http://forums.veeam.com/veeam-backup-re ... tml#p95291 And maybe ignore whatever V11 bottleneck analysis says, because there might be a bug in there. Only look at the overall backup time (and only compare similar settings)

Best regards,
Hannes
PetrN
Novice
Posts: 3
Liked: never
Joined: Mar 27, 2021 8:37 am
Full Name: Petr Neumann
Contact:

Re: StoreOnce backups slow on V11

Post by PetrN »

Hi, I have same problem after update Veeam to Veeam 11. Storeonce CIFS are terrible slow. Arround 5MB/s! Catalyst works but CIFS are faster in version 10. Case #04699964
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: StoreOnce backups slow on V11

Post by HannesK »

Hello Petr,
in your case, there seem to be issues on the StoreOnce side. Please continue investigations with support.

In general, just because of the overall amount of customers with StoreOnce systems, there is a statistically significant chance that something looks related to the upgrade that would also happen without upgrade.

For everyone else with the same issue: please contact support and post the case number here. We need to find out whether it's really an issue on Veeam side just just random StoreOnce issues, that happened with "bad timing"

Thanks,
Hannes

PS: ah, and use catalyst. Inline dedupe appliances perform poor in general. Using them with SMB makes the performance worse.
soncscy
Veteran
Posts: 643
Liked: 312 times
Joined: Aug 04, 2019 2:57 pm
Full Name: Harvey
Contact:

Re: StoreOnce backups slow on V11

Post by soncscy »

Out of curiosity, what keeps you using the SMB shares? (remember, CIFS is dead)

I know that SMBSigning hits Storeonce devices hard hard so ensure that's disabled on your gateway. I would strongly advise that if you're pushing SMB shares across WAN that you immediately rethink your strategy as this is incredibly poor design.

Does the cache bypass introduced by v11 also apply to SMB? Maybe that's the issue.
m.mohan@al-dhow.com
Novice
Posts: 3
Liked: never
Joined: Mar 27, 2021 1:39 pm
Full Name: Murali Mohan
Contact:

Re: StoreOnce backups slow on V11

Post by m.mohan@al-dhow.com »

Case ID# 04715674
Hi... I too have same issue, I have upgraded to version 11 and backup time increased to thrice. Its very strange, same storage, same data size but backup time is terrible. Raised a ticket - Case ID# 04715674 and they asked to install patch (KB4126), which doesn't help even and waiting for Veaam support team. For me incremental backup time/one time full backup .. all jobs having same issue. If anyone found solution, please share..
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: StoreOnce backups slow on V11

Post by foggy »

Hi Murali, just to confirm - is it SMB as well (no Catalyst)?
m.mohan@al-dhow.com
Novice
Posts: 3
Liked: never
Joined: Mar 27, 2021 1:39 pm
Full Name: Murali Mohan
Contact:

Re: StoreOnce backups slow on V11

Post by m.mohan@al-dhow.com »

@foggy .. Yes It is SMB ( Local disk on backup server)
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: StoreOnce backups slow on V11

Post by HannesK »

Hello Murali,
this thread is about HPE StoreOnce and SMB.

If you really have an SMB share on the backup server, then this is a different scenario. Your scenario should be fixed by using "direct attached storage" instead of SMB protocol.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110
https://helpcenter.veeam.com/docs/backu ... 110#server


Best regards,
Hannes
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: StoreOnce backups slow on V11

Post by foggy »

Might be a general SMB issue, not StoreOnce-specific, we're investigating.
PetrN
Novice
Posts: 3
Liked: never
Joined: Mar 27, 2021 8:37 am
Full Name: Petr Neumann
Contact:

Re: StoreOnce backups slow on V11

Post by PetrN »

HannesK wrote: Mar 27, 2021 6:26 pm Hello Petr,
in your case, there seem to be issues on the StoreOnce side. Please continue investigations with support.

In general, just because of the overall amount of customers with StoreOnce systems, there is a statistically significant chance that something looks related to the upgrade that would also happen without upgrade.

For everyone else with the same issue: please contact support and post the case number here. We need to find out whether it's really an issue on Veeam side just just random StoreOnce issues, that happened with "bad timing"

Thanks,
Hannes

PS: ah, and use catalyst. Inline dedupe appliances perform poor in general. Using them with SMB makes the performance worse.
Hi Hannes,

I tried to save data to another CIFS share to another data center in another repository that shows no errors and the same 5MB / s problem. Additionally, storage file errors are for Veeam-managed tasks only. Rather, it is a question of whether Veeam11 caused consistency errors due to its slow writing.
darryl
Enthusiast
Posts: 35
Liked: 4 times
Joined: May 11, 2011 1:37 pm
Contact:

[MERGED] Very slow backup copy to StoreOnce after upgrade from 9.5 U4 to 11

Post by darryl »

Support case 04726060

Our environment:
Single physical backup server, 192 GB, 24/48 cores
Backing up mostly ESXi VMs, some Hyper-V
FC connected SAN repository

We do backup copies to a HP StoreOnce 4500 share (no catalyst). We set the backup copies for 99 increments, and just start a new backup copy chain every month. We do this to avoid any random or read I/O from the StoreOnce, which would be brutal. Each chain ends up being the original VBK and 30 or so VIBs.

We've been doing this since 2015, across various Veeam versions, no issue. Not fancy but it works.

Upgraded from 9.5 U4 to 11 20210319 and backup copy jobs have terrible performance now. Like 40 MB/sec processing down to 1 MB/sec. So backup copy jobs aren't completing.

Before upgrade, on 9.5, backup copy jobs show 10 percent source, 99 percent target.
After upgrade, on 11, backup copy jobs show 50 percent source, 99 percent target.

I'm wondering if there's been a behaviour change that's affecting I/O on my target (is it being read for some reason?).

I opened a case yesterday with no response. I'm open to suggestions of where to start looking.
darryl
Enthusiast
Posts: 35
Liked: 4 times
Joined: May 11, 2011 1:37 pm
Contact:

Re: StoreOnce backups slow on V11

Post by darryl »

Doh - I had started a thread earlier today but misses this one:
veeam-backup-replication-f2/very-slow-b ... 73236.html

Also seeing very slow HP StoreOnce NAS (not Catalyst) performance after moving from 9.5 U4 to version 11. Case 04726060 opened without any response so far.

In our case we're doing backup copies to the StoreOnce - straight writes, no synthetics / merges or anything like that - and seeing terrible performance.
LiamGP
Novice
Posts: 6
Liked: never
Joined: Sep 09, 2014 11:14 am
Full Name: Liam Grant
Contact:

Re: StoreOnce backups slow on V11

Post by LiamGP »

Hmm, I have seen issues like this backing up to my StoreOnce too. Unfortunately I also did some network reconfiguration at the same time (upgrade from 1GB to 10GB) I upgraded to v11 and have not been back in the office to revert the network change. I was putting it down to something on the network but now it seems other people are having the same problem, so perhaps it's not something I've done wrong. I'm currently deploying the latest patch, but it seems from someone above that doesn't help.

I'm only using community edition so have not logged a support case or anything, but will follow this thread for updates.

My symptoms are the same, just extremely slow speeds.
KT752
Novice
Posts: 5
Liked: 1 time
Joined: Mar 31, 2021 2:25 pm
Full Name: Kortenay Templeton

Re: StoreOnce backups slow on V11

Post by KT752 »

I am experiencing the same issues since updating to 11. Case #04718891. Disk and Tape Backups crawl for hours and days. Never completing or just taking long amounts of time. On Veeam 10 I had no issues. Quick backups. Using StoreOnce as well.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: StoreOnce backups slow on V11

Post by Andreas Neufert »

Can you please check if on the Repository Advanced settings the Block Allignment is disabled?
KT752
Novice
Posts: 5
Liked: 1 time
Joined: Mar 31, 2021 2:25 pm
Full Name: Kortenay Templeton

Re: StoreOnce backups slow on V11

Post by KT752 »

I can confirm that option is not checked.
darryl
Enthusiast
Posts: 35
Liked: 4 times
Joined: May 11, 2011 1:37 pm
Contact:

Re: StoreOnce backups slow on V11

Post by darryl »

Anyone get anywhere with this issue?

In our case we spool backup copies to the Store Once, and they're so slow now that the next backup job kicks off before the previous backup copy has completed.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: StoreOnce backups slow on V11

Post by veremin »

Do you have a case opened for this issue so we can ask QA team to follow the investigation? Thanks!
darryl
Enthusiast
Posts: 35
Liked: 4 times
Joined: May 11, 2011 1:37 pm
Contact:

Re: StoreOnce backups slow on V11

Post by darryl »

Yes, 04726060.
KT752
Novice
Posts: 5
Liked: 1 time
Joined: Mar 31, 2021 2:25 pm
Full Name: Kortenay Templeton

Re: StoreOnce backups slow on V11

Post by KT752 »

Does anyone have any new information? I am now looking at downgrading to version 10. I removed parallel processing. Tape backups proceed at just a trickle of speed. No hardware or network changes were performed. Just curious if anyone else in considering a downgrade or has done one. Thank you.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: StoreOnce backups slow on V11

Post by veremin »

What was the resolution of the support case you'd opened? Have support team come to any conclusion? Thanks!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: StoreOnce backups slow on V11

Post by foggy »

We're investigating the slow backup speed to StoreOnce via SMB internally and with customers who have opened support cases. I will report back when we have any valuable info to share.
KT752
Novice
Posts: 5
Liked: 1 time
Joined: Mar 31, 2021 2:25 pm
Full Name: Kortenay Templeton

Re: StoreOnce backups slow on V11

Post by KT752 »

Nothing conclusive yet. I'm still working with the case and Veeam support. I will update when I have something concrete. Thank you
darryl
Enthusiast
Posts: 35
Liked: 4 times
Joined: May 11, 2011 1:37 pm
Contact:

Re: StoreOnce backups slow on V11

Post by darryl »

KT752 wrote: Apr 04, 2021 1:52 pm Does anyone have any new information? I am now looking at downgrading to version 10. I removed parallel processing. Tape backups proceed at just a trickle of speed. No hardware or network changes were performed. Just curious if anyone else in considering a downgrade or has done one. Thank you.
Is it actually possible to downgrade?

I'm stuck here, backup copies won't complete as they're processing at 1/30th the speed that they did with 9.5U4.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: StoreOnce backups slow on V11

Post by HannesK »

downgrade is possible if you import an old configuration backup to an earlier version and rescan everything.

But I would change to a recommended setup and use catalyst.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: StoreOnce backups slow on V11

Post by foggy »

By an old configuration, Hannes is meaning the one created by v10 instance as importing a configuration of a newer version is not supported.
darryl
Enthusiast
Posts: 35
Liked: 4 times
Joined: May 11, 2011 1:37 pm
Contact:

Re: StoreOnce backups slow on V11

Post by darryl »

HannesK wrote: Apr 07, 2021 7:10 am downgrade is possible if you import an old configuration backup to an earlier version and rescan everything.

But I would change to a recommended setup and use catalyst.
...which requires a Catalyst license on the StoreOnce side.
FedericoV
Technology Partner
Posts: 35
Liked: 37 times
Joined: Aug 21, 2017 3:27 pm
Full Name: Federico Venier
Contact:

Re: StoreOnce backups slow on V11

Post by FedericoV » 1 person likes this post

Hello, I do not know if this is a specific problem between V11 and StoreOnce SMB, or it is a generic one between V11 and every SMB share (as reported above by PetrN on march 29th)
Catalyst is generally preferable to NAS protocols for multiple optimizations, but if someone wants to remain on NAS protocols, StoreOnce also support NFS.
It is worth to see whether NFS does not trigger the same (yet unknown) issue
darryl
Enthusiast
Posts: 35
Liked: 4 times
Joined: May 11, 2011 1:37 pm
Contact:

Re: StoreOnce backups slow on V11

Post by darryl »

NFS - great idea FedericoV, I totally forgot the StoreOnce supports that.

I'm running a test now. Unfortunately that requires creating a new empty share as the SO doesn't allow changing the access protocol on an existing share. So I'm writing to an empty share where there is no dedupe to process, so take my results so far for what they are.
I was seeing around 40 MB/sec on ver 9.5 via SMB to StoreOnce.
Dropped to 1 MB/sec on ver 11 via SMB to StoreOnce.
Seeing around 60 MB/sec on ver 11 via NFS to StoreOnce.

I'll do more testing.
Unfortunately I haven't gotten anything useful from the Veeam support case that I have open.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 151 guests