-
- Novice
- Posts: 3
- Liked: never
- Joined: Dec 12, 2016 8:09 pm
- Full Name: Ian Gorman
- Contact:
StoreOnce backups slow on V11
Hi,
I have Case # 04688930 open with support, it is really dragging on unfortunately and I'm reaching out to the community to see if anyone else has run in to this or has tips. I think it is specific to the Storeonce given my troubleshooting so far.
Backstory:
We were on Veeam 9.5U4 with vSphere 6.7, running two Veeam servers, the secondary just as a proxy. The primary server only had access to the backup device via CIFS on a dual 10gig connection -we were not using Catalyst. I'd followed the setup recommendations so that dedupe was handled by the appliance, in general this has been working well for at least 6 months. We had been planning on an upgrade for a while but Veeam just worked, so.. Anyway we did run in to an issue with an instant recovery recently, and during support it was revealed that this issue was addressed in newer versions. So some motivation came in. Given the great track record with Veeam I went all-in and upgraded directly to Version 11.
That night our 40 or so backup jobs for approx 75 VM's started up and it became clear very quickly that there was a problem. The jobs reported the bottleneck as the target. I took a look at the setup and it seemed like maybe they were having an issue because the secondary proxy server didnt have CIFS access to the backup appliance, so I modified the job proxy preferences accordingly. This made no difference. I even opened up access to the CIFS share from the proxy to no avail. Looking through job logs, even a single job running at a time was getting an average of 5MB/s and even then the little traffic graph would simply stop for periods. Our jobs piled up and I had to abort them for the next day.
After giving up on Veeam11 because we were falling behind on backups I installed Veeam 10 on another server and rebuilt the jobs from scratch. That was a pain.. and unfortunately we didnt have a snapshot of the backup server to roll back on, this just seemed like the best way to get backups caught up. Of course then I had to redo many base images as some of the backups had completed and now were a newer version than my V10 server supported.
Once caught up I took a snapshot of the backup server and did the Veeam 11 upgrade then tried backups on a couple jobs. Same issue as before (keep in mind this is a completely new backup server). Jobs which ran 100+ MB/s previously were struggling around 5 megs. I pulled logs out and reverted the snapshot. Backups are running smoothly on Veeam10, but we need a solution here..
Cheers,
Ian
I have Case # 04688930 open with support, it is really dragging on unfortunately and I'm reaching out to the community to see if anyone else has run in to this or has tips. I think it is specific to the Storeonce given my troubleshooting so far.
Backstory:
We were on Veeam 9.5U4 with vSphere 6.7, running two Veeam servers, the secondary just as a proxy. The primary server only had access to the backup device via CIFS on a dual 10gig connection -we were not using Catalyst. I'd followed the setup recommendations so that dedupe was handled by the appliance, in general this has been working well for at least 6 months. We had been planning on an upgrade for a while but Veeam just worked, so.. Anyway we did run in to an issue with an instant recovery recently, and during support it was revealed that this issue was addressed in newer versions. So some motivation came in. Given the great track record with Veeam I went all-in and upgraded directly to Version 11.
That night our 40 or so backup jobs for approx 75 VM's started up and it became clear very quickly that there was a problem. The jobs reported the bottleneck as the target. I took a look at the setup and it seemed like maybe they were having an issue because the secondary proxy server didnt have CIFS access to the backup appliance, so I modified the job proxy preferences accordingly. This made no difference. I even opened up access to the CIFS share from the proxy to no avail. Looking through job logs, even a single job running at a time was getting an average of 5MB/s and even then the little traffic graph would simply stop for periods. Our jobs piled up and I had to abort them for the next day.
After giving up on Veeam11 because we were falling behind on backups I installed Veeam 10 on another server and rebuilt the jobs from scratch. That was a pain.. and unfortunately we didnt have a snapshot of the backup server to roll back on, this just seemed like the best way to get backups caught up. Of course then I had to redo many base images as some of the backups had completed and now were a newer version than my V10 server supported.
Once caught up I took a snapshot of the backup server and did the Veeam 11 upgrade then tried backups on a couple jobs. Same issue as before (keep in mind this is a completely new backup server). Jobs which ran 100+ MB/s previously were struggling around 5 megs. I pulled logs out and reverted the snapshot. Backups are running smoothly on Veeam10, but we need a solution here..
Cheers,
Ian
-
- Product Manager
- Posts: 14840
- Liked: 3086 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: StoreOnce backups slow on V11
Hello,
if you say that it works faster with 9.5U4 and V10 than with V11, that sounds strange. I can only recommend to continue with support to find the reason.
75 VMs should be a hand full of jobs. StoreOnce should use Catalyst integration. For dedupe appliance without integration (SMB in your case), active full is the way to go. I also assume that you have configured per-machine backup chains and active full backups https://bp.veeam.com/vbr/VBP/3_Build_st ... appliances
In general, the setup is against the best practices. So having slow speed in general is expected. Having target as bottleneck is no problem in general. Please see Forum FAQ. http://forums.veeam.com/veeam-backup-re ... tml#p95291 And maybe ignore whatever V11 bottleneck analysis says, because there might be a bug in there. Only look at the overall backup time (and only compare similar settings)
Best regards,
Hannes
if you say that it works faster with 9.5U4 and V10 than with V11, that sounds strange. I can only recommend to continue with support to find the reason.
75 VMs should be a hand full of jobs. StoreOnce should use Catalyst integration. For dedupe appliance without integration (SMB in your case), active full is the way to go. I also assume that you have configured per-machine backup chains and active full backups https://bp.veeam.com/vbr/VBP/3_Build_st ... appliances
In general, the setup is against the best practices. So having slow speed in general is expected. Having target as bottleneck is no problem in general. Please see Forum FAQ. http://forums.veeam.com/veeam-backup-re ... tml#p95291 And maybe ignore whatever V11 bottleneck analysis says, because there might be a bug in there. Only look at the overall backup time (and only compare similar settings)
Best regards,
Hannes
-
- Novice
- Posts: 3
- Liked: never
- Joined: Mar 27, 2021 8:37 am
- Full Name: Petr Neumann
- Contact:
Re: StoreOnce backups slow on V11
Hi, I have same problem after update Veeam to Veeam 11. Storeonce CIFS are terrible slow. Arround 5MB/s! Catalyst works but CIFS are faster in version 10. Case #04699964
-
- Product Manager
- Posts: 14840
- Liked: 3086 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: StoreOnce backups slow on V11
Hello Petr,
in your case, there seem to be issues on the StoreOnce side. Please continue investigations with support.
In general, just because of the overall amount of customers with StoreOnce systems, there is a statistically significant chance that something looks related to the upgrade that would also happen without upgrade.
For everyone else with the same issue: please contact support and post the case number here. We need to find out whether it's really an issue on Veeam side just just random StoreOnce issues, that happened with "bad timing"
Thanks,
Hannes
PS: ah, and use catalyst. Inline dedupe appliances perform poor in general. Using them with SMB makes the performance worse.
in your case, there seem to be issues on the StoreOnce side. Please continue investigations with support.
In general, just because of the overall amount of customers with StoreOnce systems, there is a statistically significant chance that something looks related to the upgrade that would also happen without upgrade.
For everyone else with the same issue: please contact support and post the case number here. We need to find out whether it's really an issue on Veeam side just just random StoreOnce issues, that happened with "bad timing"
Thanks,
Hannes
PS: ah, and use catalyst. Inline dedupe appliances perform poor in general. Using them with SMB makes the performance worse.
-
- Veteran
- Posts: 643
- Liked: 312 times
- Joined: Aug 04, 2019 2:57 pm
- Full Name: Harvey
- Contact:
Re: StoreOnce backups slow on V11
Out of curiosity, what keeps you using the SMB shares? (remember, CIFS is dead)
I know that SMBSigning hits Storeonce devices hard hard so ensure that's disabled on your gateway. I would strongly advise that if you're pushing SMB shares across WAN that you immediately rethink your strategy as this is incredibly poor design.
Does the cache bypass introduced by v11 also apply to SMB? Maybe that's the issue.
I know that SMBSigning hits Storeonce devices hard hard so ensure that's disabled on your gateway. I would strongly advise that if you're pushing SMB shares across WAN that you immediately rethink your strategy as this is incredibly poor design.
Does the cache bypass introduced by v11 also apply to SMB? Maybe that's the issue.
-
- Novice
- Posts: 3
- Liked: never
- Joined: Mar 27, 2021 1:39 pm
- Full Name: Murali Mohan
- Contact:
Re: StoreOnce backups slow on V11
Case ID# 04715674
Hi... I too have same issue, I have upgraded to version 11 and backup time increased to thrice. Its very strange, same storage, same data size but backup time is terrible. Raised a ticket - Case ID# 04715674 and they asked to install patch (KB4126), which doesn't help even and waiting for Veaam support team. For me incremental backup time/one time full backup .. all jobs having same issue. If anyone found solution, please share..
Hi... I too have same issue, I have upgraded to version 11 and backup time increased to thrice. Its very strange, same storage, same data size but backup time is terrible. Raised a ticket - Case ID# 04715674 and they asked to install patch (KB4126), which doesn't help even and waiting for Veaam support team. For me incremental backup time/one time full backup .. all jobs having same issue. If anyone found solution, please share..
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: StoreOnce backups slow on V11
Hi Murali, just to confirm - is it SMB as well (no Catalyst)?
-
- Novice
- Posts: 3
- Liked: never
- Joined: Mar 27, 2021 1:39 pm
- Full Name: Murali Mohan
- Contact:
Re: StoreOnce backups slow on V11
@foggy .. Yes It is SMB ( Local disk on backup server)
-
- Product Manager
- Posts: 14840
- Liked: 3086 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: StoreOnce backups slow on V11
Hello Murali,
this thread is about HPE StoreOnce and SMB.
If you really have an SMB share on the backup server, then this is a different scenario. Your scenario should be fixed by using "direct attached storage" instead of SMB protocol.
https://helpcenter.veeam.com/docs/backu ... ml?ver=110
https://helpcenter.veeam.com/docs/backu ... 110#server
Best regards,
Hannes
this thread is about HPE StoreOnce and SMB.
If you really have an SMB share on the backup server, then this is a different scenario. Your scenario should be fixed by using "direct attached storage" instead of SMB protocol.
https://helpcenter.veeam.com/docs/backu ... ml?ver=110
https://helpcenter.veeam.com/docs/backu ... 110#server
Best regards,
Hannes
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: StoreOnce backups slow on V11
Might be a general SMB issue, not StoreOnce-specific, we're investigating.
-
- Novice
- Posts: 3
- Liked: never
- Joined: Mar 27, 2021 8:37 am
- Full Name: Petr Neumann
- Contact:
Re: StoreOnce backups slow on V11
Hi Hannes,HannesK wrote: ↑Mar 27, 2021 6:26 pm Hello Petr,
in your case, there seem to be issues on the StoreOnce side. Please continue investigations with support.
In general, just because of the overall amount of customers with StoreOnce systems, there is a statistically significant chance that something looks related to the upgrade that would also happen without upgrade.
For everyone else with the same issue: please contact support and post the case number here. We need to find out whether it's really an issue on Veeam side just just random StoreOnce issues, that happened with "bad timing"
Thanks,
Hannes
PS: ah, and use catalyst. Inline dedupe appliances perform poor in general. Using them with SMB makes the performance worse.
I tried to save data to another CIFS share to another data center in another repository that shows no errors and the same 5MB / s problem. Additionally, storage file errors are for Veeam-managed tasks only. Rather, it is a question of whether Veeam11 caused consistency errors due to its slow writing.
-
- Enthusiast
- Posts: 36
- Liked: 4 times
- Joined: May 11, 2011 1:37 pm
- Contact:
[MERGED] Very slow backup copy to StoreOnce after upgrade from 9.5 U4 to 11
Support case 04726060
Our environment:
Single physical backup server, 192 GB, 24/48 cores
Backing up mostly ESXi VMs, some Hyper-V
FC connected SAN repository
We do backup copies to a HP StoreOnce 4500 share (no catalyst). We set the backup copies for 99 increments, and just start a new backup copy chain every month. We do this to avoid any random or read I/O from the StoreOnce, which would be brutal. Each chain ends up being the original VBK and 30 or so VIBs.
We've been doing this since 2015, across various Veeam versions, no issue. Not fancy but it works.
Upgraded from 9.5 U4 to 11 20210319 and backup copy jobs have terrible performance now. Like 40 MB/sec processing down to 1 MB/sec. So backup copy jobs aren't completing.
Before upgrade, on 9.5, backup copy jobs show 10 percent source, 99 percent target.
After upgrade, on 11, backup copy jobs show 50 percent source, 99 percent target.
I'm wondering if there's been a behaviour change that's affecting I/O on my target (is it being read for some reason?).
I opened a case yesterday with no response. I'm open to suggestions of where to start looking.
Our environment:
Single physical backup server, 192 GB, 24/48 cores
Backing up mostly ESXi VMs, some Hyper-V
FC connected SAN repository
We do backup copies to a HP StoreOnce 4500 share (no catalyst). We set the backup copies for 99 increments, and just start a new backup copy chain every month. We do this to avoid any random or read I/O from the StoreOnce, which would be brutal. Each chain ends up being the original VBK and 30 or so VIBs.
We've been doing this since 2015, across various Veeam versions, no issue. Not fancy but it works.
Upgraded from 9.5 U4 to 11 20210319 and backup copy jobs have terrible performance now. Like 40 MB/sec processing down to 1 MB/sec. So backup copy jobs aren't completing.
Before upgrade, on 9.5, backup copy jobs show 10 percent source, 99 percent target.
After upgrade, on 11, backup copy jobs show 50 percent source, 99 percent target.
I'm wondering if there's been a behaviour change that's affecting I/O on my target (is it being read for some reason?).
I opened a case yesterday with no response. I'm open to suggestions of where to start looking.
-
- Enthusiast
- Posts: 36
- Liked: 4 times
- Joined: May 11, 2011 1:37 pm
- Contact:
Re: StoreOnce backups slow on V11
Doh - I had started a thread earlier today but misses this one:
veeam-backup-replication-f2/very-slow-b ... 73236.html
Also seeing very slow HP StoreOnce NAS (not Catalyst) performance after moving from 9.5 U4 to version 11. Case 04726060 opened without any response so far.
In our case we're doing backup copies to the StoreOnce - straight writes, no synthetics / merges or anything like that - and seeing terrible performance.
veeam-backup-replication-f2/very-slow-b ... 73236.html
Also seeing very slow HP StoreOnce NAS (not Catalyst) performance after moving from 9.5 U4 to version 11. Case 04726060 opened without any response so far.
In our case we're doing backup copies to the StoreOnce - straight writes, no synthetics / merges or anything like that - and seeing terrible performance.
-
- Novice
- Posts: 6
- Liked: never
- Joined: Sep 09, 2014 11:14 am
- Full Name: Liam Grant
- Contact:
Re: StoreOnce backups slow on V11
Hmm, I have seen issues like this backing up to my StoreOnce too. Unfortunately I also did some network reconfiguration at the same time (upgrade from 1GB to 10GB) I upgraded to v11 and have not been back in the office to revert the network change. I was putting it down to something on the network but now it seems other people are having the same problem, so perhaps it's not something I've done wrong. I'm currently deploying the latest patch, but it seems from someone above that doesn't help.
I'm only using community edition so have not logged a support case or anything, but will follow this thread for updates.
My symptoms are the same, just extremely slow speeds.
I'm only using community edition so have not logged a support case or anything, but will follow this thread for updates.
My symptoms are the same, just extremely slow speeds.
-
- Novice
- Posts: 5
- Liked: 1 time
- Joined: Mar 31, 2021 2:25 pm
- Full Name: Kortenay Templeton
Re: StoreOnce backups slow on V11
I am experiencing the same issues since updating to 11. Case #04718891. Disk and Tape Backups crawl for hours and days. Never completing or just taking long amounts of time. On Veeam 10 I had no issues. Quick backups. Using StoreOnce as well.
-
- VP, Product Management
- Posts: 7081
- Liked: 1511 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: StoreOnce backups slow on V11
Can you please check if on the Repository Advanced settings the Block Allignment is disabled?
-
- Novice
- Posts: 5
- Liked: 1 time
- Joined: Mar 31, 2021 2:25 pm
- Full Name: Kortenay Templeton
Re: StoreOnce backups slow on V11
I can confirm that option is not checked.
-
- Enthusiast
- Posts: 36
- Liked: 4 times
- Joined: May 11, 2011 1:37 pm
- Contact:
Re: StoreOnce backups slow on V11
Anyone get anywhere with this issue?
In our case we spool backup copies to the Store Once, and they're so slow now that the next backup job kicks off before the previous backup copy has completed.
In our case we spool backup copies to the Store Once, and they're so slow now that the next backup job kicks off before the previous backup copy has completed.
-
- Product Manager
- Posts: 20413
- Liked: 2302 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: StoreOnce backups slow on V11
Do you have a case opened for this issue so we can ask QA team to follow the investigation? Thanks!
-
- Enthusiast
- Posts: 36
- Liked: 4 times
- Joined: May 11, 2011 1:37 pm
- Contact:
Re: StoreOnce backups slow on V11
Yes, 04726060.
-
- Novice
- Posts: 5
- Liked: 1 time
- Joined: Mar 31, 2021 2:25 pm
- Full Name: Kortenay Templeton
Re: StoreOnce backups slow on V11
Does anyone have any new information? I am now looking at downgrading to version 10. I removed parallel processing. Tape backups proceed at just a trickle of speed. No hardware or network changes were performed. Just curious if anyone else in considering a downgrade or has done one. Thank you.
-
- Product Manager
- Posts: 20413
- Liked: 2302 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: StoreOnce backups slow on V11
What was the resolution of the support case you'd opened? Have support team come to any conclusion? Thanks!
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: StoreOnce backups slow on V11
We're investigating the slow backup speed to StoreOnce via SMB internally and with customers who have opened support cases. I will report back when we have any valuable info to share.
-
- Novice
- Posts: 5
- Liked: 1 time
- Joined: Mar 31, 2021 2:25 pm
- Full Name: Kortenay Templeton
Re: StoreOnce backups slow on V11
Nothing conclusive yet. I'm still working with the case and Veeam support. I will update when I have something concrete. Thank you
-
- Enthusiast
- Posts: 36
- Liked: 4 times
- Joined: May 11, 2011 1:37 pm
- Contact:
Re: StoreOnce backups slow on V11
Is it actually possible to downgrade?KT752 wrote: ↑Apr 04, 2021 1:52 pm Does anyone have any new information? I am now looking at downgrading to version 10. I removed parallel processing. Tape backups proceed at just a trickle of speed. No hardware or network changes were performed. Just curious if anyone else in considering a downgrade or has done one. Thank you.
I'm stuck here, backup copies won't complete as they're processing at 1/30th the speed that they did with 9.5U4.
-
- Product Manager
- Posts: 14840
- Liked: 3086 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: StoreOnce backups slow on V11
downgrade is possible if you import an old configuration backup to an earlier version and rescan everything.
But I would change to a recommended setup and use catalyst.
But I would change to a recommended setup and use catalyst.
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: StoreOnce backups slow on V11
By an old configuration, Hannes is meaning the one created by v10 instance as importing a configuration of a newer version is not supported.
-
- Enthusiast
- Posts: 36
- Liked: 4 times
- Joined: May 11, 2011 1:37 pm
- Contact:
-
- Technology Partner
- Posts: 36
- Liked: 38 times
- Joined: Aug 21, 2017 3:27 pm
- Full Name: Federico Venier
- Contact:
Re: StoreOnce backups slow on V11
Hello, I do not know if this is a specific problem between V11 and StoreOnce SMB, or it is a generic one between V11 and every SMB share (as reported above by PetrN on march 29th)
Catalyst is generally preferable to NAS protocols for multiple optimizations, but if someone wants to remain on NAS protocols, StoreOnce also support NFS.
It is worth to see whether NFS does not trigger the same (yet unknown) issue
Catalyst is generally preferable to NAS protocols for multiple optimizations, but if someone wants to remain on NAS protocols, StoreOnce also support NFS.
It is worth to see whether NFS does not trigger the same (yet unknown) issue
-
- Enthusiast
- Posts: 36
- Liked: 4 times
- Joined: May 11, 2011 1:37 pm
- Contact:
Re: StoreOnce backups slow on V11
NFS - great idea FedericoV, I totally forgot the StoreOnce supports that.
I'm running a test now. Unfortunately that requires creating a new empty share as the SO doesn't allow changing the access protocol on an existing share. So I'm writing to an empty share where there is no dedupe to process, so take my results so far for what they are.
I was seeing around 40 MB/sec on ver 9.5 via SMB to StoreOnce.
Dropped to 1 MB/sec on ver 11 via SMB to StoreOnce.
Seeing around 60 MB/sec on ver 11 via NFS to StoreOnce.
I'll do more testing.
Unfortunately I haven't gotten anything useful from the Veeam support case that I have open.
I'm running a test now. Unfortunately that requires creating a new empty share as the SO doesn't allow changing the access protocol on an existing share. So I'm writing to an empty share where there is no dedupe to process, so take my results so far for what they are.
I was seeing around 40 MB/sec on ver 9.5 via SMB to StoreOnce.
Dropped to 1 MB/sec on ver 11 via SMB to StoreOnce.
Seeing around 60 MB/sec on ver 11 via NFS to StoreOnce.
I'll do more testing.
Unfortunately I haven't gotten anything useful from the Veeam support case that I have open.
Who is online
Users browsing this forum: No registered users and 89 guests