StoreOnce backups slow on V11

[mr_iang]

Hi,
I have Case # 04688930 open with support, it is really dragging on unfortunately and I'm reaching out to the community to see if anyone else has run in to this or has tips. I think it is specific to the Storeonce given my troubleshooting so far.

Backstory:
We were on Veeam 9.5U4 with vSphere 6.7, running two Veeam servers, the secondary just as a proxy. The primary server only had access to the backup device via CIFS on a dual 10gig connection -we were not using Catalyst. I'd followed the setup recommendations so that dedupe was handled by the appliance, in general this has been working well for at least 6 months. We had been planning on an upgrade for a while but Veeam just worked, so.. Anyway we did run in to an issue with an instant recovery recently, and during support it was revealed that this issue was addressed in newer versions. So some motivation came in. Given the great track record with Veeam I went all-in and upgraded directly to Version 11.

That night our 40 or so backup jobs for approx 75 VM's started up and it became clear very quickly that there was a problem. The jobs reported the bottleneck as the target. I took a look at the setup and it seemed like maybe they were having an issue because the secondary proxy server didnt have CIFS access to the backup appliance, so I modified the job proxy preferences accordingly. This made no difference. I even opened up access to the CIFS share from the proxy to no avail. Looking through job logs, even a single job running at a time was getting an average of 5MB/s and even then the little traffic graph would simply stop for periods. Our jobs piled up and I had to abort them for the next day.

After giving up on Veeam11 because we were falling behind on backups I installed Veeam 10 on another server and rebuilt the jobs from scratch. That was a pain.. and unfortunately we didnt have a snapshot of the backup server to roll back on, this just seemed like the best way to get backups caught up. Of course then I had to redo many base images as some of the backups had completed and now were a newer version than my V10 server supported.

Once caught up I took a snapshot of the backup server and did the Veeam 11 upgrade then tried backups on a couple jobs. Same issue as before (keep in mind this is a completely new backup server). Jobs which ran 100+ MB/s previously were struggling around 5 megs. I pulled logs out and reverted the snapshot. Backups are running smoothly on Veeam10, but we need a solution here..

Cheers,
Ian

[HannesK]

Hello,
if you say that it works faster with 9.5U4 and V10 than with V11, that sounds strange. I can only recommend to continue with support to find the reason.

75 VMs should be a hand full of jobs. StoreOnce should use Catalyst integration. For dedupe appliance without integration (SMB in your case), active full is the way to go. I also assume that you have configured per-machine backup chains and active full backups https://bp.veeam.com/vbr/VBP/3_Build_st ... appliances

In general, the setup is against the best practices. So having slow speed in general is expected. Having target as bottleneck is no problem in general. Please see Forum FAQ. http://forums.veeam.com/veeam-backup-re ... tml#p95291 And maybe ignore whatever V11 bottleneck analysis says, because there might be a bug in there. Only look at the overall backup time (and only compare similar settings)

Best regards,
Hannes

[PetrN]

Hi, I have same problem after update Veeam to Veeam 11. Storeonce CIFS are terrible slow. Arround 5MB/s! Catalyst works but CIFS are faster in version 10. Case #04699964

[HannesK]

Hello Petr,
in your case, there seem to be issues on the StoreOnce side. Please continue investigations with support.

In general, just because of the overall amount of customers with StoreOnce systems, there is a statistically significant chance that something looks related to the upgrade that would also happen without upgrade.

For everyone else with the same issue: please contact support and post the case number here. We need to find out whether it's really an issue on Veeam side just just random StoreOnce issues, that happened with "bad timing"

Thanks,
Hannes

PS: ah, and use catalyst. Inline dedupe appliances perform poor in general. Using them with SMB makes the performance worse.

[soncscy]

Out of curiosity, what keeps you using the SMB shares? (remember, CIFS is dead)

I know that SMBSigning hits Storeonce devices hard hard so ensure that's disabled on your gateway. I would strongly advise that if you're pushing SMB shares across WAN that you immediately rethink your strategy as this is incredibly poor design.

Does the cache bypass introduced by v11 also apply to SMB? Maybe that's the issue.

[m.mohan@al-dhow.com]

Case ID# 04715674
Hi... I too have same issue, I have upgraded to version 11 and backup time increased to thrice. Its very strange, same storage, same data size but backup time is terrible. Raised a ticket - Case ID# 04715674 and they asked to install patch (KB4126), which doesn't help even and waiting for Veaam support team. For me incremental backup time/one time full backup .. all jobs having same issue. If anyone found solution, please share..

[foggy]

Hi Murali, just to confirm - is it SMB as well (no Catalyst)?

[m.mohan@al-dhow.com]

@foggy .. Yes It is SMB ( Local disk on backup server)

[HannesK]

Hello Murali,
this thread is about HPE StoreOnce and SMB.

If you really have an SMB share on the backup server, then this is a different scenario. Your scenario should be fixed by using "direct attached storage" instead of SMB protocol.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110
https://helpcenter.veeam.com/docs/backu ... 110#server


Best regards,
Hannes

[foggy]

Might be a general SMB issue, not StoreOnce-specific, we're investigating.

[PetrN]

Hello Petr,
in your case, there seem to be issues on the StoreOnce side. Please continue investigations with support.

In general, just because of the overall amount of customers with StoreOnce systems, there is a statistically significant chance that something looks related to the upgrade that would also happen without upgrade.

For everyone else with the same issue: please contact support and post the case number here. We need to find out whether it's really an issue on Veeam side just just random StoreOnce issues, that happened with "bad timing"

Thanks,
Hannes

PS: ah, and use catalyst. Inline dedupe appliances perform poor in general. Using them with SMB makes the performance worse.

Hi Hannes,

I tried to save data to another CIFS share to another data center in another repository that shows no errors and the same 5MB / s problem. Additionally, storage file errors are for Veeam-managed tasks only. Rather, it is a question of whether Veeam11 caused consistency errors due to its slow writing.

[darryl]

Support case 04726060

Our environment:
Single physical backup server, 192 GB, 24/48 cores
Backing up mostly ESXi VMs, some Hyper-V
FC connected SAN repository

We do backup copies to a HP StoreOnce 4500 share (no catalyst). We set the backup copies for 99 increments, and just start a new backup copy chain every month. We do this to avoid any random or read I/O from the StoreOnce, which would be brutal. Each chain ends up being the original VBK and 30 or so VIBs.

We've been doing this since 2015, across various Veeam versions, no issue. Not fancy but it works.

Upgraded from 9.5 U4 to 11 20210319 and backup copy jobs have terrible performance now. Like 40 MB/sec processing down to 1 MB/sec. So backup copy jobs aren't completing.

Before upgrade, on 9.5, backup copy jobs show 10 percent source, 99 percent target.
After upgrade, on 11, backup copy jobs show 50 percent source, 99 percent target.

I'm wondering if there's been a behaviour change that's affecting I/O on my target (is it being read for some reason?).

I opened a case yesterday with no response. I'm open to suggestions of where to start looking.

[darryl]

Doh - I had started a thread earlier today but misses this one:
veeam-backup-replication-f2/very-slow-b ... 73236.html

Also seeing very slow HP StoreOnce NAS (not Catalyst) performance after moving from 9.5 U4 to version 11. Case 04726060 opened without any response so far.

In our case we're doing backup copies to the StoreOnce - straight writes, no synthetics / merges or anything like that - and seeing terrible performance.

[LiamGP]

Hmm, I have seen issues like this backing up to my StoreOnce too. Unfortunately I also did some network reconfiguration at the same time (upgrade from 1GB to 10GB) I upgraded to v11 and have not been back in the office to revert the network change. I was putting it down to something on the network but now it seems other people are having the same problem, so perhaps it's not something I've done wrong. I'm currently deploying the latest patch, but it seems from someone above that doesn't help.

I'm only using community edition so have not logged a support case or anything, but will follow this thread for updates.

My symptoms are the same, just extremely slow speeds.

[KT752]

I am experiencing the same issues since updating to 11. Case #04718891. Disk and Tape Backups crawl for hours and days. Never completing or just taking long amounts of time. On Veeam 10 I had no issues. Quick backups. Using StoreOnce as well.

[Andreas Neufert]

Can you please check if on the Repository Advanced settings the Block Allignment is disabled?

[KT752]

I can confirm that option is not checked.

[darryl]

Anyone get anywhere with this issue?

In our case we spool backup copies to the Store Once, and they're so slow now that the next backup job kicks off before the previous backup copy has completed.

[veremin]

Do you have a case opened for this issue so we can ask QA team to follow the investigation? Thanks!

[darryl]

Yes, 04726060.

[KT752]

Does anyone have any new information? I am now looking at downgrading to version 10. I removed parallel processing. Tape backups proceed at just a trickle of speed. No hardware or network changes were performed. Just curious if anyone else in considering a downgrade or has done one. Thank you.

[veremin]

What was the resolution of the support case you'd opened? Have support team come to any conclusion? Thanks!

[foggy]

We're investigating the slow backup speed to StoreOnce via SMB internally and with customers who have opened support cases. I will report back when we have any valuable info to share.

[KT752]

Nothing conclusive yet. I'm still working with the case and Veeam support. I will update when I have something concrete. Thank you

[darryl]

Does anyone have any new information? I am now looking at downgrading to version 10. I removed parallel processing. Tape backups proceed at just a trickle of speed. No hardware or network changes were performed. Just curious if anyone else in considering a downgrade or has done one. Thank you.

Is it actually possible to downgrade?

I'm stuck here, backup copies won't complete as they're processing at 1/30th the speed that they did with 9.5U4.

[HannesK]

downgrade is possible if you import an old configuration backup to an earlier version and rescan everything.

But I would change to a recommended setup and use catalyst.

[foggy]

By an old configuration, Hannes is meaning the one created by v10 instance as importing a configuration of a newer version is not supported.

[darryl]

downgrade is possible if you import an old configuration backup to an earlier version and rescan everything.

But I would change to a recommended setup and use catalyst.

...which requires a Catalyst license on the StoreOnce side.

[FedericoV]

Hello, I do not know if this is a specific problem between V11 and StoreOnce SMB, or it is a generic one between V11 and every SMB share (as reported above by PetrN on march 29th)
Catalyst is generally preferable to NAS protocols for multiple optimizations, but if someone wants to remain on NAS protocols, StoreOnce also support NFS.
It is worth to see whether NFS does not trigger the same (yet unknown) issue

[darryl]

NFS - great idea FedericoV, I totally forgot the StoreOnce supports that.

I'm running a test now. Unfortunately that requires creating a new empty share as the SO doesn't allow changing the access protocol on an existing share. So I'm writing to an empty share where there is no dedupe to process, so take my results so far for what they are.
I was seeing around 40 MB/sec on ver 9.5 via SMB to StoreOnce.
Dropped to 1 MB/sec on ver 11 via SMB to StoreOnce.
Seeing around 60 MB/sec on ver 11 via NFS to StoreOnce.

I'll do more testing.
Unfortunately I haven't gotten anything useful from the Veeam support case that I have open.