Backup jobs speed become very slowly when the Copy Jobs starts after we upgrade our Veeam from 9.5 to V11

[apolloxm]

Hello,
Case ID:04968629

Recently, We upgraded our Veeam Backup Server from 9.5 U4 to V11. Our Backup jobs with agent become very slowly(from 300 MB/s to 10MB/s) when the copy jobs started. the issue happened by every week.

Our environment:
For the backup Jobs,We are using the QNAP ISCSI as our Backup Job repository. For the backup copy jobs, We are using the Synology SMB Share as Backup Copy Jobs Repository.
We have two backup jobs with agent. One's name as A, the other's name as B. A was running as an active full and started at 11:30PM on Sunday, B was running as an incremental and started at the 11:50PM on the Sunday. We are starting to run the backup copy jobs at 12:00AM on the Monday. The Backup Copy Jobs was triggered by the Powershell. We set the concurrent task to 6 for the backup copy jobs. In the Veeam 9.5 U4, all of the Backup jobs and Backup Copy jobs were running well without any issue. After We upgraded to V11. When all the Backup Copy Jobs started, sometimes Job A or Job B's backup speed bacome very slowly.the speed can be reached to 300 MB/s, before the backup copy jobs started, the speed down to 10 MB/s after the backup copy jobs stared. I tried to disabled all the Backup Copy Jobs, then the backup speed of A job or B Job back to 300 MB/s.


We saw similar customers having issues with backups going to SMB/CIFS share repository after upgrading to V11. but it looks like the behavior is difference compare with our environment.
To alleviate those issues, we created the below registry key on the Veeam backup server and CIFS :
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
UseUnbufferedAccess
• Type: REG_DWORD
• Set it to 0

We have opened a ticket, Case ID:04968629.Your support engineer found that bottleneck on the Target Server (SOBR repository server),but they don't know why the speed become slowly when the backup copy jobs strated, Can someone help me on this?

EDIT moderator: removed logs

Powershell,
$jobs = Get-VBRJob | ?{$_.JobType -eq "backupsync"}
foreach($job in $jobs){
$jobsname=$job.name
Sync-VBRBAckupCopyJob -Job $job -FullBackup

[HannesK]

Hello,
I can only recommend to continue working with support if the solutions you already found on the forums for SMB shares don't work (there are some more registry keys than the one you mentioned).

In general, it's normal (especially with low-end NAS systems) that write speed goes down if there is also something else reading from the storage at the same time. The only thing to research is, why there is a slow-down between 9.4U4 an V11.

Just to be 100% sure: if you ran backup jobs and backup copy jobs at the same in 9.5U4, then the speed was constantly around 300MByte/s? And did you change anything else? I remember from another thread that you added capacity tier in copy mode which also reads data.

We set the concurrent task to 6 for the backup copy jobs

can you explain where that is? Tasks are a repository setting (not on the BCJ). Did you change that between 9.5U4 and V11?

Best regards,
Hannes

[apolloxm]

Q:Can you explain where that is? Tasks are a repository setting (not on the BCJ). Did you change that between 9.5U4 and V11?
Answer:Yes, We set the concurrent task to 6 on the repository setting.

Q:Just to be 100% sure: if you ran backup jobs and backup copy jobs at the same in 9.5U4, then the speed was constantly around 300MByte/s? And did you change anything else? I remember from another thread that you added capacity tier in copy mode which also reads data.
Answer:No, the only thing that I have changed just upgrade from 9.5 u4 to V11. for the capacity tier, We just tested in our Testlab, We are planning to go-live by this moth,but not now.

[apolloxm]

IN 9.5 U4, everything goes well, the issue happened after we upgraded to V11.

[HannesK]

okay, 6 concurrent tasks might be too much for the hardware. I recommend setting it back to the 9.5U4 value.

If that doesn't help, then I recommend to ask support which other reg keys could help (I remember disabling async read helped in some situations and also block alignment from my link above)

[apolloxm]

The setting is the same with 9.5U4.

I am feeling so bad for your APAC Team,I contacted them several times, Most of time, they can't fix my issue.
The case has been escalated to your support Level 2 Team. Hopefully they can fix the issue. but I am still worrying about that they can't fix my issue,

[pschwarz]

Dear all,
I've got a problem with a Veeam Backup to a Synology NAS Server using a SMB Share. Synology is on the latest Version, SMBv3 is activated, MTU size is 1500, Network Hardware is new (HP Aruba 1930 GBit Switch). Veeam is running within a MS 2019 Server installed on a HP ML350 Server with HPE Customized ESXi 7.0.1 Image version 701.0.0.10.7.0 for HPE Servers based on VMware ESXi 7.0.1. The VM instance is around 11 TB large and when backup is started the transfer rate goes up to 150 - 200 MB/s which is OK. After around 1-2 hours the speed drops down to 8 MB/s and the backup wents on but slow. If I start a -t ping to the NAS I can see that the ping is answered within 1 ms, when the speed drops, the ping is answered between 2000ms an 39ms, also a timeout is possible. I tried everything to fix that and with the information I found in the internet reagarding to this problem, but all didn't help. With older Verisons of ESXi (v6) adn older Versions of Veeam (v9) I never had this problem.
Kind regards, Patrick Schwarz

[HannesK]

Hello,
and welcome to the forums.

As you say, that you did not have that issue with V9, I merged your topic to one that is very similar. I also replaced your support ID with the support case number.

I can only recommend to continue working with support to find out which setting / registry key needs to be changed.

Best regards,
Hannes

[pschwarz]

Hello Hannes,
thank you for the reply.
I also tried the REG Key HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
UseUnbufferedAccess
• Type: REG_DWORD
• Set it to 0
This made the backup a little bit faster. For testing a ran a job on another server of a customer of mine which is using ESXi v6 and Veeam v9 on an HP ML350 installed in 2017. Backup is written to a Synology DS1520 which is not one of the fastest but it's doing it's job. Writespeed to NAS is around 200MB/s which is a normal speed for me and this would be OK.
For the new installation I sold Veeam Essentials to my customer. Is there any "official" Support at Veeam for the Essential Product, because I wans't able to choose it in the product section.
Kind regards,
Patrick

[foggy]

Hi Patrick, yes, support for Essentials new installations is provided on a Production level.

[pschwarz]

Hi Foggy,
that means if I've bought Veeam Essentials and I'm using Veeam Backup and Replication as the license, then I'll get traditional support from Veeam. That's how I would understand Offerings included: Veeam Backup & Replication, Veeam ONE. Is that correct?
Kind regards,
Patrick

[Mildur]

Yes, Veeam Backup Essentials includes both products for the licenced workload (VUL or socket)

[HannesK]

Hello,
the physical limit for a 1 Gbit/s connection is 125 MByte/s. So I assume the 200MByte/s is more "read speed".

There is production and standard support (I assume you mean "standard" with "traditional") If you have a paid license, then you have support until the date mentioned in the license (license information or bottom right of the VBR console). Veeam ONE (incl. support) is included in the essentials package, yes.

which is using ESXi v6 and Veeam v9

now I'm confused. are we talking about two different environments? Two different storage systems? In any case: support should check the logs to see which part is causing the issues.

Best regards,
Hannes

[pschwarz]

Hi Hannes,
You are completly right, I mixed that up! 200 MByte/s was the processing rate.
Sorry if I confused you. The ESXi v6 and Veeam v9 enviroment is a different customer of mine. I only wanted to illustrate, that Veeam is working with Synology in a correct way.
With the new installation - ESXi 7.0.1 und Veeam B&R v11 I've got the trouble with the slow processing rate and slow write speed to the NAS and I have no clue which part is responsible for the delay - ESXi 7.0.1, Veeam B&R v11.
Best regards,
Patrick

[chris.childerhose]

"With the new installation - ESXi 7.0.1 und Veeam B&R v11 I've got the trouble with the slow processing rate and slow write speed to the NAS and I have no clue which part is responsible for the delay - ESXi 7.0.1, Veeam B&R v11."

What does the backup bottleneck show when you hover over it in the job? It will show percentages for Source, Proxy, Network, and Target. Which of these has the highest as that is where you start to troubleshoot.

[pschwarz]

I think I found the solution! The Synology DS1817 has got four LAN Ports. Port 1 & 2 are 10GBit Ports and Port 1 was connected to the HP Aruba 1930 GBit Switch. In normal usage (File access) there is no problem, if the LAN is connected to Port 1, only if a huge amount of data is copied to the NAS, which is a normal procedure when a backup of a VM is done by Veeam. The solution with the existing hardware is to use Port 3 or 4 of the Synolgy NAS which are "only" 1 GBit Ports. Now the backup job is done with the expected throughput of 120 MB/sec. It has no advantage to use the 10 GBit Port because the Switch and the HP Server can only handle 1 GBit over their NICs. Don't ask me why the customer installed the NAS in his network with the 10 GBit Port, maybe he thought it is faster....
Kind regards,
Patrick

[chris.childerhose]

Glad to hear you solved it.