Full backup speed decreases while it runs

[mkr]

Hello Veeam Community,
the "processing rate" of my Full Backup decreases while it runs. After more than 20 hours, the processing rate has dropped from 100 MB/s to less than 30 MB/s.
I have already opened a support ticket with the Case ID #04666080.
Overall, 12TB of data must be backed up from 4 partitions of a Windows server.

Backup Job-Configuration:
Volume level Backup
Shared Folder
Advanced > Storage: Compression level Optimal, Storage optimization: LAN target
Agent-Configuration:
General:
"Throttle backup activity when system is busy" is not active.

The CPU load of the server is between 20% - 60%. The RAM utilisation is always between 60% - 80%. The temperatures remain stable. No abnormalities can be found in the Windows logs either.
The same backup job on an internal HDD was written continuously with around 150 MB/s. The full backup took only 17 hours.
The server is directly connected to the NAS to exclude other network components as a source of error. Both have 2 1Gbps NICs.
SMB 3 is active on the NAS.
To check whether the server and the NAS can generally handle the continuous writing of large files, I manually copied the backup file (10TB) from the internal hard disk to an SMB share on the NAS. Here the transfer speed does not slow down. Even after more than 10 hours, the file is still being written at over 100 MB/s.

[Dima P.]

Hello and welcome to R&D forums mkr,

Can you please clarify:

1. Amount of RAM you have on the machine?
2. How these 4 partitions are allocated on the machine (i.e. single disk or multiple disks)? Are those disks regular HDDs?
3. Any chance you see the performance decrease for some specific volume?
4. That was initial full backup or you see the same behavior during incremental job rub?

Thank you in advance!

[mkr]

1. 16 GB of RAM

2.
Partition 1: (C:) RAID 1 of 2 250GB NVME SSDs
Partition 2: RAID 1 of 2 10TB HDDs
Partition 3: RAID 10 of 4 8TB HDDs
Partition 4: Singel 120GB SSD

3. The problem occurs when Partition 2 and 3 is processed
4. Unfortunately, I have never been able to run the backup to the NAS completely, as it would take far too long.

Thank you for the quick reply!

[Rainbow68]

Hello community,

I'm reffering to case ID #04666080.
I found this topic facing the same problems. Are there any news?

The system: i9-9900KF/32GB I try to do an initial full backup to a Zyxel NAS540.
C and D are on SSDs, F, G and H are on HDDs.
PC and NAS are both showing 1000/1000Mbps on the network adapter.
The backup starts at 109MB/s with drive C and then dropping drive by drive to only 5MB/s

This is the current status:

Thanks.

[mkr]

Hi Rainbow68,

yes, I solved the problem by disabling the parameter "strict allocate" in the smb.conf file. Unfortunately, QNAP doesn't provide a proper solution to this so you have to edit the file over the shell. After a reboot, all changes in this file are reset to default, so you must write a script to persist the changes.
Sorry, but I have not had a chance to write a detailed answer yet. But I will take care of it today.

[mkr]

How I solved the Issue:

My solution is for a QNAP NAS. From what I have read, Synology products offer the possibility to disable strict allocation via the web interface.

To disable strict allocation on a QNAP NAS you must edit the smb.conf which is located at "/etc/config/smb.conf". Simply change the parameter "strict allocate =" from yes to no.
To apply the changes, you must restart the smb.sh service. After that, strict allocation should be disabled.

Unfortunately, after a reboot, all settings in the smb.conf will be reset to default. To persist the changes, you must write an autorun.sh script. For that I followed the guide from OneCD, a user in the QNAP forum.
https://forum.qnap.com/viewtopic.php?t=130345
Please follow his instructions.

My final autorun.sh script looks like this:
"sed -i 's/strict allocate = yes/strict allocate = no/g' /mnt/HDA_ROOT/.config/smb.conf && /etc/init.d/smb.sh restart"
This script disables strict allocation for every public folder, that is created.
The patch differs from the "clear" patch where you can change the settings manually! Very likely, something needs to be adjusted in the file path to make it work for you.

I hope I can help some of you out there!

[pirx]

Thanks a lot! After chasing performance problems at our remote locations where we use different QNAP model, I've finally found the root cause! Now - even with SMB - the synthetic backup at one location is back from 100 to 5 hours!

Support only pointed at poor SMB protocol and more or less that it's always a SMB issue. It would have been really helpful if support had checked it's own support database.

I run different diskspd tests while backup was crawling (even writing >1TB) and it always showed that the device can be really fast.