SOLVED! Veeam Backup to Synology NAS takes forever

[Dieter]

I would like to share with you the solution to a performance problem I had when doing backups of large disks to Synology NAS:

My setup -----------------------------------------------------------------------------------------------------------------------
Source system
- Windows 10 Professional using Veeam Agent for Windows 3.0.2.1170
- C Drive 512 GB SSD with 186 GB used
- D Drive 2 TB SATA HDD with 1.2 TB used
- E Drive 2 TB SATA HDD with 1.2 TB used
Target System
- Synology DS216j with 2 10 TB HDDs as Raid 1
Source and target system connected in local LAN with 1 Gigabit Ethernet

My problem ---------------------------------------------------------------------------------------------------------------------
When I did a full system backup the transfer rate for each disk got "significantly" slower, e.g. disk c was 52 MB/s, disc d was 13 MB/s and disc e was 9 (!) MB/s. As a consequence a full backup would take days.

Some observations ---------------------------------------------------------------------------------------------------------
- Both systems were really connected with 1 Gigabit/s and when I copied single files I got good performance
- Incremental backups of the full system performed well
- Disk, network and CPU load on source and target systems where below 50% during the "slow" backup
- I am backing up a notebook (just one single 256 GB SSD) to the same Synology system without any performance problems - the problem is just with the system having the larger volumes

The solution -------------------------------------------------------------------------------------------------------------------
It took me some signifianct time to search the internet on Veeam's and Synology's web sites and some other support forums to find a hint which really improved the situation. In the end the fix was "trivial", just changing a single checkbox in the Synology setup: On the Synology system just check "Do not reserve disk space when creating files"
within Control Panel -> File Services -> SMB -> Advanced Settings . Actually you even have to scroll down in that dialogue in order to see that option ...

This single change improved my backup speed for the 3 disks to 81 MB/s (was 52 MB/s), 66 MB/s (was 13 MB/s) and 58 MB/s (was 9 MB/s).

Kudos to jwhite711 to provide me this hint. Seems like many people have similar problems (lots of entries - most without solutions - in the forums of Synology, Veeam and other backup vendors), but this was the single hint which made the difference https://community.spiceworks.com/topic/ ... -nas-3mb-s

[EdgarRicharte]

We do a majority of our onsite backups on synology NAS's. This is a helpful tip. Will try next time. Ty!

[SnakeSK]

Thanks a lot for this! Just finished my 3days backup

[Dieter]

So it shortened your backup from 3 days to ???

[thorsfall]

Dieter - What did the volume free space % look like on your Synology when you experienced this issue?

I've seen something similar when an ext4-formatted volume on a Synology is >80% full, then all of a sudden SMB/CIFS file operations (i.e. copy) start taking forever, to the point that they sometimes timeout. I'm just wondering if your tweak will help, or if my issue is just the way ext4 behaves will all that slack space between files.

[Dieter]

Hi Trevor, the volume is configured as RAID 1, filesystem ext4 and had about 40 % used capacity. So about 6 TB were free, and the largest file (the full backup file to be written) was 2.3TB large.

[mlabbe]

Thanks for this. I've upgraded a pc to 1909 and troubleshooted for a couple hours why active full backup was crawling and stalling at 2% (whole network was going down below 1 Mbits/sec, tested with iperf3, to the point Win10 reports no internet access).

After checking this option on the syno, full backup (130 GB) completed in 14 mins and PC is running just fine (back to 940 Mbits/sec in iperf3).

You saved my day

[mlabbe]

Can't edit my previous post for some reason, but I meant "whole network on the PC". The other devices on the network were just fine.

[Gerald]

Thank you! This problem has been bugging me for months. Finally performance for backing up large HDDs is back to what it should be. It was so slow that the initial backup would often fail (after running for 2-3 days just to backup the initial ~6TB to an GbE connected Synology). For those wondering what the option "Do not reserve disk space when creating files" actually does: It sets the option "strict allocate=no" in the [global] section of /etc/samba/smbinfo.conf

The Samba man page describes the option as doing the following:

strict allocate (S)
This is a boolean that controls the handling of disk space allocation in the server. When this is set to yes the server will change from UNIX behaviour of not committing real disk storage blocks when a file is extended to the Windows behaviour of actually forcing the disk system to allocate real storage blocks when a file is created or extended to be a given size. In UNIX terminology this means that Samba will stop creating sparse files.

This option is really designed for file systems that support fast allocation of large numbers of blocks such as extent-based file systems. On file systems that don't support extents (most notably ext3) this can make Samba slower. When you work with large files over >100MB on file systems without extents you may even run into problems with clients running into timeouts.

When you have an extent based filesystem it's likely that we can make use of unwritten extents which allows Samba to allocate even large amounts of space very fast and you will not see any timeout problems caused by strict allocate. With strict allocate in use you will also get much better out of quota messages in case you use quotas. Another advantage of activating this setting is that it will help to reduce file fragmentation.

To give you an idea on which filesystems this setting might currently be a good option for you: XFS, ext4, btrfs, ocfs2 on Linux and JFS2 on AIX support unwritten extents. On Filesystems that do not support it, preallocation is probably an expensive operation where you will see reduced performance and risk to let clients run into timeouts when creating large files. Examples are ext3, ZFS, HFS+ and most others, so be aware if you activate this setting on those filesystems.

Default: strict allocate = no

It's a bit weird that Synology seems to default to "strict allocate = yes", while the samba default is "strict allocate = no". Advantage seems to be reduced fragmentation and quota improvements?
I experienced this on both a Synology DS413j and a DS418j. Both with ext4 Filesystem, this bug/problem could at least in theory not be present when using btrfs? Or it could be worse when using btrfs... Hard to tell without actually testing it.

[robjohnston]

One other thing if you are backing up to a NAS, CIFS/SMB while works, and is the easiest to setup, is the slowest option for Veeam Repo's, so if you can present your Synology repo as ISCSI target, you should get better performance over SMB targets.