Synology NAS as repo

[that backup guy]

Hi all,

We were about to pull the trigger on a new NAS to expand our backup environment, and then I read about the data loss issues from Gostev's (excellent) newsletter. My questions are:

1. How widespread does this appear to be?
2. When does this appear? Is it only during a failed restore, or does/would a health check catch this?
3. Does any particular configuration (teamed NICs, iSCSI vs NFS, etc.) cause this to happen? The letter only mentioned NFS

Thanks!

[Mildur]

If you must use a NAS (don‘t do it ), then use iscsi.
It‘s better, because veeam has access to the file system. And you can leverage FastClone if you mount it to a Windows 2016 or above and format the iscsi LUN with refs (On Linux also possible with xfs).
CIFS is really bad. NFS now has some issues too according to antons newsletter. And both protocols cannot be used with FastClone on a NAS.

But depending on the size of your company, have a look for a dedicated hardware machine. It doesn‘t need to be a monster server. You could use a linux hardened repo. It‘s a good protection for your backup data.


Here is another statement from Anton about NAS:
post430954.html#p430954

And contrary to that, the stream of customers with corrupted/unrecoverable backups sitting on low-end NAS is never ending in our Customer Support.

[cgsm]

@that backup guy

We tried using a Synology box and had corruption issues with incremental backups, and thus Full VM Restores of those backups. We have, what I consider, a mid-range model (RP2818RP+, 8x 4TB WD Gold HDDs in RAID 10, 10Gb Intel ethernet NIC, dual PSUs). We were connecting to the repo with iSCSI and formatted with ReFS. We have since started using an old Dell server with a battery backed hardware RAID controller and have had zero issues.

The issue we had was incrementals would get randomly corrupted. We run the backup health check daily so we catch this issue quickly. When we performed an Active Full the issue went away, as expected. However, after a number of days of incrementals, one or two VMs would be corrupted again. A weekly Synthetic Full would not be corrupt, which is quite odd to me considering it is "made from" the incrementals marked as corrupt but maybe Synthetic Fulls "heal" the data? I have no clue. We confirmed the Synthetic Fulls were valid by restoring the entire VM which was not possible using a corrupt incremental.

Of note, when we formatted the Synology repo with NTFS we did not experience corruption.

Since we are now using a Dell server as the repo, we are now running a Backup Copy job to the Synology box (iSCSI, ReFS) and have yet to experience corruption (we run the backup health check daily here too). Not sure why, but maybe copy jobs have less I/O to possibly get corrupted?

I too, wish Synology units were more supported/reliable with Veaam. I have never had a corruption issue with Synology prior so it is hard for me to put all the blame on them. Maybe I just never noticed the corruption prior? Maybe Veeam checks for corruption better?

[Gostev]

You're spot on @cgsm... you just never noticed, because you did not have Veeam. See my posts from a few weeks ago which explain why this happens, and also how we catch storage-based data corruptions. It's unlikely that any other applications which data you stored on your NAS had similar capabilities. Besides, I'm sure these applications operated with much less data comparing to the size of image-level backups Veeam creates to start with.

[EDIT] Sorry, didn't realize until now the thread I'm referring to was between me and you

[Gostev]

My questions are:
1. How widespread does this appear to be?
2. When does this appear? Is it only during a failed restore, or does/would a health check catch this?
3. Does any particular configuration (teamed NICs, iSCSI vs NFS, etc.) cause this to happen? The letter only mentioned NFS

1. It has always been the absolute top reason for failed recoveries we've been seeing in our customer support.
2. Health check aka "storage-level corruption guard" will ALWAYS catch this issue, but keep in mind it only checks the latest (most recent) restore point.
3. By now we've seen issues with all protocols (SMB/NFS/iSCSI) and with many different NAS vendors serving consumer/SOHO market.

I personally always suspected that these reliability issues are due to NAS vendors making questionable optimizations in their software in order to score higher in performance tests, which is super important for their market... but to be crystal clear, it's my own suspicion and nothing more. It could be due to honest bugs on their end, or due to the limitations of software RAID technology they are using, or really anything along these lines. We can never know the root cause because NAS is a black box for us: we don't have its source code nor means to troubleshoot its hardware. All we can do is issue commands against it and observe results, which are really scary at times.

[cgsm]

@Gostev

Would you suggest any Synology model for use as a backup repo? While I know the documentation recommends hardware RAID controller, DAS storage, etc., I think it would be worth also stating the opposite much more strongly: "we strongly advise against using a NAS". From my issue, and now seeing how many others encounter the same issue, I don't think Veeam's position on this is strong enough; just trying to save you some time from having to reiterate the same thing over and over!

[Gostev]

As far as suggested NAS vendors and models, I've already answered your similar question a few weeks ago. Basically, if you really want NAS as your backup target, go with some enterprise vendor to avoid surprises. This almost never makes sense though from either performance or costs perspective vs. using a general-purpose server.

We're not in position to make such a generic statement as you suggest: "we strongly advise against using a NAS". We have plenty of customers using enterprise-grade NAS like NetApp without any issues whatsoever, because these systems were designed with reliability in mind and further polished by decades of serving enterprise IT environments. But it's naive to expect the same level of reliability from 10x cheaper storage systems designed for a different market segment.

We could probably say "be careful with low-end storage, because you get what you pay for" but I'm sure everybody realizes this even without our help

[that backup guy]

Hi all,

I appreciate the info! This is exactly what I was looking for.

@Gostev: I wouldn't be so sure everyone knows about these issues. IT/Server admin is a very wide field, and people tend to gravitate towards and away from different subjects. At my old shop, Synology was king and nobody ever thought twice about it. I've been around for over a decade and if I didn't get your newsletter, I never would have even heard about it until this week. These are the kinds of things that keep me up at night

[Wocka]

We run a new (April 2021) Synology RS3621xs+ with 2 x 10GB ethernet. I was also getting Backup copy jobs corrupting from a remote location.

Here is my set-up:
Local Office:
- 2 x 10GB Ethernet Bonded as 1 interface in the Synology
- MTU 9000
- Repository presented to Veeam B&R as a NFS share (NFS 4.1 Enabled)

Remote Office (1000km away)
- Backups saved locally to USB HDD
- Backup Copy jobs copied daily to Synology
- Repository presented to Veeam B&R as a SMB share
- Connected by 50MB WAN link (no WAN accelerator in-place)

I noticed after upgrading to B&R V11.0.0.837 that the remote location's Backup Copy jobs would corrupt. Recreating new Backup chains solved this issue for about a week. The full .vbk file would take 34 hours to copy to the Synology so this was a large issue for us. Upon Googling the error I received in the Backup Copy job console, I found a Veeam forum post where someone had the same issue but had dual RAID cards in their repository.

I realised that I had dual NIC's in the Synology. So I plugged in an additional 1 x 1GB connection in and gave it a separate IP to my Bonded 10GB NIC's. I configured Veeam B&R in the remote location to use the new single port IP and I haven't have a Backup Copy become corrupt since. This has now run perfectly for 2 months.

@cgsm
I would not recommend a Synology NAS device for a Veeam Repo. We have one because our director has a soft spot for Synology and bought one after our Dell DR4100 died.
This non-recommendation is based on the support I had (or lack of) from Synology here in Australia. We had a failed hard drive after 3 months (and you can only buy Synology branded drives with our product) and the process to jump through for support, returning drives, requesting new ones was painful. I'm lucky that the Synology was working fine with the RAID config that we had. If we had a business critical non-working hardware failure we would have been dead in the water for 8 days. When you buy a NAS at the level of the RS3621, I would have though the support would have been better and aimed at the mid-level business's.

[marcinbojko]

Worst cases I've had with Synology and Qnap was almost always related to their iSCSI or ODX support.
Disabled ODX, stopped using their iSCSI and as NFS it works fine.

[ITP-Stan]

We use a lot of Synology NAS devices as back-up repositories for our SMB customers, we always use iSCSI (with CHAP password) and don't use bonded/teamed NIC's.
Granted, we don't do daily health checks, mostly monthly or weekly at most.
I don't have any experience with corrupted back-ups and I don't believe any of my colleagues had either.

When using ReFS over iSCSI (on a NAS), it is not a supported (by Microsoft) configuration.
So you have to realize that when power loss occurs, you could possibly loose the ReFS volume, without any way of healing/reparing it.

So I think that when you have the parts or the budget, a general purpose server with disks is absolutely the most recommended solution.
When you use a Synology NAS, and especially when also using ReFS you have to consider and own up to the risks.

However, I don't have any faith in other SMB NAS vendors other than Synology, out of personal experience.
I would never use it as a SMB repo and not as a Linux repo, always iSCSI. I can't say about NFS, no experience with it really.
And based on some posts here, I would avoid bonded/teamed NICS on it.

[Coldfirex]

Is the NFS issue something that has been reported to Synology?

[mweissen13]

Many of our customers are on a budget and are using various Synology NAS Models as their backup targets. We have seen a few corruptions and a have boiled it down to one of the following two things:
a) forgetting to disable the write cache of all disks. Oddly enough, Synology seems to enable the write cache per default. Supposedly in order to shine in performance tests. But do you really want that 10% performance while risking to lose all your data? Also when swapping a defective disk you must always keep in mind to manually disable the write cache again.
b) not using a small UPS which is connected directly to the NAS via USB. You must also enable the NAS support in the NAS's interface.

When both a) and b) were set correctly, we have so far never seen corruption. This is surely not a 100% thing, but seems to greatly reduce the odds of corruption happening. As always, YMMV and this is or course also not a setup which is recommended by Microsoft for ReFS volumes.

[Mildur]

And based on some posts here, I would avoid bonded/teamed NICS on it.

If I must use a NAS, then I would go with iSCSI too. For iSCSI, I use MPIO.
I would never use LACP on the iSCSI Interface.

I have stopped to design a backup infrastructure with NAS devices. For our small customers, a dedicated backup server with 5-15 TB Storage and Veeam Cloud Connect as a backup copy target is the perfect solution.

[Robvil]

I am using both Windows Servers using Refs, but also Synology. But with the Synology i am using ssh connection and never had a corruption.

/Robert

[aceit]

Synologys are nice machines, well thought package overall, we have one too with some terabytes just as commodity storage for the convenience of our users as home dir to store photo from phones for their companion app etc. and files without too much importance, temporary/personal stuff.

For an super heavy high I/O software like veeam that backups usually critical stuff, (IMHO of course) please consider something else, each thing has its place, there are lot of moving parts and protocol semantics to respect in a I/O path, and Veeam, due to its heavy use of I/O (and the double check of added layer of checksums/ecc) can stumble upon them.

Budget wise there are lot of exceptional storage arrays in the now "flourishing" used market programs by the big brand enterprise names themselves, those are sold directly from the vendors and certified and supported (i.e. expired leases / contracts etc.), you can find very good hardware sometimes LESS or equal in price than a synology (if used only for the storage and not the apps), to build high capacity robust backup servers; it is also a very environmental friendly solution if you just need capacity and not the last software feature of the new stuff (that we leave to the primary online application in case).

My favourite configuration is to use a as backup a standard all-in-one veeam windows server with an external enterprise block array attached i.e. fibrechan or iscsi with hardware host adapters (not internal storage). The external array usually will last more then the server, if you replace the server you just attach the array to the new one, or you can add arrays in time, much faster for migrations and to expand capacity etc....
The Veeam software will just see such storage as a normal block local disk and work on it robustly and happily

[TitaniumCoder477]

I am using both Windows Servers using Refs, but also Synology. But with the Synology i am using ssh connection and never had a corruption.

/Robert

We also are integrating Synology NAS to Veeam over SSH, mainly so that the transform processes take place on disk rather than over the network. Unfortunately, there is an incompatibility between DSM6.x and v11 that will be even worse in v12. I am not sure about DSM7 because I haven't upgraded any of our clients to it yet. I did create a post over on Synology's forum about the issue with DSM6.x, and I have also talked with our sales manager about the situation. Unfortunately, it seems like neither Synology nor Veeam are interested in addressing the problem, or even discussing it, to be honest.

I actually agree with both sides of the discussion posted above. For large to enterprise clients, or clients with a lot of data, it does totally make sense to go with server/SAN hardware. But we deal with a lot of medium size clients too, and they are usually already struggling with the cost. Synology provides three things better than any other vendor, including QNAP, in my opinion: (1) the hardware is very high quality, (2) the operating system interface is very intuitive (though I haven't tried out DSM7.0, so can't speak for changes therein), and (3) the options are vast. That is, Synology offers a VERY wide range of shapes and forms, which means you can build any design for any kind of environment. Got a rack? They have a slew of rack options that support any number of disks you need. Got no rack and a tiny shelf in a closet? No problem there either. They have a dozen toaster models to choose from with almost as many drive bays. They also offer RAM upgrade kits so you can always make sure the NAS has enough RAM according to Veeam system requirements for the repository and mount roles etc.

[Mildur]

it seems like neither Synology nor Veeam are interested in addressing the problem, or even discuss it, to be honest.

Totally understandable to me, because this is a unsupported scenario. As a vendor, the resources should be put in supported scenarios.
No NAS vendor likes if they customers are playing around in the operating system and installing software by themselves.

[Robvil]

We also are integrating Synology NAS to Veeam over SSH, mainly so that the transform processes take place on disk rather than over the network. Unfortunately, there is an incompatibility between DSM6.x and v11 that will be even worse in v12. I am not sure about DSM7 because I haven't upgraded any of our clients to it yet. I did create a post over on Synology's forum about the issue with DSM6.x, and I have also talked with our sales manager about the situation. Unfortunately, it seems like neither Synology nor Veeam are interested in addressing the problem, or even discussing it, to be honest.

I actually agree with both sides of the discussion posted above. For large to enterprise clients, or clients with a lot of data, it does totally make sense to go with server/SAN hardware. But we deal with a lot of medium size clients too, and they are usually already struggling with the cost. Synology provides three things better than any other vendor, including QNAP, in my opinion: (1) the hardware is very high quality, (2) the operating system interface is very intuitive (though I haven't tried out DSM7.0, so can't speak for changes therein), and (3) the options are vast. That is, Synology offers a VERY wide range of shapes and forms, which means you can build any design for any kind of environment. Got a rack? They have a slew of rack options that support any number of disks you need. Got no rack and a tiny shelf in a closet? No problem there either. They have a dozen toaster models to choose from with almost as many drive bays. They also offer RAM upgrade kits so you can always make sure the NAS has enough RAM according to Veeam system requirements for the repository and mount roles etc.

I have DSM6 with version 11 running. It is working (but will stop working with version 12) unless Synology does something.

/Robert

[TitaniumCoder477]

Yes, sorry I wasn't too clear... by incompatibility, I meant that it gives that warning/error and some people have mentioned performance degradations already. And as you rightly pointed out, the changes in v12 will result in completely incompatibility unless Synology and/or Veeam decide this is important enough to their customers to fix!

[kaffeine]

@Robvil / @TitaniumCoder477: what incompatibilities are you exactly referring to and what are the protocols affected?

[cgsm]

When using ReFS over iSCSI (on a NAS), it is not a supported (by Microsoft) configuration.

@ITP-Stan, can you explain why ReFS isn't supported over iSCSI? This is the first I have heard of this. Do you have any backup documentation?

[cgsm]

So I have some more diagnostic information on Veeam + iSCSI + Synolology. As I have stated previously, I encountered corruption while using ReFS and found this via backup health checks (and subsequently trying to fully restore a VM).

Through a support case, I was directed to run a tool called "COFI.exe" on the repository and see what it found and it too found corruption, and did so after a few hours. I reformatted the repo as NTFS, reran the COFI tool for nearly 48 hours, and no corruption was found.

Based on this, I think the best route is to use NTFS on Synology hosted iSCSI volumes. I am testing this and will try to remember to report back.

On another note, while I have been diagnosing this, I have been using the ReFS formatted iSCSI volume hosted on my Synology box for backup copy jobs and no corruption has occurred. Maybe backup copy jobs have less chances of corruptions since they are just copying VBK and VIB files?

[Robvil]

@Robvil / @TitaniumCoder477: what incompatibilities are you exactly referring to and what are the protocols affected?

veeam-backup-replication-f2/veeam-v11-a ... 72376.html

[chris.childerhose]

We also are integrating Synology NAS to Veeam over SSH, mainly so that the transform processes take place on disk rather than over the network. Unfortunately, there is an incompatibility between DSM6.x and v11 that will be even worse in v12. I am not sure about DSM7 because I haven't upgraded any of our clients to it yet. I did create a post over on Synology's forum about the issue with DSM6.x, and I have also talked with our sales manager about the situation. Unfortunately, it seems like neither Synology nor Veeam are interested in addressing the problem, or even discussing it, to be honest.

I actually agree with both sides of the discussion posted above. For large to enterprise clients, or clients with a lot of data, it does totally make sense to go with server/SAN hardware. But we deal with a lot of medium size clients too, and they are usually already struggling with the cost. Synology provides three things better than any other vendor, including QNAP, in my opinion: (1) the hardware is very high quality, (2) the operating system interface is very intuitive (though I haven't tried out DSM7.0, so can't speak for changes therein), and (3) the options are vast. That is, Synology offers a VERY wide range of shapes and forms, which means you can build any design for any kind of environment. Got a rack? They have a slew of rack options that support any number of disks you need. Got no rack and a tiny shelf in a closet? No problem there either. They have a dozen toaster models to choose from with almost as many drive bays. They also offer RAM upgrade kits so you can always make sure the NAS has enough RAM according to Veeam system requirements for the repository and mount roles etc.

I am running DMS7 with ReFS volumes for Veeam backups and have no issues. The DSM7 is a much improved version for Synology and has many improvements for iSCSI, etc. I run a DS920+ at home in my lab and have had no problems as yet.

[guitarfish]

I've been using QNAP devices since 2012 for 3 small remote sites, CIFS only. I would start up the VM from the backup weekly and it always came up fine. Occasionally I did some file restores, and once had to restore a server. All indications were that these were working fine. But with the repeated mention of NAS device corruption I was starting to feel like I needed to make a change, or I'd be one of those people with corrupt backups.

I ended up going with some Dell Precisions with 8TB drives running Win10. These came in under $2K per system and were perfect for my small sites. At HQ I use a larger Dell server with a lot more storage. So, no more NAS's for me for Veeam backups. I preferred the NAS over a Windows system for many reasons, but reliability and integrity is obviously first.

[Gostev]

Maybe backup copy jobs have less chances of corruptions since they are just copying VBK and VIB files?

That is not a correct statement. Backup Copy jobs are not "just copying files" but rather creating new files and populating them with the data just like primary backup jobs do. Literally the only difference is the source of this data (production storage vs. backup repository).

[cgsm]

@Gostev Thanks for the clarification. It is truly confusing to me why a backup copy job then results in no corruption being detected while when I use the repo for the primary backup target corruption is found. Clearly there is something odd happening.

[cgsm]

It seems to be that there are two distinct cases here: either (a) we have experienced corruption when using a Synology box, or (b) the Synology box runs fine without any corruption detected. I wonder if there are some DSM settings some of us have different than others resulting in this. I am providing my settings, with as much detail as possible, to help diagnose this.

- RS2818RP+. DMS 7.0-41890. Dual PSUs.
- 8x 4TB WD Gold drives in a RAID 10. Write cache is turned off on each disk. No hot spare. No SSD cache.
- PCIe Dual port 10Gb-BaseT (ethernet). Ports are bonded in DSM using Adaptive Load Balancing, both connect to same switch.
- iSCSI with default settings, no CHAP. Volume created in Storage Manager, then LUN created in Volume, thick provisioned. Buffered I/O. FUA and Sync Cache SCSI commands unchecked. I/O queue depth set to the default 64.
- Mounted in Server 2016. Formatted with ReFS/64k results in corruption. Formatted with NTFS does not have corruption.

[cgsm]

I have been doing some testing with some different configurations on Synology's DSM. It seems that the write cache is the culprit of the corruption.

My tests:
1) Enable write cache on one or more drives in the pool, create a volume, create a LUN, format with ReFS 64k, run COFI. Result = corruption detected within houts.
2) Disable write cache on ALL disks in the pool create a volume, create a LUN, format with ReFS 64k, run COFI. Result = no corruption detected for 24+ hours (still running).

Now, I don't have any Veeam jobs running on these volume/LUNs since I am just testing currently, but since COFI isn't finding any corruption I would wager that a backup health check or the manually run backup validator executable won't find corruption either.