B&R 9.5 to Synology - current best solution

[Trin]

We're looking at getting a Synology RS2416RP+ to store our backups and I'm trying to find out the best way of connecting our Veeam installation to the NAS.

I've read a bunch of articles, some of which say connect directly from Veeam with the NAS as a Linux server and other that suggest connecting the NAS to the Windows server using iSCSI or NFS and then connecting Veeam to that.

Is there a current best practice for this situation at all?

Many thanks

[jmmarton]

If you connect to the Synology directly as NFS, we'll have to install our data mover on the appliance. But if you set it up as iSCSI you can connect it to either a Windows or Linux server, using it as a Windows repo or an NFS repo with the data mover going on a general purpose OS rather than the customized Linux-based firmware on the appliance. If it were up to me, I'd avoid making that sort of change to the appliance if possible.

Joe

[nmdange]

If you haven't actually bought anything yet, I would highly recommend using a physical Windows server with directly attached storage (either disks internal to the server chassis or in a SAS JBOD attached to a hardware RAID controller.)

[Trin]

...If it were up to me, I'd avoid making that sort of change to the appliance if possible.

Joe

I presume you're wanting to avoid your first suggestion there, the direct attach method?

We've picked the Synology for more than just a backup repository as it's the best method for various things we want to do.

[Trin]

In addition, is there any benefit of using iSCSI file level or iSCSI block level?
The plan is to have, perhaps, several LUNs on the storage, with certain servers being backed up to different LUNs, so I'm guessing Veeam will be better off using the 'iSCSI block level multiple LUNs on RAID' option...?

[final]

Hi,

I can give you some (maybe useful, maybe not) insights as we're using Synology NAS and B&R 9.5

We use two NAS (2x RS3617RPxs) for Veeam Backups. They're stuffed with 6TB disks (WD Red Pro) and in a raid6 config, giving ~55TB of useable space. They're connected via iSCSI with a DA cable to the server using jumbo frames. We found that connecting them via a switch results in performance loss, regardless of jumbo-frames. The NAS and servers use 10gbps NICs. The unit price per NAS (including disks and NIC) is about $6000.

iSCSI does not give you the best performance (NFS / CIFS has better performance), BUT iSCSI allows you to use Server 2016 dedup functionality, which dramatically increases the retention periods, especially if you do regular active full backups. And the penalty is not that large, especially for large block sizes which is what you usually have with backups.

We get about 350MB/s write and 650MB/s read performance this way (on non-deduped data). Deduped data reads at about 30MB/s average, but the rate is very inconsistent (between 10MB/s and 150MB/s).

On a closer look:
- About 13TB of VMs (~40VMs total, largest is 2.3TB)
- physical server at the datacenter, connected to one NAS as well as the SAN storage
- physical server at our main office, connected to the other NAS as well as a tape library (MSL 4048 with one LTO-7 Ultrium drive).

Backup jobs run as follows:
- Backup all VMs from the NAS to the SAN (during lunch and at evenings of business days, with weekly active full, starting on friday, 7pm).
- As soon as a backup job finishes, copy the backups to the NAS at the main office. (10gbps line)
- As soon as the weekly active full backup copy job finishes, write the copy of the backups to tape
- Dedup any data that is older than 10 days

Our learnings after playing a lot with this setup:
- We can only achieve this our backup schedule by using post-job scripts
- Use single file per VM for your repositories, as this makes the life of the dedup job a lot easier.
- iSCSI is not the bottleneck. It's usually the disks themselves, because the backup takes place at large block sizes.
- We use GFS Tape Jobs. As GFS Tape Jobs do not support parallel processing, there is no point in having two Tape drives in your library (and those LTO-7 drives are quite expensive, we paid ~$5000 for ours).
- Do not dedup your newest active full, as it would slow down your desaster recovery significantly (30MB/s read instead of 600MB/s read). We're deduping only data that is older than 10 days.
- Dedup needs sh*tloads of RAM. In our case, we have about: 80TB of deduped user data (8TB dedup chunk stores), and 40GB of avaibale RAM (available for the dedup job, the server has 48GB of RAM). This is the limit - increasing the deduped data results in aborted dedup jobs. We will upgrade our servers to 192GB of RAM in the near future in the hope that this will increase the potential deduped data significantly.

The result of this setup is:
- desaster recovery speeds of about 600MB/s (newest restore point), which gives a full recovery time of <8 hours (including some admin overhead) in case of a SAN failure.
- 100 restore points on disk at two sites (which will hopefully go to 200 - 300 restore points after the RAM upgrade).
- Weekly, Monthly, Quarterly and Yearly backups on Tape (2 Tapes per full backup set).

[Trin]

That's really useful, thanks. A couple of questions, if I may;
How did you configure your Synology boxes - single LUN or multiple?
Are you using Windows to do the dedupe rather than Veeam?

We're in the middle of a large storage and backup overhaul, hence this thread, so I'm just after best practice or successful implementation really. Our original use of VB&R was a rather hasty requirement to replace the previous solution which crashed and burned one afternoon so I'm fairly new to it.

[final]

They're configured as a single LUN, but I don't think that makes much of a difference really.
We're using Windows dedup. Make sure to use a (fully patched) Server 2016, as it improves deduplication significantly. Our setup would not work with server 2012 R2.

The physical server at the datacenter is also the main veeam B&R server (as well as our vSphere server). The physical server at our main office is a DC and DHCP for the local subnet.

[Trin]

The only reason I ask about the LUNs is because we currently backup our production VMs to one NFS and backup the data from our media server, specifically, to another so I was looking to perhaps replicate this. Not sure if it would make management any easier as we wouldn't be backing up the media to tape...

Do you see any benefits to having the VB&R on your vCenter server?

[final]

server 2016 dedup only works on "physical" volumes to the server. That's why it has to be connected via iSCSI if you want to use that. If you don't want to use server 2016 dedup, then you can use NFS or CIFS as well. I'm not sure if B&R works with CIFS / NFS shares on the NAS as (I think?) it wants to install its transport agent on the box, which will fail on the nas.

And no, there are no benefits of having the vCenter and B&R on the same box. We just felt more comfortable on having the vCenter box on a physical machine rather than a virtual one (out of the feeling that you don't want the management box to die as well if you have a problem with your VMs / Virtual Infrastructure), and our backup-server was the only physical box available We call it the "stuff" server, because it runs "a lot of stuff"

[jmmarton]

I presume you're wanting to avoid your first suggestion there, the direct attach method?

Yes, direct to Synology via NFS is what I'd try to avoid if iSCSI is an option.

Joe

[Trin]

Thanks Joe.

Are there any thoughts on the one vs multiple LUNs point? I was hoping to be able to essentially point different backups at different drives for ease of admin. Does that sound reasonable?

[final]

Are there any thoughts on the one vs multiple LUNs point? I was hoping to be able to essentially point different backups at different drives for ease of admin. Does that sound reasonable?

I actually recommend against that because you'd need to know your backup sizes quite well so that you could size your luns right. Also, you lose flexibility and gain no security, as both luns reside on the same disks. But I guess in specific use cases, this is viable.

[Trin]

That sounds reasonable, thanks.

We're going to end up with about 100TB to play with, I believe, so I'll perhaps look at the following:

Synology
Single disk group (all same size) on RAID 6 with one iSCSI target with one LUN presented to the windows server on which Veeam is installed.

Veeam
One job for production VMs with one file per VM and one job for the media server, all pointing at the same LUN.

Windows
Not sure about the deduplication yet... (if I don't use it I may stick to single backup file for all VMs).

Obviously you're not aware of our general setup here, but does that sound like a workable configuration?

[BIGNOOKIE]

@Final, we have a Synology 3614RPxs that I am going to re purpose as a Veeam Repository.
I see you are using RAID 6. What about RAID 10 whilst acknowledging the lost capacity of doing this?

Also have you explored SSD Cache and would there be any benefit in implementing this for Veeam?

[WingDog]

I have several Synology boxes as VBR backup repos and switched from SMB to iSCSI because of problems with "merge" process during GFS rotation - it took several days.
and with iSCSI you can MPIO.

[nickforum]

Hi,

How have you configured Veeam B&R to Dedupe data that is older than 10 days?



Thanks
Nick Rigas


[quote="final"]Hi,


Backup jobs run as follows:
- Dedup any data that is older than 10 days

Our learnings after playing a lot with this setup:
- We can only achieve this our backup schedule by using post-job scripts
- Use single file per VM for your repositories, as this makes the life of the dedup job a lot easier.
- Do not dedup your newest active full, as it would slow down your desaster recovery significantly (30MB/s read instead of 600MB/s read). We're deduping only data that is older than 10 days.

[sg_sc]

You might consider using the NAS as ISCSI block storage connected to a Windows 2016 server and format the volume using ReFS 3.1 (NO deduplication).
if you performs synthetic fulls you will see blazing fast backups, and also the merge operations (which are very problematic on some setups) will be super fast!
Another benefit of the synthetic fullls is that they do not take up the full space, on ReFS 3.1 they will only consume the changed blocks in space.
This gives you deduplication benefits without the need for deduplication.
Very handy also for a long term backup copy job with GFS scheme!

[final]

Sorry for getting back to you so late.
@Trin: Your config sounds ok. You will have to play with it for a bit to gain experience with your solution.

@BIGNOOKIE: Raid6 has two advantages over Raid10: capacity and resilience. Raid10 obviously gives you more performance. I had to stop and think for a moment why Raid6 is more resilient, but the reason is this: When a disk failed and is replaced, Raid10 has to copy the contents of one disk to the new disk. All other disks of the Raid10 are idle during recovery. If that source disk has a single read error, your data is compromised. Raid6 on the other hand can handle a read i/o error during rebuilding (in fact, it can handle another entire disk failure during rebuilding).

Hence, from a resilience perspective, Raid10 is only marginally better than Raid5 (roughly n-1 times more resilient than Raid5, where n is the number of disks in your raid). Raid6 is order of magnitudes more resilient than Raid5. Let's do some math, cause I feel like it
Raid10
According to the spec sheet of a WD Red Pro Disk, it has a unrecoverable bit read error rate of <10 in 10^15 bits. I'll use 10^15 for arguments sake, making the disk slightly more reliable than the spec sheet says it is. A 6TB disk has about 5*10^13 bits. Assuming failures happen uniformly at random, then each read bit has a failure probability of 1/10^15. The chance of success hence is (1 - 1/10^15). Thus, the chance of successfully reading an entire 6TB disk is (1 - 1/10^15)^(5*10^13) = ~95.1%.

It follows that your Raid10 rebuild fails in more than one of twenty cases using 6TB WD Red Pro disks (more because we've taken a slightly lower error rate than what the spec sheet says).

Raid6
This is a little harder to calculate, because if a bit cannot be read from one disk, the other disks can correct this if they successfully read the raid chunk (usually 512KB or something like this), but bear with me Let's start with the simple thing: The chance of a Raid6 recovery succeeding without hitting a single bit read error. This is essentially the same as before, only we need to read a lot more data (n-1 disks instead of just one as in the Raid10 case). I use a Raid6 over 12 disks in this example (so we need to read the data off 10 disks), hence:
(1 - 1/10^15)^(10*5*10^13) = 60.67%

Ouch. So the probability of successfully recover such a large array without a single bit read error is quite low. This is the reason why people say Raid5 is dead (this figure above is the probability of recovering a Raid5 over 11 disks without a bit read error).
But we're Raid6. So in case we do hit a bit read error on one disk, we can recover from that if the remaining 10 disks can read their chunk of ~500KB in which the read error occurred. So:
(1 - 1/10^15)^(10 * 500 * 1024) = 99.99999948% (that's six 9s after the dot).
So the chance of a successful recovery after a bit read error is tremendously high. Combining the two figures, the chance of a Raid6 recovery not failing due to a bit read error is:
(1 - 1/10^15)^(10*5*10^13) + (1-(1 - 1/10^15)^(10*5*10^13)) * (1 - 1/10^15)^(10 * 500 * 1024) = 99.9999998% (again, six 9s after the dot).

Note: We've ignored the chance of the entire disk failing, because it's much harder to get statistics for that. But even then, Raid6 will do much better than Raid10.

[final]

How have you configured Veeam B&R to Dedupe data that is older than 10 days?

I tell Windows Server 2016 dedup to do that, not Veeam B&R.

[AFalb]

Hello,
I just want to add, that you can't add a Synology NAS with DSM 6.1 as Linux Server to use NFS!
Just had the experience myself and talked about it with Veeam and Synology Support. A lot of packages are missing on the NAS and the only way to get them there is to crosscompile them.

[mongie]

Where possible, I'd recommend using REFS instead of Windows server dedupe.

I've found that enabling dedupe just makes your storage too slow. REFS doesn't seem to do this.

[thomasellaby]

We were using CIFS to connect to a synology and had no end of problems...
Followed a doc off a blog and configured it as a linux repo and it works like 110% better now.. the only catch is the synology has to have a intel processer not an ARM Processor

[AFalb]

Could you tell me where this blog is and is it working with DSM 6.1 or with an older Version?
I've an RS3614xs+ with an Intel Xeon E3-1230 v2 CPU

[AlainRussell]

These should still be relevant (apart from the SSH change - this can now be done on the web GUI .. Control Panel -> Teminal & SSH -> Advanced - > Set to "medium" - ok for us as this is not exposed externally) http://blog.millard.org/2014/11/use-syn ... itory.html We've been running ours as a Linux repo for ~2 years, the most reliable way imo. 2416+, Intel CPU running latest 6.1 patch.

[AFalb]

Yes, so you installed and configured it as a Linux Server when you still had DSM 5!
Upgrading from DSM 5 to DSM 6 keeps a lot of files so it still works, but when you try it on a fresh install of DSM 6.1 it will not work.

[jamerson]

We are trying to get a Synology Synology RS3617xs with 6X 10TB Seagate Ironwolf Pro however after reading this post, don't know now if we will go for a dedicated physical server or still for the synolgy.

[mark.roberts]

I have been using Synologys in my Veeam 9.5 environment for about a year now (I have about 7 Synologys and 3 Veeam BR servers). I have run them on DSM 5 and upgraded to 6. If you install PERL on the Synologys and enable SSH and home shares you can use them as straight Linux Repos.

This has worked pretty well for me. It makes deployment and management of them very easy.

Someone already posted this but I used these instructions.

http://blog.millard.org/2014/11/use-syn ... itory.html

Thanks,

Mark

[sg_sc]

Only on higher-end x86 Synology devices though, the lower ones can only be used as ISCSI or CIFS.

[AFalb]

As already mentioned in my post from may, using the Synology NAS as a Linux Server is impossible with a fresh DSM 6.1.
I use it with CIFS Repo but I'm not so happy about the performance.