Backups to Synology NAS

Availability for the Always-On Enterprise

Re: Backups to Synology NAS

Veeam Logoby sketchy » Tue Mar 25, 2014 2:31 pm 1 person likes this post

As I outlined in my Post "Practical Tips for a Veeam Backup and Recovery Depeloyment" at http://vmpete.com/2014/02/25/practical- ... eployment/ when using a Synology as a target, you will NOT want to use an NFS based connection from the Veeam server to the storage target. It will be noticeably slower, and as reported here, will have the inability to report the correct space, or perform any job management functions like cleaning up old jobs.
sketchy
Influencer
 
Posts: 15
Liked: 5 times
Joined: Sat Jul 18, 2009 2:50 am
Full Name: Pete Koehler

Re: Backups to Synology NAS

Veeam Logoby sbbots » Tue Mar 25, 2014 6:28 pm

Vitaliy S. wrote:If you decide to use CIFS shares as a backup target, then it will increase your data traffic between proxy and the NAS box, moreover the rebuild traffic (synthetic full) will not be kept locally and will pushed back and forth over the network which will decrease the overall job performance. Additionally, throttling rules (if you decide to use them) will not be applied as they require our run-time on the NAS box for communication. Finally, with all network glitches that might happen during the backup job run network shares cannot be considered as the best possible backup target.

I am using Synology DS412+ NASes at two remote locations as CIFS shares for Veeam backups and they run perfectly, although these are both very small sites (each with 1 host, 6 VMs, under 2TB of data). I did notice the long synthetic-full rebuild times and thus run active-full backups instead (prefer active anyway).

That being said, after reading this thread I feel like I am doing it wrong. Can anyone give an example of the size of their environment where they need to use the Synology NAS itself as the backup agent? Basically, at what point does the trade-off between performance vs complexity make it worthwhile NOT to simply use SMB?
sbbots
Enthusiast
 
Posts: 96
Liked: 25 times
Joined: Fri Aug 16, 2013 5:44 pm
Full Name: Matt B

Re: Backups to Synology NAS

Veeam Logoby Baud » Wed Apr 02, 2014 9:53 am

Hi,

I have the same error (8388608 TB) of free space of my DS713+ instead of ~3 TB.

I only have found "-x vmfs" in Mount.pm like jim3cantos.
Code: Select all
my $df_info = $self->runCommand("$self->{df} -P -x vmfs $_[0]");

This seems to be what we are looking for
Code: Select all
/bin/df -P -x vmfs

I tweak the file (removing -x vmfs) and re-tar, and rescan and get an error.

So, which Perl file in which tar file we can tweak to correct the free space displayed ?
Baud
Lurker
 
Posts: 1
Liked: never
Joined: Wed Apr 02, 2014 9:36 am

Re: Backups to Synology NAS

Veeam Logoby avit » Fri Apr 11, 2014 1:47 am

Vitaliy S. wrote:
avit wrote:2. Run a full backup (reverse-incremental 'mode') - this works fine, resulting in one full set of backups for all 11 of my VMs
3. The next time (and every subsequent time) the incremental runs, it will fail with the error message:

If you run an active full backup, then your job creates a new backup chain and should no longer care about previous points, at least your backup job should not fail.

="avit" Erase all restore points for my "Daily Backup" job, from the repository (Synology NAS acting as Linux server)

Also can you please clarify why do you this or this was done just on testing purposes?


Just for testing purposes, Vitaliy, i.e. just to reproduce the problem every time, with a "clean slate".
avit
Influencer
 
Posts: 13
Liked: 5 times
Joined: Thu Feb 20, 2014 1:06 am
Full Name: AVIT Engineer VAVA II

Re: Backups to Synology NAS

Veeam Logoby avit » Fri Apr 11, 2014 2:06 am

jim3cantos wrote:
avit wrote:What is happening here is that the VolumesHostDiscover module of Veeam Backup is trying to issue the command "/bin/df -P -x vmfs", but the BusyBox implementation of Linux used on my Synology NAS (and probably all of them) does not support the -x command, which tells it to exclude from the df listing all filesystems of type 'vmfs'. What Veeam sees is the "usage" reply from the df command, which obviously doesn't make sense to it.

I tweaked the perl script that sends this df with the -x option, and simply removed the text "-x vmfs" then saved the perl script and tar'd up the veeam_soap.tar package and put it back in the Veeam folder on my server. Now if I Rescan that repository, I see the proper size reported. There was no risk to me tweaking the script because my NAS doesn't have any VMFS filesystems on it anyway. I can only guess that when VolumesHostDiscover (the module that populates dialog boxes with your repository information, size, free space left etc) receives the wrong response back from BusyBox's df, it results in that spurious terabyte disk size.

We still haven't resolved the main issue I'm having with backing up to the NAS as a Linux server, and in actual fact this issue with df had absolutely nothing to do with the problem... I'm glad I looked into it, because I was all ready to accept Veeam Support's diagnosis that it was a problem with the df command. But having put a workaround in place by tweaking the script to remove "-x vmfs", I have proven that this isn't the cause.


Can you elaborate a little more how to make this change? I have found veeam_soap.tar in veeam server but I can't see any perl file inside with the command (df) to tweak.

thanks.


Sorry for the late response - I had to actually dig through a lot of files to find the VolumesHostDiscover module. I suggest you just grep all the files in the veeam_soap.tar file for the phrase "-x vmfs" and you will find what I was talking about. Oh okaaaaay, I'll do it for you:

Edit this file (assuming you have unpacked the veeam_soap tar file and are in the top level of that folder):
lib/Esx/System/Filesystem/Mount.pm

Look at line 72, it reads: my $df_info = $self->runCommand("$self->{df} -P -x vmfs $_[0]");

Change it to read: my $df_info = $self->runCommand("$self->{df} -P $_[0]");

Re-tar the file and throw it back into the folder you got it from (i.e. on your Veeam Backup & Recovery server). It goes without saying that you should have first made a backup copy of the tar file!

You'll then be able to scan the repository and see that it reports the correct file size.
avit
Influencer
 
Posts: 13
Liked: 5 times
Joined: Thu Feb 20, 2014 1:06 am
Full Name: AVIT Engineer VAVA II

Re: Backups to Synology NAS

Veeam Logoby avit » Fri Apr 11, 2014 2:16 am

jim3cantos wrote:update: I found the file to tweak (mount.pm) in veeam_soap.tar. I "untared" it, updated with worpad and "tar'd" again with 7-zip. I copied back to original folder and tried to resyncronize repository but it fails (after a while) with following error:

Code: Select all
[i]21/03/2014 10:30:30          Starting synchronization of backup repositories for all backup jobs
21/03/2014 10:30:31          Found 1 backup repositories
21/03/2014 10:32:12 Error    Processing backup repositories
21/03/2014 10:32:12 Error    Failed to synchronize backup repository backup copy linux direct Error: Timed out waiting for operation "(cd /tmp && perl veeam_soap3e961f6f-3bf5-46bc-87e9-ced1c41d2384.pl -d -c -l lib3e961f6f-3bf5-46bc-87e9-ced1c41d2384 -e /tmp/veeam_error3e961f6f-3bf5-46bc-87e9-ced1c41d2384 2>> /tmp/veeam_error3e961f6f-3bf5-46bc-87e9-ced1c41d2384) || cat /tmp/veeam_error3e961f6f-3bf5-46bc-87e9-ced1c41d2384 2>&1", timeout: 100000 ms
21/03/2014 10:32:12 Error    Failed to perform backup repositories synchronization[/i]


restored original veeam_soap.tar and syncronitation works ok....enough testing for today....


I think just because you changed your veeam_soap.tar file back to the original, that does not mean that the DF edit I described is the cause. In fact there is almost no way that my suggested edit could be the cause. All that's happened here is by fixing the 'df' command, the problem has moved from being one problem area (Mount.pm getting unexpected output from the Synology NAS's df command), and unearthed the next problem in the chain. So actually, the next problem that needs to be solved is this one you've shown in your logs (and in my log, below):

Code: Select all
Error    Timed out waiting for operation "(cd /tmp && perl veeam_soap{random-hex-chars}.pl -d -c -l lib{random-hex-chars} -e /tmp/veeam_error{random-hex-chars} 2>> /tmp/veeam_error{random-hex-chars}) || cat /tmp/veeam_error{random-hex-chars} 2>&1", timeout: 100000 ms   at Veeam.Backup.Common.CBackgroundFunctionRunner.Run(Function function, Form splashForm, Int32 minShowTime)


I'm guessing the timeout is because another command is failing to run. I'm sure Veeam's devs can look at the error message and figure out something?
I think my Veeam support case got closed (had some major crises at work which required a couple of weeks to resolve), but I might re-open it. Any devs on this forum, can you help me get the case escalated to the right level immediately? I don't want to have to fight my way through 1st-line support just to get to the level that this needs to be troubleshooted at. That said, I'm totally amenable to walking through the obvious initial checks etc so no assumptions are made!
avit
Influencer
 
Posts: 13
Liked: 5 times
Joined: Thu Feb 20, 2014 1:06 am
Full Name: AVIT Engineer VAVA II

Re: Backups to Synology NAS

Veeam Logoby avit » Fri Apr 11, 2014 4:16 am

Just a quick one in case any Veeam techs are reading this. My case numbers are:

00548863 - new case referencing the old case, just opened it a few hours ago
00517970 - old case, lots of notes and stuff but due to inactivity from my side, it got closed (I had other crises I had to focus on for a few weeks)
avit
Influencer
 
Posts: 13
Liked: 5 times
Joined: Thu Feb 20, 2014 1:06 am
Full Name: AVIT Engineer VAVA II

Re: Backups to Synology NAS

Veeam Logoby avit » Fri Apr 11, 2014 6:13 am

Hmm this is slightly embarrassing - well, not so much embarrassing as interesting - but no sooner than I had opened my new case, but I edited my backup job to use the original repository (direct to the Synology NAS as a Linux Server) and ran the job (not as Active Full, but as an incremental) and it ran, absolutely no hitches and no problems.

Things that changed today:

1) Updated to the latest Veeam B&R patch - I am now on version 7.0.0.839
2) Updated Synology DSM to the very latest version - I am now on version DSM 5.0-4458 Update 2
3) That allowed me to update the Perl package on the DSM to the latest version
4) While googling other forum posts regarding this issue, I noticed that someone said "install the Python package using Synology package manager" - so I did that too

Anyway... there's still more testing to be done, because this incremental that just ran OK, was an incremental on top of a job that I ran earlier today using a full-blown Linux server (a VM actually) that has an NFS connection to the Synology NAS. What I want to see is a full backup run directly to the Synology NAS (as a Linux server), then subsequent daily incrementals run OK.

I'll keep you posted with my results...
avit
Influencer
 
Posts: 13
Liked: 5 times
Joined: Thu Feb 20, 2014 1:06 am
Full Name: AVIT Engineer VAVA II

Re: Backups to Synology NAS

Veeam Logoby avit » Fri Apr 11, 2014 6:14 am

Sorry to keep posting... but here is that forum post I'd read that said you need the Python package - http://forum.synology.com/enu/viewtopic ... 59&t=76978 ... I have to wonder if the guy was confused and meant the Perl package. I don't really see why Veeam would need Python but I'm happy to be corrected on that... Veeam techs?
avit
Influencer
 
Posts: 13
Liked: 5 times
Joined: Thu Feb 20, 2014 1:06 am
Full Name: AVIT Engineer VAVA II

Re: Backups to Synology NAS

Veeam Logoby foggy » Fri Apr 11, 2014 11:23 am

avit wrote:Any devs on this forum, can you help me get the case escalated to the right level immediately? I don't want to have to fight my way through 1st-line support just to get to the level that this needs to be troubleshooted at. That said, I'm totally amenable to walking through the obvious initial checks etc so no assumptions are made!

I've asked support guys to handle your new case properly.

avit wrote:Sorry to keep posting... but here is that forum post I'd read that said you need the Python package - http://forum.synology.com/enu/viewtopic ... 59&t=76978 ... I have to wonder if the guy was confused and meant the Perl package. I don't really see why Veeam would need Python but I'm happy to be corrected on that... Veeam techs?

I also think there's a confusion here, since a Perl-based wrapper script is used to deploy Veeam B&R run-time agent on Linux-type repository.
foggy
Veeam Software
 
Posts: 14346
Liked: 1054 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Backups to Synology NAS

Veeam Logoby avit » Sat Apr 12, 2014 11:31 pm 2 people like this post

Update: It's working!

Just to re-cap, here's my setup:

Veeam Backup & Recovery patched to 7.0.0.839 (aka "patch 3")
One ESX server, 11 VMs, total size 4.0TB
Storage device: Synology DS1513+ NAS
Backup repository settings: NB: any settings not mentioned are set to defaults
  • Linux Server - Synology NAS
  • Advanced Repository settings: none
  • Limit max concurrent tasks: 2
Backup job settings:
  • Backup repository: the above NAS/Linux repository
  • Reverse incremental with a monthly Active Full backup
Veeam_soap.tar --> Mount.pm patched to remove "-x vmfs" from the "df" command on line 72

Synology changes made:
Upgraded Synology DSM to the latest available release (as of 9th April 2014) - DSM 5.0-4458 Update 2
After upgrade, a DSM Perl package upgrade was available, which I also installed - 5.18.1-0011

After the above changes, I did the following backup job runs:

  1. I already had a completed Active Full job done using a separate full blown Linux server with the Synology NAS mounted via NFS. So with the above upgraded setup, I tried an incremental - mainly out of impatience to see if it had fixed the problem. Success.
  2. Ran another incremental to ensure I wasn't seeing things. Success.
  3. Ran an Active Full - success
  4. Ran two more incrementals - success
  5. Ran another Active Full last night - success
  6. Also note that any and all Populate buttons in the various Repository and Backup Job edit pages, are all reporting the correct full size of my NAS and the correct space available (AFAIK, this is the only thing that the Mount.pm fix within veeam_soap.tar addresses)

On my last run of the Active Full, I got the following speeds:
  • Duration: 6:00:59
  • Processing rate: 292 MB/s
  • Bottleneck: source
  • Data processed: 5.9 TB (100%)
  • Read: 5.9 TB
  • Transferred: 1.5 TB (3.9x)
The reverse incrementals are running at: 1h27m, 61 MB/s, source, 2.6 TB, 255 GB, 5.1 GB (49.6x) with the speed graph showing "Speed: 95.8 MB/s"

When I was trying to run these jobs via CIFS and Linux-->NFS-->NAS I was getting a lot of "connection was forcibly closed" and other connectivity errors, along with about 20 MB/s speeds. (I have 2x 1GE link aggregation although due to the load-balancing algorithm used with link aggregation, I doubt it's only ever going to fully utilise 1x1GE - it may do so currently as I have two backup proxies). I'm happy with those speeds though - it means I can easily get a full backup done between 11am and 7am.

So I think I have proven that my setup is now working. I feel a bit stupid because I had just re-opened my case with Veeam Support! But hopefully we can use these findings to help get this setup working for other users.
avit
Influencer
 
Posts: 13
Liked: 5 times
Joined: Thu Feb 20, 2014 1:06 am
Full Name: AVIT Engineer VAVA II

Re: Backups to Synology NAS

Veeam Logoby ITP-Stan » Mon Apr 14, 2014 8:08 am

We use a lot of Synology NAS devices to backup to from VEEAM and have no problems using CIFS.
ITP-Stan
Veeam ProPartner
 
Posts: 63
Liked: 5 times
Joined: Mon Feb 18, 2013 10:45 am
Full Name: Stan (IF-IT4U)

Re: Backups to Synology NAS

Veeam Logoby ArnoutBoer » Mon Apr 14, 2014 1:29 pm

Our remote setup is:

Synology NAS + Veeam Proxy Windows PC.

Can we use only the Synology NAS? Is the Synology running the Transport software?
ArnoutBoer
Lurker
 
Posts: 2
Liked: never
Joined: Thu Oct 31, 2013 8:16 am
Full Name: Arnout Boer

Re: Backups to Synology NAS

Veeam Logoby foggy » Mon Apr 14, 2014 2:26 pm

ArnoutBoer wrote:Can we use only the Synology NAS? Is the Synology running the Transport software?

Only in case it has ordinary Linux installed, as SSH and Perl are required to deploy Veeam agent.
foggy
Veeam Software
 
Posts: 14346
Liked: 1054 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Backups to Synology NAS

Veeam Logoby bdwill » Mon Apr 14, 2014 3:11 pm

Definitely going to give this a try. I constantly get "connection was forcibly closed" errors using CIFS.
@bdwill
New Orleans VMUG Leader
vExpert 2012, VCP5-DV
bdwill
Lurker
 
Posts: 2
Liked: never
Joined: Sat May 29, 2010 9:05 pm
Full Name: Brandon

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Inferno131, jim3cantos and 40 guests