Backups to Synology NAS

[sketchy]

As I outlined in my Post "Practical Tips for a Veeam Backup and Recovery Depeloyment" at http://vmpete.com/2014/02/25/practical- ... eployment/ when using a Synology as a target, you will NOT want to use an NFS based connection from the Veeam server to the storage target. It will be noticeably slower, and as reported here, will have the inability to report the correct space, or perform any job management functions like cleaning up old jobs.

[sbbots]

If you decide to use CIFS shares as a backup target, then it will increase your data traffic between proxy and the NAS box, moreover the rebuild traffic (synthetic full) will not be kept locally and will pushed back and forth over the network which will decrease the overall job performance. Additionally, throttling rules (if you decide to use them) will not be applied as they require our run-time on the NAS box for communication. Finally, with all network glitches that might happen during the backup job run network shares cannot be considered as the best possible backup target.

I am using Synology DS412+ NASes at two remote locations as CIFS shares for Veeam backups and they run perfectly, although these are both very small sites (each with 1 host, 6 VMs, under 2TB of data). I did notice the long synthetic-full rebuild times and thus run active-full backups instead (prefer active anyway).

That being said, after reading this thread I feel like I am doing it wrong. Can anyone give an example of the size of their environment where they need to use the Synology NAS itself as the backup agent? Basically, at what point does the trade-off between performance vs complexity make it worthwhile NOT to simply use SMB?

[Baud]

Hi,

I have the same error (8388608 TB) of free space of my DS713+ instead of ~3 TB.

I only have found "-x vmfs" in Mount.pm like jim3cantos.

Code: Select all

my $df_info = $self->runCommand("$self->{df} -P -x vmfs $_[0]");

This seems to be what we are looking for

Code: Select all

/bin/df -P -x vmfs

I tweak the file (removing -x vmfs) and re-tar, and rescan and get an error.

So, which Perl file in which tar file we can tweak to correct the free space displayed ?

[avit]

If you run an active full backup, then your job creates a new backup chain and should no longer care about previous points, at least your backup job should not fail.
Also can you please clarify why do you this or this was done just on testing purposes?

Just for testing purposes, Vitaliy, i.e. just to reproduce the problem every time, with a "clean slate".

[avit]

Can you elaborate a little more how to make this change? I have found veeam_soap.tar in veeam server but I can't see any perl file inside with the command (df) to tweak.

thanks.

Sorry for the late response - I had to actually dig through a lot of files to find the VolumesHostDiscover module. I suggest you just grep all the files in the veeam_soap.tar file for the phrase "-x vmfs" and you will find what I was talking about. Oh okaaaaay, I'll do it for you:

Edit this file (assuming you have unpacked the veeam_soap tar file and are in the top level of that folder):
lib/Esx/System/Filesystem/Mount.pm

Look at line 72, it reads: my $df_info = $self->runCommand("$self->{df} -P -x vmfs $_[0]");

Change it to read: my $df_info = $self->runCommand("$self->{df} -P $_[0]");

Re-tar the file and throw it back into the folder you got it from (i.e. on your Veeam Backup & Recovery server). It goes without saying that you should have first made a backup copy of the tar file!

You'll then be able to scan the repository and see that it reports the correct file size.

[avit]

update: I found the file to tweak (mount.pm) in veeam_soap.tar. I "untared" it, updated with worpad and "tar'd" again with 7-zip. I copied back to original folder and tried to resyncronize repository but it fails (after a while) with following error:

Code: Select all

[i]21/03/2014 10:30:30          Starting synchronization of backup repositories for all backup jobs
21/03/2014 10:30:31          Found 1 backup repositories
21/03/2014 10:32:12 Error    Processing backup repositories
21/03/2014 10:32:12 Error    Failed to synchronize backup repository backup copy linux direct Error: Timed out waiting for operation "(cd /tmp && perl veeam_soap3e961f6f-3bf5-46bc-87e9-ced1c41d2384.pl -d -c -l lib3e961f6f-3bf5-46bc-87e9-ced1c41d2384 -e /tmp/veeam_error3e961f6f-3bf5-46bc-87e9-ced1c41d2384 2>> /tmp/veeam_error3e961f6f-3bf5-46bc-87e9-ced1c41d2384) || cat /tmp/veeam_error3e961f6f-3bf5-46bc-87e9-ced1c41d2384 2>&1", timeout: 100000 ms
21/03/2014 10:32:12 Error    Failed to perform backup repositories synchronization[/i] 

restored original veeam_soap.tar and syncronitation works ok....enough testing for today....

I think just because you changed your veeam_soap.tar file back to the original, that does not mean that the DF edit I described is the cause. In fact there is almost no way that my suggested edit could be the cause. All that's happened here is by fixing the 'df' command, the problem has moved from being one problem area (Mount.pm getting unexpected output from the Synology NAS's df command), and unearthed the next problem in the chain. So actually, the next problem that needs to be solved is this one you've shown in your logs (and in my log, below):

Code: Select all

Error    Timed out waiting for operation "(cd /tmp && perl veeam_soap{random-hex-chars}.pl -d -c -l lib{random-hex-chars} -e /tmp/veeam_error{random-hex-chars} 2>> /tmp/veeam_error{random-hex-chars}) || cat /tmp/veeam_error{random-hex-chars} 2>&1", timeout: 100000 ms   at Veeam.Backup.Common.CBackgroundFunctionRunner.Run(Function function, Form splashForm, Int32 minShowTime)

I'm guessing the timeout is because another command is failing to run. I'm sure Veeam's devs can look at the error message and figure out something?
I think my Veeam support case got closed (had some major crises at work which required a couple of weeks to resolve), but I might re-open it. Any devs on this forum, can you help me get the case escalated to the right level immediately? I don't want to have to fight my way through 1st-line support just to get to the level that this needs to be troubleshooted at. That said, I'm totally amenable to walking through the obvious initial checks etc so no assumptions are made!

[avit]

Just a quick one in case any Veeam techs are reading this. My case numbers are:

00548863 - new case referencing the old case, just opened it a few hours ago
00517970 - old case, lots of notes and stuff but due to inactivity from my side, it got closed (I had other crises I had to focus on for a few weeks)

[avit]

Hmm this is slightly embarrassing - well, not so much embarrassing as interesting - but no sooner than I had opened my new case, but I edited my backup job to use the original repository (direct to the Synology NAS as a Linux Server) and ran the job (not as Active Full, but as an incremental) and it ran, absolutely no hitches and no problems.

Things that changed today:

1) Updated to the latest Veeam B&R patch - I am now on version 7.0.0.839
2) Updated Synology DSM to the very latest version - I am now on version DSM 5.0-4458 Update 2
3) That allowed me to update the Perl package on the DSM to the latest version
4) While googling other forum posts regarding this issue, I noticed that someone said "install the Python package using Synology package manager" - so I did that too

Anyway... there's still more testing to be done, because this incremental that just ran OK, was an incremental on top of a job that I ran earlier today using a full-blown Linux server (a VM actually) that has an NFS connection to the Synology NAS. What I want to see is a full backup run directly to the Synology NAS (as a Linux server), then subsequent daily incrementals run OK.

I'll keep you posted with my results...

[avit]

Sorry to keep posting... but here is that forum post I'd read that said you need the Python package - http://forum.synology.com/enu/viewtopic ... 59&t=76978 ... I have to wonder if the guy was confused and meant the Perl package. I don't really see why Veeam would need Python but I'm happy to be corrected on that... Veeam techs?

[foggy]

Any devs on this forum, can you help me get the case escalated to the right level immediately? I don't want to have to fight my way through 1st-line support just to get to the level that this needs to be troubleshooted at. That said, I'm totally amenable to walking through the obvious initial checks etc so no assumptions are made!

I've asked support guys to handle your new case properly.

Sorry to keep posting... but here is that forum post I'd read that said you need the Python package - http://forum.synology.com/enu/viewtopic ... 59&t=76978 ... I have to wonder if the guy was confused and meant the Perl package. I don't really see why Veeam would need Python but I'm happy to be corrected on that... Veeam techs?

I also think there's a confusion here, since a Perl-based wrapper script is used to deploy Veeam B&R run-time agent on Linux-type repository.

[avit]

Update: It's working!

Just to re-cap, here's my setup:

Veeam Backup & Recovery patched to 7.0.0.839 (aka "patch 3")
One ESX server, 11 VMs, total size 4.0TB
Storage device: Synology DS1513+ NAS
Backup repository settings: NB: any settings not mentioned are set to defaults

• Linux Server - Synology NAS
• Advanced Repository settings: none
• Limit max concurrent tasks: 2

Backup job settings:

• Backup repository: the above NAS/Linux repository
• Reverse incremental with a monthly Active Full backup

Veeam_soap.tar --> Mount.pm patched to remove "-x vmfs" from the "df" command on line 72

Synology changes made:
Upgraded Synology DSM to the latest available release (as of 9th April 2014) - DSM 5.0-4458 Update 2
After upgrade, a DSM Perl package upgrade was available, which I also installed - 5.18.1-0011

After the above changes, I did the following backup job runs:

1. I already had a completed Active Full job done using a separate full blown Linux server with the Synology NAS mounted via NFS. So with the above upgraded setup, I tried an incremental - mainly out of impatience to see if it had fixed the problem. Success.
2. Ran another incremental to ensure I wasn't seeing things. Success.
3. Ran an Active Full - success
4. Ran two more incrementals - success
5. Ran another Active Full last night - success
6. Also note that any and all Populate buttons in the various Repository and Backup Job edit pages, are all reporting the correct full size of my NAS and the correct space available (AFAIK, this is the only thing that the Mount.pm fix within veeam_soap.tar addresses)

On my last run of the Active Full, I got the following speeds:

• Duration: 6:00:59
• Processing rate: 292 MB/s
• Bottleneck: source
• Data processed: 5.9 TB (100%)
• Read: 5.9 TB
• Transferred: 1.5 TB (3.9x)

The reverse incrementals are running at: 1h27m, 61 MB/s, source, 2.6 TB, 255 GB, 5.1 GB (49.6x) with the speed graph showing "Speed: 95.8 MB/s"

When I was trying to run these jobs via CIFS and Linux-->NFS-->NAS I was getting a lot of "connection was forcibly closed" and other connectivity errors, along with about 20 MB/s speeds. (I have 2x 1GE link aggregation although due to the load-balancing algorithm used with link aggregation, I doubt it's only ever going to fully utilise 1x1GE - it may do so currently as I have two backup proxies). I'm happy with those speeds though - it means I can easily get a full backup done between 11am and 7am.

So I think I have proven that my setup is now working. I feel a bit stupid because I had just re-opened my case with Veeam Support! But hopefully we can use these findings to help get this setup working for other users.

[ITP-Stan]

We use a lot of Synology NAS devices to backup to from VEEAM and have no problems using CIFS.

[ArnoutBoer]

Our remote setup is:

Synology NAS + Veeam Proxy Windows PC.

Can we use only the Synology NAS? Is the Synology running the Transport software?

[foggy]

Can we use only the Synology NAS? Is the Synology running the Transport software?

Only in case it has ordinary Linux installed, as SSH and Perl are required to deploy Veeam agent.

[bdwill]

Definitely going to give this a try. I constantly get "connection was forcibly closed" errors using CIFS.

[bdwill]

I spent some time trying to get this to work and hit a speed bump. Here's the error I'm getting when trying to backup:

Code: Select all

4/15/2014 2:38:01 AM :: Error: scp: /volume1/VEEAM/Linux/Linux.vbm: set mode: Operation not permitted

I added the admin credentials for the Synology but I don't have elevate to root checked (subsequently I don't have automatically add account to suoders checked). Is this necessary? Also, the credentials work when populating the storage and directories on the repository.When I do check the box for "elevate non-root," I get the timeout that avit previously described.

Code: Select all

Error    Timed out waiting for operation "(cd /tmp && perl veeam_soap{random-hex-chars}.pl -d -c -l lib{random-hex-chars} -e /tmp/veeam_error{random-hex-chars} 2>> /tmp/veeam_error{random-hex-chars}) || cat /tmp/veeam_error{random-hex-chars} 2>&1", timeout: 100000 ms   at Veeam.Backup.Common.CBackgroundFunctionRunner.Run(Function function, Form splashForm, Int32 minShowTime)

Opened case # 550935

[avit]

I spent some time trying to get this to work and hit a speed bump. Here's the error I'm getting when trying to backup:

Code: Select all

4/15/2014 2:38:01 AM :: Error: scp: /volume1/VEEAM/Linux/Linux.vbm: set mode: Operation not permitted

I added the admin credentials for the Synology but I don't have elevate to root checked (subsequently I don't have automatically add account to suoders checked). Is this necessary? Also, the credentials work when populating the storage and directories on the repository.When I do check the box for "elevate non-root," I get the timeout that avit previously described.

Code: Select all

Error    Timed out waiting for operation "(cd /tmp && perl veeam_soap{random-hex-chars}.pl -d -c -l lib{random-hex-chars} -e /tmp/veeam_error{random-hex-chars} 2>> /tmp/veeam_error{random-hex-chars}) || cat /tmp/veeam_error{random-hex-chars} 2>&1", timeout: 100000 ms   at Veeam.Backup.Common.CBackgroundFunctionRunner.Run(Function function, Form splashForm, Int32 minShowTime)

Opened case # 550935

Hi bdwill, that's a shame that you're getting that error... hmm... it does seem like a permissions thing, doesn't it. Did you check the ownership of the folder in question? (/volume1/VEEAM/Linux etc etc) ... also I made some changes ages ago (which didn't fix my problem at the time, but you never know, I may have ended up suffering the Operation Not Permitted issue) -- to do with the permissions of the .ssh folder - it was just one of those things I found while desperately Googling for solutions, but I don't remember exactly what it was - but basically there is a hidden folder called ".ssh" in the root of your home folder, i.e. for the Synology 'admin' user this would be /volume1/homes/admin. My .ssh folder has the following permissions:

Code: Select all

drwxr--r--    2 admin    users         4096 Feb  4 23:39 .ssh

And the contents of the folder:

Code: Select all

MYV2-NAS-001> ls -la
drwxr--r--    2 admin    users         4096 Feb  4 23:39 .
drwxr-xr-x    5 admin    users         4096 Apr 14 12:54 ..
-rwxr--r--    1 admin    users          394 Feb  4 23:39 known_hosts

Just in case that helps. Also, note that I am backing up to this folder: /volume1/homes/admin/veeam_backups ... the permissions, bizarrely, are actually rwxrwxrwx, I'm not sure if I changed that out of desperation also:

Code: Select all

MYV2-NAS-001> ls -la
drwxrwxrwx    4 admin    users         4096 Apr  8 14:15 veeam_backups

But I notice that the subfolders the Veeam itself has created below that, also have rwxrwxrwx permissions, and those folders would have been created by Veeam itself, so I'm assuming the veeam_backup folder's permissions are correct.

Anyway, have a good look at all the permissions, post them here, and maybe let us know what you come back with. The other really good source of digging into these errors (and the way we found the "df" issue in Mount.pm within veeam_soap.tar) was trawling through the logs. The logs will be huge so what I'd recommend you do is zip up your existing logs, nuke the log files and run the backup again, and that way you'll have way less stuff to trawl through. I found the "df" errors within the following path: logfilepath/This server/VeeamShell.log - so in theory that'll be where you see other Linuxy/NAS related errors.

Let's keep digging! Veeam is surprisingly easy to troubleshoot because it's so good with logging everything. And always let me know if there's anything I can do to help - I'm always happy to dig through my setup and compare settings, permissions, etc. I think ultimately it would be nice to build a sticky post that outlines the Must-haves for getting this setup to work.... i.e. Synology NAS with x86 processor, latest DSM version, Veeam 7.x patch #3, latest Perl version, tweak permission X, etc. Ultimately it would be nice if Veeam would be able to alter Mount.pm so that it detects the "Incorrect usage" response that comes back from BusyBox's ash shell's "df" command, etc and alters its behaviour accordingly. The danger is that the "-x vmfs" command was obviously put there for a reason, so me just removing it for my purposes is OK but adding it as a blanket feature would require some safeguards.

Anyway let me know what you come up with on the above....

[bramsd]

Hi,

I had the same error.
I solve this by changing the ownership of the folder on the Syno with the user defined in the proxy.

Hope that can help the person who is reading this message.

Didier

[Mathieu]

Can someone please post a working version of veeam_soap.tar with the modification to correctly read the free space of a synology NAS.
I'm not getting it to work. Changing the Mount.pm shouldn't be the problem but maybe at tar'ing againg (using 7 zip) something goes wrong on windows.....

[thijsm]

I post this because it might be useful for some people
Installed veaam transport agent on Synlogy DS415p (DSM 5.0-4493 Update 3) in a few steps.

1. Create shared folder for veeam repository.
2. Install Perl, available in package center.
3. enable ssh service > control panel > system >terminal & snmp
4. enable user home service > control panel > user > Advanced > turn on enable user home service.
Ready, veeam backup will now run without any problems

You still need to adjust veeam_soap.tar to see the correct disk space
hopefully someone can post a short how to, i have not don this jet, don't know how to tar/untar on a windows box.

Be careful if you open ssh.
the default admin account and password is also the password for the root account, default ssh access only works for admin and root account.

[d.lansinklesscher]

This solution works great on both Synology and Qnap devices. If you want to get it to work on a Qnap there's some extra steps. See:
http://forum.qnap.com/viewtopic.php?f=1 ... 56#p430156

@Veeam: Is there any chance that the veeam_soap.tar wil be updated in one of the next releases to display the correct free-space?

[Gostev]

@Dennis let me check first if this change does not impact some other functionality.

[mcdonaldb]

If using the Synology as a Repository, is this in theory faster than mapping ISCSI LUN from Synology to windows Veeam backup server? Also, what about copy job's/archive, could you use the usb port on the synology for this to copy straight from nas to usb, or is this still going to move through the network?

[Vitaliy S.]

Hi Brian,

iSCSI LUN mapped to a Windows server is a preferred way to expose your Synology box to the Veeam backup server, as it is more reliable and sometimes it can give you better performance rates during synthetic full and transform operations. As regards your second question, then if you attach it to the backup server and will be able to see it as a regular disk, then the traffic will go over the USB connection to your Synology NAS, not sure it will provide better rates though... need to test this configuration first.

Thanks!

[d.lansinklesscher]

Another issue that I have come across at least on a Qnap is that after a while the ramdisk of the nas runs out of diskspace. This probably is because of logs.

As a workaround I scheduled a weekly reboot.

Maybe Veeam can also address this by putting the logs on persistent storage off the nas.

[Vitaliy S.]

I'm not sure that I fully understand the issue - do you say that Veeam logs are cleaned up when you reboot the device?

[d.lansinklesscher]

Yes, because it is a ramdisk.

[veremin]

Can you elaborate on how you're using a ramdisk of given device? Also, what logs you're talking about? The actual screenshots might be helpful in this case. Thanks.

[d.lansinklesscher]

Filesystem Size Used Available Use% Mounted on
/dev/ram0 151.1M 132.9M 18.1M 88% /
devtmpfs 487.9M 4.0k 487.9M 0% /dev
tmpfs 64.0M 22.7M 41.3M 35% /tmp
/dev/md9 509.5M 107.7M 401.8M 21% /mnt/HDA_ROOT
/dev/md0 10.8T 7.3T 3.5T 67% /share/MD0_DATA
/dev/md13 371.0M 253.8M 117.2M 68% /mnt/ext
tmpfs 32.0M 0 32.0M 0% /.eaccelerator.tmp

As you can see above, on a Qnap the filesystem is running on a ramdisk, this includes /var/log where Veeam stores it's job logs. Because this disk is pretty small it fills up easily. Therefore I think it would be better to put these logs in a location where:
-there is more space
-logs’s don’t get erased after a reboot

Maybe it’s an idea to put them in the same place as the created repository?

[Vitaliy S.]

Thanks for the output. I'm not a Linux guru, but I believe there should be a possibility to redirect /var/logs folder to any other disk in the Sinology config, right?