QNAP RSYNC breaks on VBK files

[merc123]

I'm having an issue transferring the VBK files to a remote site.

I am using a pair of QNAP TS-853 Pro's to backup a Veeam backup job offsite. I've tried NAS to NAS, RSYNC and RRTR and all jobs fail and I realized it's only on the VBK files. The VIB's transmit just fine. The VBK will begin transmitting and will get to say 15 GB of the 120 GB or 17 GB of 24 GB and fail. They all fail anywhere between 68% and 90% of the transmitted file.

Additionally I've tried to add the remote NAS as a NFS target on my Veeam backup server (Win Server 2008 R2) and copy it over that way and it fails stating the connection was reset. The QNAP during the RSYNC job states the connection was reset and the RTRR says broken pipe. Not much to go on.

VeeamBackup 9.0 and latest firmware on the QNAP's using RSYNC 3.0.7.

QNAP NAS RTRR job settings:
Delete extra files, detect sparse files (tried on and off), ignore symbolic links and replicate ACL and extended attributes

Backup Job Settings:
5 VM's (900G), each is saved separately
Local SAN repository, 5 restore points.
Incremental, active full on Saturday.
Remove deleted VMs data after 7 days
Inline data duplication and exclude swap file blocks
Compression: Optimal
Storage Optimization: LAN Target
Use CBT data, enable CBT for all protected VMs

Application Aware and Enable guest file system indexing.


Repository Settings:
Limit concurrent tasks: 4
Use per-VM backup files
Enable vPower: Disabled

[lando_uk]

I wouldn't use SAN replication of any kind for VBK's, as you could replicate corruptions. If you use a Veeam copy job for your off-sites, it'll work much better and you can also setup health checks that look at the copy to check its consistency.

[merc123]

The only way I can set it up (that I know of) is via NFS to do the copy but QNAP doesn't allow Veeam to install the agents. It errors out with a PerlSoap error. All of the work arounds I have found for this do not work.

What I may be able to do is do a secondary copy job to the local NAS which then will backup to the offsite NAS via RTRR. That way it can check the consistency on the local NAS?

[alanbolte]

Do you have any Windows or Linux servers at the same location of the offsite QNAP, so you could run the datamovers on that server, while storing the data on the QNAP?

Could there be an advanced firewall deciding to close your connections?

Are you copying data to the default Ext4 storage, or have you formatted a volume with a more limited file system, like 1KB Ext3?

[merc123]

Alan:
No servers offsite, just the NAS. I was finally able to successfully get one, 24GB VBK to transfer over night. What I did notice is the transfer speed started at around 2 MB/s, dropped to 1 MB/s after a few GB and ended up hovering around 500-700KB/s for the duration of the upload. I've started running a packet capture on the firewall to see if I can see anything. I don't think the firewall is closing it down as I see no errors or warnings in the logs indicating it's terminating the transfer as I have verbose logging.

Using ext4:

Code: Select all

[~] # mount | grep "^/dev"
/dev/md9 on /mnt/HDA_ROOT type ext3 (rw,data=ordered)
/dev/mapper/cachedev1 on /share/CACHEDEV1_DATA type ext4 (rw,usrjquota=aquota.user,jqfmt=vfsv0,user_xattr,data=ordered,delalloc,acl)
/dev/md13 on /mnt/ext type ext3 (rw,data=ordered)

[merc123]

Over the weekend it transferred 539 GB with only 3 restarts. I'm guessing it has more to do with the network connections between on and off site as opposed to Veeam or the QNAP.