Backup Copy errors to QNAP NAS via SMB

[twyrick]

We're running Veeam 7 with the latest service patches applied, and I'm still running up against an issue trying to make off-site backup copies. Our environment is fairly straightforward. Here in our Bethesda, MD office, we have a Windows Server 2008R2 machine running Veeam and doing nightly backups of 5 virtual machines running on an HP VMWare ESXi 5.1 server.

It works flawlessly doing the local backups. But our goal is to keep an off-site copy of those nightly backups on the QNAP NAS running in our Chicago office. I configured a new Backup Copy job in Veeam for the purpose after configuring a "Veeam" SMB share on the NAS and configuring the proper read/write permissions.

What's happening is, 2 of the 5 VM's make successful backup copies to the shared folder on the NAS, but the other 3 never do.

When I look at the logs in Veeam, it appears the ones that fail to transfer will complete partially, followed by reporting "An existing connection was forcibly closed by the remote host." and a retry attempt. The retry attempts appear to each get a little bit further along in the data transfer, before they happen again, until it reaches a 5 retry attempt maximum and gives up with the failure message.

All of this is happening over a 50mbit dedicated Internet circuit, via a VPN tunnel created with Cisco ASA gear at both ends. For pretty much everything else we do over this connection, we don't notice any data interruptions or issues.

What I find most interesting is that it's always the same 3 virtual machine backups which fail to copy, even if i delete everything on the NAS and start the backup copy process over from scratch. Two of these are quite large VM images, but the third one which consistently fails is actually smaller in size than one of the two that always seem to copy over successfully.

The only thing I can think of that all of the VM's have in common which fail is the fact they're all Windows Server 2008R2 servers running MS SQL Server instances.

Has anyone else run into anything similar? I know the recommended way to do this Backup Copy over a WAN link is by using Windows servers on both sides along with Veeam ... but we already invested a lot of money into the QNAP and hard drives in it, so we'd like to find a way to make this work.

[foggy]

Tom, for effective data transfer to remote location, you definitely need a Windows server at the target location. Not only this will allow you to use WAN accelerators deployed on both sides to minimize the transferred traffic, but also enable resume option on connection drops and keep all the transform traffic in the off-site location (otherwise, with CIFS repository, backup data will be sent back and forth over the WAN link).

Also I recommend you to initially seed your backup copy jobs, especially for larger VMs.

[Vitaliy S.]

Do you have an ability to present this QNAP device in other way, for example via iSCSI?

[twyrick]

Do you have an ability to present this QNAP device in other way, for example via iSCSI?

The QNAP should let me create a share via NFS as well as SMB. I believe iSCSI is only an option if we disable all of its "intelligent" sharing features (integration with our Windows Active Directory and enforcing ACLs, etc. etc.) and tell it to act like "dumb" additional drive storage for servers linked to it via iSCSI protocol. That's not really an option for us since the NAS has a number of shares configured on it for daily use by employees in the office, with various file and folder permissions configured on it. In fact, the office purchased it specifically to eliminate the use of an older Windows 2003 server and free up that Microsoft software license. So the QNAP *is* the file server for them now.

From what I'm gathering here, the copy failures I'm experiencing seem to be caused by some kind of bug or limitation of SMB/CIFS protocol over a slow link? (Perhaps it doesn't handle transfers well when multiple requests are generated simultaneously to send AND receive data, like Veeam apparently does during these backup copy jobs?)

I guess I was just hoping someone else out there was in a similar situation and might have found a work-around or alternate solution? I could try creating the shared folder on the QNAP as NFS to see if that's an improvement.

When searching for this on Google the other day, I ran across some other message thread talking about a somewhat similar issue, except I believe the person involved was trying to do Replication jobs instead of Backup Copy jobs. He got into an explanation of how he had to tweak a number of settings having to do with MTU size and so forth with his routers. That made me wonder if it's actually the Cisco ASAs and their inter-office VPN tunnels causing my problems at the heart of things? But that goes a bit beyond my comfort zone, as I'm not much of an expert in Cisco routers -- and a mistake playing with the configuration could take down the link between our offices.

[Vitaliy S.]

Got it! In this case I would also recommend to use foggy's advice that should address the issue you observe.

[twyrick]

Tom, for effective data transfer to remote location, you definitely need a Windows server at the target location. Not only this will allow you to use WAN accelerators deployed on both sides to minimize the transferred traffic, but also enable resume option on connection drops and keep all the transform traffic in the off-site location (otherwise, with CIFS repository, backup data will be sent back and forth over the WAN link).

Also I recommend you to initially seed your backup copy jobs, especially for larger VMs.

I was curious about this idea to initially seed the backup copy job.

Is there a procedure to follow to do this properly or the "best" way? I'm thinking this would involve creating a new Backup Copy job pointing to a repository I create on a local drive with sufficient free disk space. Then tell it to run only once, where it would do the full copy of the VMs involved. Then I'd copy those files over to an external USB drive and mail it to the remote site, to have our sysadmin there copy the contents to the NAS.

But once that's all done, if I point the Backup Copy job to the new repository which points at the shared folder on the NAS and schedule it to start running from that evening forward, will it seamlessly pick up the full backup images on the NAS and start doing incremental backups from there?

[Vitaliy S.]

Here is the procedure to follow if you want to pre-seed your backup copy job > Seeding offsite backup files for a Backup Copy Job

[acatic]

I have also been occasionally experiencing this issue (with my QNAP/ESX/Veeam), although it's all at the local site for me. In the vast majority of the cases, it's when a bigger job (~1TB) reaches 10hrs' duration. I don't know why it's picky about that time, but it's been rather consistent... 10hrs; although, the last run crapped at 12hrs. The one I'm struggling with is currently at 10h14. Disk is ~900GB, and job is at 20%.

It doesn't show in my ticket history as having asked Veeam about this directly, although I did discuss with them on the Spiceworks forum, as well as with QNAP some months ago (as the error says, the remote host timed out). I somehow had it clear up with persistence and let it alone, don't recall the details. (I'm having issues connecting to that forum since last week.)

Real curious if any recommended fixes have surfaced since.

[foggy]

It is recommended to contact Veeam support so they could look up the particular reason of the failure in the job logs.Thanks.