Offsite Backup Copy over slow WAN fails a lot

[kaysond]

I have Veeam B&R set up on a Windows 2k19 VM. It backs up 4 VMs on ESXi, and one Windows client using Veeam Agent for Windows.

I'd like to replicate these backups to my offsite storage over WAN which is 20Mbps upload. I've set up sshfs to mount the remote storage on my local NAS, and veeam connects to it using SMB.

I have one Backup Copy job set up for the VMs, and another for the Windows host. It fails a lot. More often than it succeeds. I frequently get errors like:

Code: Select all

{Transform.CompileFIB}: An unexpected network error occurred. Failed to write data to the file [\\10.7.7.4\veeam-offsite\Desktop Offsite Copy\Desktop Offsite CopyD2021-10-18T213207_CB19.temp].
Failed to generate points Error: Agent: Failed to process method {Transform.CompileFIB}: An unexpected network error occurred. Failed to write data to the file [\\10.7.7.4\veeam-offsite\Desktop Offsite Copy\Desktop Offsite CopyD2021-10-18T213207_CB19.temp]

I've tested both the sshfs link and the smb share by copying files, and they go through just fine, but obviously there's a problem.

What are best practices here? Any suggestions for what else I can check?

I have full control of the remote storage so I can deploy whatever would be most reliable. Thanks!

[Mildur]

Best practice would be to setup a gateway server in the remote location, if you use an smb backup repo. Or disabling synthentic full and doing only active full, if its a smb backup target.

You won‘t get a stable connection with a 20mbit link over the internet for accessing smb shares without a gateway at the remote location. A Gateway Server will help you to reduce the data transfer which is necessary.
For example, in a synthetic full copy job, veeam would need to copy all backups back to the backup server, build a full, and transfer it again over the 20mbit link to the remote location.

With a gateway server, that all happens at the remote location. Only increments are send over to the remote location.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[PetrM]

Hello Aram,

Also, please open a support case and provide us with ID. The error itself is quite generic and can have many different causes. If Fabian's suggestion does not help, then the most appropriate action plan would be to let our engineers to examine debug logs and network traffic dump.

Thanks!

[aj_potc]

It may be difficult to pinpoint the issue, especially with the mix of a NAS along with two different protocols (SMB and SSHFS) in your stack.

Have you done any basic network troubleshooting (like a ping test over time) to see if there's any packet loss between the source and destination? That would be my first suspicion based on your description of the intermittent failures. SSH connections are pretty good at withstanding some packet loss, but if it's sustained, then they'll drop.

If you can mount the remote storage via SSHFS, does that mean that it's a fully functional Linux-based system? If so, and if you have root access to that system, I'd suggest you try to set it up directly as a Veeam repository. This will remove your NAS and the SSHFS and SMB protocols from the equation completely. If that still fails, then it's very likely that the physical connection is having issues.

[kaysond]

Best practice would be to setup a gateway server in the remote location, if you use an smb backup repo. Or disabling synthentic full and doing only active full, if its a smb backup target.

The docs weren't entirely clear to me - can the gateway server itself run on linux? Or does the gateway have to be windows but the storage can be linux?

[Mildur]

The gateway server needs to be a windows system.
The storage can be a nfs or a cifs share on a linux system.

https://helpcenter.veeam.com/docs/backu ... ay-servers

[kaysond]

It may be difficult to pinpoint the issue, especially with the mix of a NAS along with two different protocols (SMB and SSHFS) in your stack.

Have you done any basic network troubleshooting (like a ping test over time) to see if there's any packet loss between the source and destination? That would be my first suspicion based on your description of the intermittent failures. SSH connections are pretty good at withstanding some packet loss, but if it's sustained, then they'll drop.

If you can mount the remote storage via SSHFS, does that mean that it's a fully functional Linux-based system? If so, and if you have root access to that system, I'd suggest you try to set it up directly as a Veeam repository. This will remove your NAS and the SSHFS and SMB protocols from the equation completely. If that still fails, then it's very likely that the physical connection is having issues.

Can you elaborate? Yes, I do have root access and its a fully functional linux OS. How would I set it up directly as a repository? Do you mean exposing the storage over WAN via nfs/smb?

[Mildur]

If it‘s a fully functional linux OS, why not adding the storage as a Linux backup repository? That way, no Gateway would be needed. What is the operating system of this „NAS“?

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[aj_potc]

Can you elaborate? Yes, I do have root access and its a fully functional linux OS. How would I set it up directly as a repository? Do you mean exposing the storage over WAN via nfs/smb?

No, you don't need to expose it that way, but you would need to open some ports so that Veeam can connect to it natively over the WAN. It uses SSH and its own encrypted transfer method.

If you have more questions after reading about it, please come back and ask. I've set up Linux based repositories many times.

Here's the link:
https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[aj_potc]

If it‘s a fully functional linux OS, why not adding the storage as a Linux backup repository? That way, no Gateway would be needed. What is the operating system of this „NAS“?

Right, that's what I suggested. I don't think there is a NAS on the destination side. I believe that was on the local (source) side, and it was being used just to mount the remote storage. Kind of a roundabout way to do it, which may be contributing to the instability of the transfer.

[kaysond]

It's Rasbian, so it should work. Thanks for the suggestion! I will give this a shot and report back. Looks like first step is to upgrade to 11a so I can use ed25519 keys

[Mildur]

I don‘t think, that a raspian can be used directly as a linux backup repo. It is not a supported operating system.

Supported linux distributions for Backup Repos:
https://helpcenter.veeam.com/docs/backu ... ory-server

[aj_potc]

Veeam's agent certainly won't run on ARM-based systems. I see a good number of requests for that feature, but it doesn't sound like Veeam is interested. I can't find any reports about whether a repository would work, but I'm also guessing no.

I do know that Veeam's requirements for agents are much more restrictive than for repositories. As an example, I've run repositories on Linux releases many months before Veeam officially listed those as supported. But that was just a small difference in versions -- not an entirely different system architecture.

Please report back and let us know how it's going.

[kaysond]

Unsurprisingly, it appears you can't set up a repository on an ARM system either. There's some kind of binary that gets deployed and presumably that was only compiled for x86.

Any other suggestions?

[Mildur]

I’ll stay at my initial recommendation.
Put a Gateway Server to the remote Location. Stability and performance should be better, if you have a windows machine managing the smb backup repo im the same site.
Without the gateway server, health checks, transform proccesses and synthetic full will need transfer all data back to the production site for processing the data. That must be a pain with a 20mbit wan connection.

If that is not enough, use a WAN Accelerator.

[kaysond]

That's probably what I'm going to end up having to do, thanks.

Both the gateway server and the wan accelerator will require an x86 windows host, right?

[Mildur]

Yes. You can also use Windows 10 as a gateway server for a smb backup repo. You can do your tests with a simple windows 10 computer before investing in something bigger.

And for the wan accelerator, you need a Enterprise Plus license.
But try it out without the wan accelerator first.

[kaysond]

Will try and report back. Thanks!

[aj_potc]

Do you have an x86-based Linux system available on the remote side? If so, you could use it as a repository. You'd need to mount the storage on your Raspbian system to it, which should be quite stable over a LAN.

This would have a much smaller footprint than a dedicated Windows installation, and might be quicker to test out.