Comprehensive data protection for all workloads
Post Reply
EDV4all38
Influencer
Posts: 11
Liked: never
Joined: Dec 04, 2017 12:08 pm
Full Name: Dirk Nielsen
Contact:

Offsite Backup without WAN Accelerator

Post by EDV4all38 »

Hi all,

we are using the "normal" Veeam Backup and Replication", so WAN Accelerators are off limits to us.
However we still would like to copy some Backups offsite from a small server setup of only 5 servers.

They are backed up in following groups
Group 1: Domaincontroller1 (around 11GB)
Group 2: Domaincontroller2 + 2 more servers (around 35GB)
Group 3: Hyper-V hypervisor via Veeam Agent (around 12GB)

The source site has around 40/45 Mbit/s upstream.
The target "site" has no servers at all and 30MBit/s downstream but is only an idependent internet connection, firewall and small NAS (QNAP).
That means no backup-gateway server that could host data mover services as well.

Just pointing the backup copy jobs at the remote NAS did not work out well, as you can think.
Even with setting up the normal backup tasks for WAN-targets and increasing compression it only gets 5% done of one 11GB in an hour.

For now I am stuck with a sync between the local and remote NAS but I think this will get serious performance issues in future.
So I want to use veeam itself to get the job done and avoid sync-problems with forever incremental (copy jobs) and such.

Is there a best practice how to achieve a proper speed even without WAN accelerators and remote datamover service?
EDV4all38
Influencer
Posts: 11
Liked: never
Joined: Dec 04, 2017 12:08 pm
Full Name: Dirk Nielsen
Contact:

Re: Offsite Backup without WAN Accelerator

Post by EDV4all38 »

After looking into rtrr and rsync it seems that the performance issue i fered for is real.
So it seems I am on the wrong track here and syncing does not work well with veeam since veeam change the filenames for fullbackups every merge in forward incremental.

Something I need to add: Both sites have a site2site VPN, forget to mention that above.
csydas
Expert
Posts: 193
Liked: 47 times
Joined: Jan 16, 2018 5:14 pm
Full Name: Harvey Carel
Contact:

Re: Offsite Backup without WAN Accelerator

Post by csydas »

Hi Dirk,

Is this still the initial Copy you're trying to run? If so, you might want to plan to seed the Backup Copy using the links here: https://www.veeam.com/kb1856

It's some planning (you'll have to physically move the copy over to the remote site, but hey, don't underestimate the bandwidth of a sedan with a USB Hard Drive speeding down the freeway ;) to put a spin on a classic), but it'll overcome that big first push.

Keep in mind though, that if the data movers are on Veeam's side, any of the merging operations are going to be bottlenecked with the WAN connection.

Is there any chance you could spin up a Linux box on the remote site and just mount the NAS there as an NFS mount or SMB mount, and then add the Linux box as a repository to Veeam? That would eliminate the back and forth over the VPN for your merges.
EDV4all38
Influencer
Posts: 11
Liked: never
Joined: Dec 04, 2017 12:08 pm
Full Name: Dirk Nielsen
Contact:

Re: Offsite Backup without WAN Accelerator

Post by EDV4all38 »

Hi Harvey,

thanks for your reply.
For now I think a machine at the destination is not possible because of multiple reasons (lacking spare hardware for example).
But the NAS itself might be possible, at least it is some kind of Linux.
I did not work with Veeam on Linux machines for now, do I need anything special on the machine to act as a repository "server"?
As long as it is small and needs low ressources that might be a chance.

But no, it is not only the initial copy :/
Example above indeed is the initial copy but inrementals are slow as ... as well.
Currently I am shuffling a little with the schedules to get some incrementals run at workhours to get some more information.

For seeding the backup: I don't get what the use of the supporting backup-job is in this case.
The documentation sounds like that is only usefull for reverse incremental backups. Guess that does not apply here.
Mapping is no option for now, since there is nothing to map a share for now (see above).
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Offsite Backup without WAN Accelerator

Post by foggy »

Unless your NAS is capable of running Veeam data mover, you definitely need some box in the remote location to serve as a repository.

Btw, what are the bottleneck stats that your jobs currently report?
EDV4all38
Influencer
Posts: 11
Liked: never
Joined: Dec 04, 2017 12:08 pm
Full Name: Dirk Nielsen
Contact:

Re: Offsite Backup without WAN Accelerator

Post by EDV4all38 »

For now it seems other peope were not successfull setting up Veeam data mover services on a QNAP NAS.
At least that is what the search in this forum reveals.
Guess that is a dead end then.

Can't say anything about the bottleneck because I have cleaned up all the different tasks, discontinued backup chains, ... to start anew.
Currently I have set up a second copy job for a vm for testing purposes but that does not seem to work like expected.
The task has the same settings as the original copy task but stays idle (no new restorepoits).
However there are new restorepoints to copy.

I will keep it running since the logic behind copy times and schedule somehow differs from my expectations.
Maybe the following cycle will copy something to further trace the problem.

On top of all I was assigned to another project which puts everything else on hold for now.
I will report back, but currently I do not know when that will happen.
Thanks a lot for help thus far!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Offsite Backup without WAN Accelerator

Post by foggy »

EDV4all38 wrote:The task has the same settings as the original copy task but stays idle (no new restorepoits).
However there are new restorepoints to copy.
Unless the restore points were created outside of the last copy interval, they should be copied.
xudaiqing
Influencer
Posts: 21
Liked: 4 times
Joined: Apr 14, 2017 5:25 pm
Full Name: xudaiqing
Contact:

Re: Offsite Backup without WAN Accelerator

Post by xudaiqing »

We do offisite backup without WAN accelerator, and it runs pretty well.
But we run it through a repository server on the destination site, and i don't think NFS/SMB over internet will give you good result.

Why not setup a linux repository server in old PC / intel NUC on the destination site?
EDV4all38
Influencer
Posts: 11
Liked: never
Joined: Dec 04, 2017 12:08 pm
Full Name: Dirk Nielsen
Contact:

Re: Offsite Backup without WAN Accelerator

Post by EDV4all38 »

@ xudaiqing: Because I have no chance to do so at the moment (see above).

Before I got the new project I set up a test job to copy a single server over wan.
Today I got time to check how things are going and it seems that after a few intervals copying the incremental backups works fine.

However, the merge of a new full backup fails:

Code: Select all

Failed to merge full backup file Error: Unerwarteter Netzwerkfehler Failed to read data from the file [\\fw-rathaus\GFS Copy\test\testD2018-06-04T013000.vbk]. Failed to backup text locally. Backup: [VBK: 'veeamfs:-1:fc6df34f-897d-4c3c-a24f-416275b64e6c (d0f20b38-902f-4fce-a309-d14bf36714d2)\summary.xml@\\fw-rathaus\GFS Copy\test\testD2018-06-04T013000.vbk']. Agent failed to process method {DataTransfer.BackupText}. 
Failed to create restore point 05.06.2018 01:30:00
Failed to generate points Error: Unerwarteter Netzwerkfehler Failed to read data from the file [\\fw-rathaus\GFS Copy\test\testD2018-06-04T013000.vbk]. Failed to backup text locally. Backup: [VBK: 'veeamfs:-1:fc6df34f-897d-4c3c-a24f-416275b64e6c (d0f20b38-902f-4fce-a309-d14bf36714d2)\summary.xml@\\fw-rathaus\GFS Copy\test\testD2018-06-04T013000.vbk']. Agent failed to process method {DataTransfer.BackupText}.
And after that everything looks like:

Code: Select all

Failed to merge full backup file Error: Unerwarteter Netzwerkfehler Failed to read data from the file [\\nas01\GFS Copy\test\testD2018-06-05T013000.vbk]. Agent failed to process method {Stg.OpenRead}. 
Failed to generate points Error: Unerwarteter Netzwerkfehler Failed to read data from the file [\\nas01\GFS Copy\test\testD2018-06-05T013000.vbk]. Agent failed to process method {Stg.OpenRead}. 
Failed to merge full backup file Error: Unerwarteter Netzwerkfehler Failed to read data from the file [\\nas01\GFS Copy\test\testD2018-06-06T013000.vib]. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{2bbc3a2f-3a18-4ab2-bbfc-4d7978c7eaf6}]. CHMOD mask: [208]. Agent failed to process method {DataTransfer.RestoreText}. 
Failed to generate points Error: Unerwarteter Netzwerkfehler Failed to read data from the file [\\nas01\GFS Copy\test\testD2018-06-06T013000.vib]. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{2bbc3a2f-3a18-4ab2-bbfc-4d7978c7eaf6}]. CHMOD mask: [208]. Agent failed to process method {DataTransfer.RestoreText}. 
Failed to merge full backup file Error: Unerwarteter Netzwerkfehler Failed to read data from the file [\\nas01\GFS Copy\test\testD2018-06-06T013000.vib]. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{701de485-6955-4c26-8046-399542cd9fdd}]. CHMOD mask: [190]. Agent failed to process method {DataTransfer.RestoreText}. 
Failed to generate points Error: Unerwarteter Netzwerkfehler Failed to read data from the file [\\nas01\GFS Copy\test\testD2018-06-06T013000.vib]. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{701de485-6955-4c26-8046-399542cd9fdd}]. CHMOD mask: [190]. Agent failed to process method {DataTransfer.RestoreText}. 
Failed to merge full backup file Error: Unerwarteter Netzwerkfehler Failed to read data from the file [\\nas01\GFS Copy\test\testD2018-06-06T013000.vib]. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{3348d5b1-c4d5-48ca-a766-0ebfcf621228}]. CHMOD mask: [243]. Agent failed to process method {DataTransfer.RestoreText}. 
Failed to generate points Error: Unerwarteter Netzwerkfehler Failed to read data from the file [\\nas01\GFS Copy\test\testD2018-06-06T013000.vib]. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{3348d5b1-c4d5-48ca-a766-0ebfcf621228}]. CHMOD mask: [243]. Agent failed to process method {DataTransfer.RestoreText}. 
Failed to merge full backup file Error: Unerwarteter Netzwerkfehler Failed to read data from the file [\\nas01\GFS Copy\test\testD2018-06-05T013000.vbk]. Agent failed to process method {Stg.OpenRead}. 
Failed to generate points Error: Unerwarteter Netzwerkfehler Failed to read data from the file [\\nas01\GFS Copy\test\testD2018-06-05T013000.vbk]. Agent failed to process method {Stg.OpenRead}. 
Failed to merge full backup file Error: Unerwarteter Netzwerkfehler Failed to read data from the file [\\nas01\GFS Copy\test\testD2018-06-06T013000.vib]. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{c1a26476-0906-4010-b0e7-a7a7ad05d6ce}]. CHMOD mask: [164]. Agent failed to process method {DataTransfer.RestoreText}. 
Failed to generate points Error: Unerwarteter Netzwerkfehler Failed to read data from the file [\\nas01\GFS Copy\test\testD2018-06-06T013000.vib]. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{c1a26476-0906-4010-b0e7-a7a7ad05d6ce}]. CHMOD mask: [164]. Agent failed to process method {DataTransfer.RestoreText}. 
Unerwarteter Netzwerkfehler = unknown network error

Strange enough: DNS is working fine, the data exists on the target (nas01), we have no network outages and the new incremental copy works just fine, only the merge fails.

Searching logfiles, web and forums did not provide any usefull information about cause or solution for now.
Still searching when I have some spare time (not much atm) but for now I have no clue why this happens or where it comes from.

Edit:
The weekly backup was created without a problem.
Seems just to be the merge of daily incremental to the new full backup that runs into problems.
EDV4all38
Influencer
Posts: 11
Liked: never
Joined: Dec 04, 2017 12:08 pm
Full Name: Dirk Nielsen
Contact:

Re: Offsite Backup without WAN Accelerator

Post by EDV4all38 »

meh... could not edit anymore.

\\fw-rathaus = nas01 for comprehension, it's the same system. I was just too slow to edit :/
EDV4all38
Influencer
Posts: 11
Liked: never
Joined: Dec 04, 2017 12:08 pm
Full Name: Dirk Nielsen
Contact:

Re: Offsite Backup without WAN Accelerator

Post by EDV4all38 »

Got answer from Veeam Support.

To keep things short: It is not possible without a gateway/agent at the destination site.
Seems like there is no way around this setup.
We will do some purchses and try again.

Thanks for your help so far.
EDV4all38
Influencer
Posts: 11
Liked: never
Joined: Dec 04, 2017 12:08 pm
Full Name: Dirk Nielsen
Contact:

Re: Offsite Backup without WAN Accelerator

Post by EDV4all38 »

A small update.

We got a new switch and an desktop machine to run a test scenario with a gateway at the remote site (including registry settings of https://www.veeam.com/kb1735).
For now it seems to be working but I stumbled over the "bottleneck" in the backup job.

Basically we have a copy job to a local NAS and a remote NAS with one vm.
It's the same vm and only this one in both jobs to test things out.

Site 1 (source) with backup-server, local NAS and vm to backup
Site 2 (target, WAN) with gateway (desktop pc for now) and remote NAS

Copy Job to local NAS (LAN)
Speed: 180mb/s
Bottleneck: Source 83% / Proxy 6% / Network 15% / Target 11%

Copy Job to remote NAS (WAN)
Speed: 8mb/s
Bottleneck: Source 60% / Proxy 1% / Network 33% / Target 0%

I would understand if the bottleneck would show the target or network an issue because we copy over WAN.
While the speed seems to be ok regarding the internet connection I still wonder how the source could be the slowest part in all of this.
After some search it seems that the term "bottleneck" might be misleading.

That leads to the following question:
What does Veeam use for the source stat (checked CPU, RAM, disc I/O)?

And something else:
The transferrate seems to be slightly higher than the WAN connection can supply.
I believe this has somtheing to do with compression settings and result in "wrong" readings?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Offsite Backup without WAN Accelerator

Post by foggy »

EDV4all38 wrote:What does Veeam use for the source stat (checked CPU, RAM, disc I/O)?
It's source storage disk I/O (and keep in mind the random reads backup copy has to do on source).
Post Reply

Who is online

Users browsing this forum: No registered users and 261 guests