Backup Copy Job - Very slow

Availability for the Always-On Enterprise

Backup Copy Job - Very slow

Veeam Logoby brichardson1991_obk » Wed Apr 27, 2016 8:04 am

Hi there,

Is there any reason our backup copy job is running incredibly slow.

It is processing 10 VM's from our incremental backup job which averages 50-80GB of change a day.
Our pipe to the VPN storage device is 30 meg upload. They accept 30 as well so it's not exactly a slow pipe.

Our processing rate seems to be 5-7 MB/s and the job takes somewhere around 9+ hours.
brichardson1991_obk
Expert
 
Posts: 112
Liked: 8 times
Joined: Thu Aug 20, 2015 2:20 pm
Full Name: Ben Richardson

Re: Backup Copy Job - Very slow

Veeam Logoby chrisdearden » Wed Apr 27, 2016 8:28 am

is that pipe 30 Megabit ?
chrisdearden
Expert
 
Posts: 1529
Liked: 225 times
Joined: Wed Jul 21, 2010 9:47 am
Full Name: Chris Dearden

Re: Backup Copy Job - Very slow

Veeam Logoby v.Eremin » Wed Apr 27, 2016 8:57 am

Also, I'm wondering what component is identified as a major bottleneck. Full job bottleneck statistics would be helpful. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Backup Copy Job - Very slow

Veeam Logoby brichardson1991_obk » Wed Apr 27, 2016 9:38 am

Hi Chris,

Yes it is a 30 Megabit dedicated line


Vladimir,

Job stats as follows:
27/04/2016 01:30:04 :: Load: Source 15% > Proxy 14% > Network 11% > Target 77%
23/04/2016 01:30:07 :: Load: Source 19% > Proxy 17% > Network 16% > Target 70%
22/04/2016 01:30:09 :: Load: Source 18% > Proxy 12% > Network 12% > Target 78%
20/04/2016 01:30:05 :: Load: Source 17% > Proxy 16% > Network 13% > Target 75%
brichardson1991_obk
Expert
 
Posts: 112
Liked: 8 times
Joined: Thu Aug 20, 2015 2:20 pm
Full Name: Ben Richardson

Re: Backup Copy Job - Very slow

Veeam Logoby foggy » Wed Apr 27, 2016 10:50 am

According to the stats, your target storage is the bottleneck, so I'd pay attention to it. What kind of storage it is and how it is added to Veeam B&R?

That said, considering 30 Mbit/s is 3.75 MB/s, your job performance doesn't look slow.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Backup Copy Job - Very slow

Veeam Logoby brichardson1991_obk » Wed Apr 27, 2016 10:58 am

Hi Alexander,

It is a Synology RS815+ with shared folders added to Veeam B&R as a CIFS share.

It just seems slow, but what isn't happening is the server keeps loosing connection to the target. However the target is not going down and other processes using this VPN tunnel are not having any connection issues.
brichardson1991_obk
Expert
 
Posts: 112
Liked: 8 times
Joined: Thu Aug 20, 2015 2:20 pm
Full Name: Ben Richardson

Re: Backup Copy Job - Very slow

Veeam Logoby foggy » Wed Apr 27, 2016 11:22 am 1 person likes this post

In case of CIFS repository over a slow link it is always recommended to have a gateway server located close to the storage itself for more optimal data transfer.
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Backup Copy Job - Very slow

Veeam Logoby brichardson1991_obk » Thu Apr 28, 2016 7:38 am

Hi Alexander,

We will look into what options we have to deploy a gateway server.

I've had a tinker with the job settings to ensure it is not overlapping with our second lot of offsite copying and this was the result from last night:

28/04/2016 08:27:37 :: Load: Source 20% > Proxy 16% > Network 15% > Target 71%
Duration 6:57:13 - Processing rate: 7MB/s
Processed: 130.1GB
Read: 130.1GB
Transferred: 61.5GB (2.1x)
brichardson1991_obk
Expert
 
Posts: 112
Liked: 8 times
Joined: Thu Aug 20, 2015 2:20 pm
Full Name: Ben Richardson

Re: Backup Copy Job - Very slow

Veeam Logoby brichardson1991_obk » Thu Apr 28, 2016 7:39 am

Would it make a difference if instead of CIF shares it was mounted as an ISCSI target?
brichardson1991_obk
Expert
 
Posts: 112
Liked: 8 times
Joined: Thu Aug 20, 2015 2:20 pm
Full Name: Ben Richardson

Re: Backup Copy Job - Very slow

Veeam Logoby v.Eremin » Thu Apr 28, 2016 8:31 am

If it were local deployment, I would say yes. But ISCSI over internet is unlikely to work stably.

Or you're willing to connect the said device to some Windows-based server located in remote site? This might help a bit, however, available bandwidth is probably going to be a limiting factor anyway.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Backup Copy Job - Very slow

Veeam Logoby brichardson1991_obk » Thu Apr 28, 2016 8:42 am

Thanks for the advice Vladimir, i will stick with CIFS and have to see if/how/when at all possible we can get a gateway server on the storage side of the VPN tunnel.
brichardson1991_obk
Expert
 
Posts: 112
Liked: 8 times
Joined: Thu Aug 20, 2015 2:20 pm
Full Name: Ben Richardson

Re: Backup Copy Job - Very slow

Veeam Logoby v.Eremin » Thu Apr 28, 2016 8:49 am

In this case there won't be any gateway server. You will add the device to the remote Windows-based machine via ISCSI first. Then, you'll add this machine to the backup console as a repository. That's it. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Backup Copy Job - Very slow

Veeam Logoby brichardson1991_obk » Thu Apr 28, 2016 8:50 am

Apologies i meant to say stick with CIFS not ISCSI
brichardson1991_obk
Expert
 
Posts: 112
Liked: 8 times
Joined: Thu Aug 20, 2015 2:20 pm
Full Name: Ben Richardson

Re: Backup Copy Job - Very slow

Veeam Logoby v.Eremin » Thu Apr 28, 2016 9:20 am

Got it. As Foggy's said, the usage of gateway server is preferable in such deployments.

Once you have it, feel free to share the results you get.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Backup Copy Job - Very slow

Veeam Logoby robin.frick » Thu Sep 15, 2016 8:36 am

Hello

Case ID: 01905617

We set up Veeam for a client. He has only one server and not much data. He has 2 Synology NAS one in LAN one remote.

When I first set up everything, the first backup took a while (10 hours), I remember the copy job being very fast, a fraction of the backup jobs time. First we setup the remote NAS in the local network, because their internet speed is rather slow and it would have taken more than 24 hours to finish. After a week, we moved the NAS to a remote location and everything worked fine.

The remote location was unmanaged for 2 weeks and they had a blackout, the NAS didn't start, more than a week later they started it again. And the 24 hour window for the copy job wasn't enough.
So we put it back to LAN and started the copy job.

Here is where their problem started.

It starts fast 80-100MB/s processing, but after 10 minute the speed go down to 5MB/s... Throughput is the weirdest. 2-3 1-5MB/s spikes every 5 minutes.

Bottleneck it says 80% Target / 60% Proxy.
The NAS says 3-10% CPU load and ~80% RAM


My question here is... why is the Copy Job slower than the Backup job? The NAS are exactly the same, backup job uses CIFS and the Copy Job uses the Linux Agent.
robin.frick
Novice
 
Posts: 4
Liked: 5 times
Joined: Mon Apr 11, 2016 3:44 pm
Location: Liechtenstein
Full Name: Robin Frick

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Exabot [Bot], Google [Bot], Yahoo [Bot] and 18 guests