Comprehensive data protection for all workloads
Post Reply
brichardson1991_obk
Expert
Posts: 114
Liked: 8 times
Joined: Aug 20, 2015 2:20 pm
Full Name: Ben Richardson
Contact:

Backup Copy Job - Very slow

Post by brichardson1991_obk »

Hi there,

Is there any reason our backup copy job is running incredibly slow.

It is processing 10 VM's from our incremental backup job which averages 50-80GB of change a day.
Our pipe to the VPN storage device is 30 meg upload. They accept 30 as well so it's not exactly a slow pipe.

Our processing rate seems to be 5-7 MB/s and the job takes somewhere around 9+ hours.

chrisdearden
Expert
Posts: 1530
Liked: 225 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Backup Copy Job - Very slow

Post by chrisdearden »

is that pipe 30 Megabit ?

veremin
Product Manager
Posts: 17777
Liked: 1628 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Copy Job - Very slow

Post by veremin »

Also, I'm wondering what component is identified as a major bottleneck. Full job bottleneck statistics would be helpful. Thanks.

brichardson1991_obk
Expert
Posts: 114
Liked: 8 times
Joined: Aug 20, 2015 2:20 pm
Full Name: Ben Richardson
Contact:

Re: Backup Copy Job - Very slow

Post by brichardson1991_obk »

Hi Chris,

Yes it is a 30 Megabit dedicated line


Vladimir,

Job stats as follows:
27/04/2016 01:30:04 :: Load: Source 15% > Proxy 14% > Network 11% > Target 77%
23/04/2016 01:30:07 :: Load: Source 19% > Proxy 17% > Network 16% > Target 70%
22/04/2016 01:30:09 :: Load: Source 18% > Proxy 12% > Network 12% > Target 78%
20/04/2016 01:30:05 :: Load: Source 17% > Proxy 16% > Network 13% > Target 75%

foggy
Veeam Software
Posts: 19285
Liked: 1743 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Copy Job - Very slow

Post by foggy »

According to the stats, your target storage is the bottleneck, so I'd pay attention to it. What kind of storage it is and how it is added to Veeam B&R?

That said, considering 30 Mbit/s is 3.75 MB/s, your job performance doesn't look slow.

brichardson1991_obk
Expert
Posts: 114
Liked: 8 times
Joined: Aug 20, 2015 2:20 pm
Full Name: Ben Richardson
Contact:

Re: Backup Copy Job - Very slow

Post by brichardson1991_obk »

Hi Alexander,

It is a Synology RS815+ with shared folders added to Veeam B&R as a CIFS share.

It just seems slow, but what isn't happening is the server keeps loosing connection to the target. However the target is not going down and other processes using this VPN tunnel are not having any connection issues.

foggy
Veeam Software
Posts: 19285
Liked: 1743 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Copy Job - Very slow

Post by foggy » 1 person likes this post

In case of CIFS repository over a slow link it is always recommended to have a gateway server located close to the storage itself for more optimal data transfer.

brichardson1991_obk
Expert
Posts: 114
Liked: 8 times
Joined: Aug 20, 2015 2:20 pm
Full Name: Ben Richardson
Contact:

Re: Backup Copy Job - Very slow

Post by brichardson1991_obk »

Hi Alexander,

We will look into what options we have to deploy a gateway server.

I've had a tinker with the job settings to ensure it is not overlapping with our second lot of offsite copying and this was the result from last night:

28/04/2016 08:27:37 :: Load: Source 20% > Proxy 16% > Network 15% > Target 71%
Duration 6:57:13 - Processing rate: 7MB/s
Processed: 130.1GB
Read: 130.1GB
Transferred: 61.5GB (2.1x)

brichardson1991_obk
Expert
Posts: 114
Liked: 8 times
Joined: Aug 20, 2015 2:20 pm
Full Name: Ben Richardson
Contact:

Re: Backup Copy Job - Very slow

Post by brichardson1991_obk »

Would it make a difference if instead of CIF shares it was mounted as an ISCSI target?

veremin
Product Manager
Posts: 17777
Liked: 1628 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Copy Job - Very slow

Post by veremin »

If it were local deployment, I would say yes. But ISCSI over internet is unlikely to work stably.

Or you're willing to connect the said device to some Windows-based server located in remote site? This might help a bit, however, available bandwidth is probably going to be a limiting factor anyway.

Thanks.

brichardson1991_obk
Expert
Posts: 114
Liked: 8 times
Joined: Aug 20, 2015 2:20 pm
Full Name: Ben Richardson
Contact:

Re: Backup Copy Job - Very slow

Post by brichardson1991_obk »

Thanks for the advice Vladimir, i will stick with CIFS and have to see if/how/when at all possible we can get a gateway server on the storage side of the VPN tunnel.

veremin
Product Manager
Posts: 17777
Liked: 1628 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Copy Job - Very slow

Post by veremin »

In this case there won't be any gateway server. You will add the device to the remote Windows-based machine via ISCSI first. Then, you'll add this machine to the backup console as a repository. That's it. Thanks.

brichardson1991_obk
Expert
Posts: 114
Liked: 8 times
Joined: Aug 20, 2015 2:20 pm
Full Name: Ben Richardson
Contact:

Re: Backup Copy Job - Very slow

Post by brichardson1991_obk »

Apologies i meant to say stick with CIFS not ISCSI

veremin
Product Manager
Posts: 17777
Liked: 1628 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Copy Job - Very slow

Post by veremin »

Got it. As Foggy's said, the usage of gateway server is preferable in such deployments.

Once you have it, feel free to share the results you get.

Thanks.

robin.frick
Novice
Posts: 4
Liked: 5 times
Joined: Apr 11, 2016 3:44 pm
Full Name: Robin Frick
Location: Liechtenstein
Contact:

Re: Backup Copy Job - Very slow

Post by robin.frick »

Hello

Case ID: 01905617

We set up Veeam for a client. He has only one server and not much data. He has 2 Synology NAS one in LAN one remote.

When I first set up everything, the first backup took a while (10 hours), I remember the copy job being very fast, a fraction of the backup jobs time. First we setup the remote NAS in the local network, because their internet speed is rather slow and it would have taken more than 24 hours to finish. After a week, we moved the NAS to a remote location and everything worked fine.

The remote location was unmanaged for 2 weeks and they had a blackout, the NAS didn't start, more than a week later they started it again. And the 24 hour window for the copy job wasn't enough.
So we put it back to LAN and started the copy job.

Here is where their problem started.

It starts fast 80-100MB/s processing, but after 10 minute the speed go down to 5MB/s... Throughput is the weirdest. 2-3 1-5MB/s spikes every 5 minutes.

Bottleneck it says 80% Target / 60% Proxy.
The NAS says 3-10% CPU load and ~80% RAM


My question here is... why is the Copy Job slower than the Backup job? The NAS are exactly the same, backup job uses CIFS and the Copy Job uses the Linux Agent.

robin.frick
Novice
Posts: 4
Liked: 5 times
Joined: Apr 11, 2016 3:44 pm
Full Name: Robin Frick
Location: Liechtenstein
Contact:

Re: Backup Copy Job - Very slow

Post by robin.frick »

The problem was the mashine, veeam was running on. The update service went completely bonkers, once I fixed it, the copy job worked nice.

sbaltic
Enthusiast
Posts: 30
Liked: 1 time
Joined: Nov 18, 2013 6:10 pm
Full Name: Sebastijan
Contact:

Re: Backup Copy Job - Very slow

Post by sbaltic »

Same problem here ... 6-15MB/s on SAS storage. So we have backup on SAS (HP storage) and some other storage on the other SAS controller (same backup server). And we get just 5-16MB/s. First 20% was fast, about 230-300MB/s, then catastrophic. No jobs in backgrounds.

What can we do?

THX

foggy
Veeam Software
Posts: 19285
Liked: 1743 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Copy Job - Very slow

Post by foggy » 1 person likes this post

Hi Sebastijan, according to Robin's case notes, he had issues with Windows Update Service. If it looks fine in your case, I recommend you contacting support for a closer look.

Post Reply

Who is online

Users browsing this forum: Google [Bot], stvoglio and 66 guests