Backup Copy Job - Very slow

[brichardson1991_obk]

Hi there,

Is there any reason our backup copy job is running incredibly slow.

It is processing 10 VM's from our incremental backup job which averages 50-80GB of change a day.
Our pipe to the VPN storage device is 30 meg upload. They accept 30 as well so it's not exactly a slow pipe.

Our processing rate seems to be 5-7 MB/s and the job takes somewhere around 9+ hours.

[chrisdearden]

is that pipe 30 Megabit ?

[veremin]

Also, I'm wondering what component is identified as a major bottleneck. Full job bottleneck statistics would be helpful. Thanks.

[brichardson1991_obk]

Hi Chris,

Yes it is a 30 Megabit dedicated line


Vladimir,

Job stats as follows:
27/04/2016 01:30:04 :: Load: Source 15% > Proxy 14% > Network 11% > Target 77%
23/04/2016 01:30:07 :: Load: Source 19% > Proxy 17% > Network 16% > Target 70%
22/04/2016 01:30:09 :: Load: Source 18% > Proxy 12% > Network 12% > Target 78%
20/04/2016 01:30:05 :: Load: Source 17% > Proxy 16% > Network 13% > Target 75%

[foggy]

According to the stats, your target storage is the bottleneck, so I'd pay attention to it. What kind of storage it is and how it is added to Veeam B&R?

That said, considering 30 Mbit/s is 3.75 MB/s, your job performance doesn't look slow.

[brichardson1991_obk]

Hi Alexander,

It is a Synology RS815+ with shared folders added to Veeam B&R as a CIFS share.

It just seems slow, but what isn't happening is the server keeps loosing connection to the target. However the target is not going down and other processes using this VPN tunnel are not having any connection issues.

[foggy]

In case of CIFS repository over a slow link it is always recommended to have a gateway server located close to the storage itself for more optimal data transfer.

[brichardson1991_obk]

Hi Alexander,

We will look into what options we have to deploy a gateway server.

I've had a tinker with the job settings to ensure it is not overlapping with our second lot of offsite copying and this was the result from last night:

28/04/2016 08:27:37 :: Load: Source 20% > Proxy 16% > Network 15% > Target 71%
Duration 6:57:13 - Processing rate: 7MB/s
Processed: 130.1GB
Read: 130.1GB
Transferred: 61.5GB (2.1x)

[brichardson1991_obk]

Would it make a difference if instead of CIF shares it was mounted as an ISCSI target?

[veremin]

If it were local deployment, I would say yes. But ISCSI over internet is unlikely to work stably.

Or you're willing to connect the said device to some Windows-based server located in remote site? This might help a bit, however, available bandwidth is probably going to be a limiting factor anyway.

Thanks.

[brichardson1991_obk]

Thanks for the advice Vladimir, i will stick with CIFS and have to see if/how/when at all possible we can get a gateway server on the storage side of the VPN tunnel.

[veremin]

In this case there won't be any gateway server. You will add the device to the remote Windows-based machine via ISCSI first. Then, you'll add this machine to the backup console as a repository. That's it. Thanks.

[brichardson1991_obk]

Apologies i meant to say stick with CIFS not ISCSI

[veremin]

Got it. As Foggy's said, the usage of gateway server is preferable in such deployments.

Once you have it, feel free to share the results you get.

Thanks.

[robin.frick]

Hello

Case ID: 01905617

We set up Veeam for a client. He has only one server and not much data. He has 2 Synology NAS one in LAN one remote.

When I first set up everything, the first backup took a while (10 hours), I remember the copy job being very fast, a fraction of the backup jobs time. First we setup the remote NAS in the local network, because their internet speed is rather slow and it would have taken more than 24 hours to finish. After a week, we moved the NAS to a remote location and everything worked fine.

The remote location was unmanaged for 2 weeks and they had a blackout, the NAS didn't start, more than a week later they started it again. And the 24 hour window for the copy job wasn't enough.
So we put it back to LAN and started the copy job.

Here is where their problem started.

It starts fast 80-100MB/s processing, but after 10 minute the speed go down to 5MB/s... Throughput is the weirdest. 2-3 1-5MB/s spikes every 5 minutes.

Bottleneck it says 80% Target / 60% Proxy.
The NAS says 3-10% CPU load and ~80% RAM


My question here is... why is the Copy Job slower than the Backup job? The NAS are exactly the same, backup job uses CIFS and the Copy Job uses the Linux Agent.

[robin.frick]

The problem was the mashine, veeam was running on. The update service went completely bonkers, once I fixed it, the copy job worked nice.

[sbaltic]

Same problem here ... 6-15MB/s on SAS storage. So we have backup on SAS (HP storage) and some other storage on the other SAS controller (same backup server). And we get just 5-16MB/s. First 20% was fast, about 230-300MB/s, then catastrophic. No jobs in backgrounds.

What can we do?

THX

[foggy]

Hi Sebastijan, according to Robin's case notes, he had issues with Windows Update Service. If it looks fine in your case, I recommend you contacting support for a closer look.