Replication Job slows to a snails pace!

[kevin_cambs_uk]

We have a VMware 5 ESX guest which is 455 Gb in size. In the world of IT probably not that large. It runs off a stand alone Dell R510 server on SATA discs.

The local backup on a Gb Ethernet network works with no problems, normally a full backup will take around 1 hour and the daily incremental, around 30 minutes.

At the weekend, we replicate the guest to our DR site, using a 100 Mb link. The first run which creates the first replica, which is obviously the whole guest takes about 12 hours, which is okay.

When we run the replication again, the following weekend, the jobs starts off okay and looks like it will take a few hours, but as time goes on the transfer rate eventually goes down to about 3kbps ! At which point, you cannot cancel the job and I have to reboot the Veeam Server in order for the job to be terminated. I must mention that when we run the job, we do shut the guest down for a complete consistent backup. The amount of changes is probably around 4 Gb per week.

I have other Veeam replication jobs that work, but not with a guest of this size.

So any ideas on what is going wrong or is there anything I can do?

Many thanks

Kevin

[foggy]

Kevin, what are the bottleneck stats for this incremental job run? Do you have a proxy deployed at the remote location and, if yes, what transport mode does it use to populate target datastore?

[kevin_cambs_uk]

Is this what your after mate?

15/06/2014 11:01:34 :: Load: Source 3% > Proxy 52% > Network 99% > Target 99%

The transfer rate was 7 Mb, which I would expect for a 100 Mbps line as a few other things use it at the same time.

The above is for the first replication run, when it creates the complete first replica.

I am on call this weekend so will run it again, but instead of starting it again from scratch I will run the replication job, with the replica already there, so it should then work thru the changes and not transfer the entire guest again, if you see what I mean.

thanks for replying, its much appreciated!

[foggy]

I was wondering what was the bottleneck for the incremental run, however it is clear that, along with the network, target storage is also overloaded, even during the full run. What kind of storage do you have in this location?

[Vitaliy S.]

I'm curious on why you have both network and target reported as bottlenecks. What version of Veeam B&R are you running? BTW, do you have a proxy server deployed on the target location?

[kevin_cambs_uk]

I was wondering what was the bottleneck for the incremental run, however it is clear that, along with the network, target storage is also overloaded, even during the full run. What kind of storage do you have in this location?

Its exactly the same machine, same R510 dell server running VMware 5. We bought 2 servers of the same spec just one at each site. The link between the 2 is only 100 Mbps, so I would have thought the discs would be able to keep up etc.

[kevin_cambs_uk]

I'm curious on why you have both network and target reported as bottlenecks. What version of Veeam B&R are you running? BTW, do you have a proxy server deployed on the target location?

6.0.0.153 64 Bit.

I don't use any proxy the remote site is linked via routers, which along with the network speed will never exceed 100 Mbps, so a transfer rate of & Mb/sec is quite good really.

The problem is the following week when the replication runs again, it should just work out the changes between the 2 which should be no more than say 10 Gb, and then copy those bits across but once it starts it starts off okay but it just gradually grinds to halt, but this week I will get better stats and post them if that's okay.

Cheers mate

Kev

[Vitaliy S.]

I don't use any proxy the remote site is linked via routers, which along with the network speed will never exceed 100 Mbps, so a transfer rate of & Mb/sec is quite good really.

Yes, but I was referring to Veeam proxy on the target site, that can optimize data transfer (assuming that network component is reported as a bottleneck). Let us know the bottleneck stats for the incremental run. Also is there any reason why you haven't upgraded to our latest version?

[kevin_cambs_uk]

I will and no!
I must get around to that!

[foggy]

Btw, there were some improvements to how bottleneck analysis was performed introduced in patch 3 for Veeam B&R v6, so it is worth to upgrade to get more realistic stats:

Improved bottleneck analysis formulas for proxy and network processing stages. For example, heavy backup target load should no longer affect the Network stage percent busy value.

[kevin_cambs_uk]

Thanks for that I will apply that patch as well,

thanks again for your help

Kev

[foggy]

Keep us posted on the corrected stats.

[kevin_cambs_uk]

Keep us posted on the corrected stats.

Will do mate
the job will run Sat eve so I will post up Monday what happened

Thanks again, we could make me and my colleagues very happy if we nail this!!

[Gostev]

6.0.0.153 64 Bit

Just wanted to note that support for version 6.0 ended in October 2012. The oldest supported version is currently 6.5.

I hope that the correct bottlenecks stats will help you to identify the issue, otherwise you would have to start working with our technical support on this issue, and this will require that you upgrade to a supported version first.

[kevin_cambs_uk]

I have applied the patch so we will see how it goes this weekend.
Thanks for this, are the upgrades all part of the annual maintenance?

[foggy]

Yes, if you have an active maintenance, you can upgrade for free.

[kevin_cambs_uk]

I think I have solved why this is happening...
Such a school boy error....
I do not think there is enough space on the destination data store.
After applying the patch it came up with a different error to what happened before, here it just gradually ground to a halt after 12 hours.

So I am expanding the data store this week and I think we will be back in the game!

Thanks again for your help. Will keep it posted when I get it sorted.

[kevin_cambs_uk]

Well I sorted the destination data store and did the full replication from start which took 18 hours, which is what I expected.
I now run the job daily, which takes 30 minutes, as there is only about 2 -3 Gb of data change but of a very critical nature.
so its now running perfectly and we are all very happy as the weekend on call working just got so much easier!

Thanks to all who replied.

Kevin