Slow WAN Replication

[virtking]

Yes, I do realize that there are dozens of posts concerning this topic. I was not able to find the answers I was looking for, so here it goes:
- 1 ESX 4.1 @main site
- MSA 2012sa (SAS connection) SAN @main site (where VM resides)
- VBR 4.1 @main site with gigE, and SAS connection direct to the SAN
- 1 ESX 4.1 @DR site
- Replication job with SAN/NBD, CBT on, all default settings for compression, etc.
- 1 VM in the job, 200 GB vmdk
- Initial seeding done to USB, copied to DR site successfully
- 6 Mbps circuits for both sites with site-to-site IPSec VPN

I ran the first ever incremental over the WAN and it completed successfully, but with dismal performance. Even though .vbr at the DR site shows only 4.5 GB of changed blocks, results show:
- Processing rate: 5 MB/s
- Processing time: 12.5 hours

Questions:
1. It looks as if VBR "processes" the entire 200 GB even for incrementals, even though CBT is turned on and is working according to logs? At 5 MB/s, even if the changed delta is only 5 GB, it's going to take 12+ hours if it's going to look at the entire 200 GB of data. Is that by design?
2. What type of processing rate should I expect with this setup?
3. Can the direct-from-SAN work with SAS connected SANs? I only found one post that talks about this but could not determine whether direct-from-SAN is possible via SAS or not.

Even if direct-from-SAN wasn't working, and it fell back to network, I would expect much higher processing rate than 5 MB/s.

I'm helping a customer evaluate VBR and am interested in selling more if I can get this to work! Please help.

Thank you.

[tsightler]

It's full ESX, not ESXi at the replication site? If so, have you added SSH credential for the ESX host so that it can use the host agent?

[virtking]

ESX, not ESXi
SSH creds were used for service console access

[tsightler]

Also, you should, note that a 6Mbps is only .75/MB sec and will actually take almost 3 days to transfer 200GB, assuming the full bandwidth is available and no compression, so you're definitely not transferring the entire 200GB on the replication. You can look at the rollback and determine roughly the delta transfer amount. The actual amount transferred over the WAN will be even more but it's a good starting point. How big is your delta?

[virtking]

I understand the bandwidth implications, but .vbr at the destination is only 4.5 GB. So at the theoretical maximum of 6 Mbps, I should be able to complete the job in a couple of hours at most, not 12. Am I correct in assuming that the .vbr file is where the delta is kept?

[tsightler]

I'm sorry, I missed that part of your first post, guess I should have read it twice. So you're right, 4.5GB should make it in a couple of hours. There is a good bit of additional overhead in Veeam replication protocol however and latency can have a big impact as well. Have you monitored to see how much data is actually transferred? Is there other traffic on the link?

[virtking]

No probs. No, I have not monitored the WAN link to see the actual data transferred. There should not be a whole lot going on when the job is going. I don't know if this is a WAN issue though. It seems like there might be something going on locally even before the bytes are sent over the pipe.

One part of my question that has not been addressed is the issue of direct-from-san backups using SAS-connected SANs. Even if that wasn't working and NBD took over, I still think I should be getting much higher processing rate.

I wonder if virtual appliance mode would make a difference?

[Vitaliy S.]

Michael,

Based on the description provided it should work, but you can look through an HTML report for this job to see if there was a failover or not. Another way to find out about the failover is to check out the corresponding job log.

I would definitely try to use Virtual Appliance mode to see if it makes a big difference for you or not. Please note that you should grant VM with Veeam Backup installed with 4 vCPU to get best performance rates.

Thanks!

[virtking]

Where are the job logs kept?

[st0623]

Are you doing any compression in Veeam? What about offloading compression to a WAN accelerator and filling the pipe? HyperIP?

[Vitaliy S.]

You may find all the logs if you navigate to Help|Support Information. The log is named after the job name.

[Vitaliy S.]

Steve is right, you may want to try using WAN acceleration tool, such as HyperIP, that should give much better performance rates.

[joemagjr]

[merged into existing discussion]

Does anyone use a product called called HyperIP by netex. We only have a 3meg line between our headquraters and DR site and the replication jobs are taking an very long time to complete. We have just placed a veeam server in the DR site to perform the Replications to pull from the HQ site to see if that helps. If you have used this product, Could you give us some feedback on what you use it for and what type of improvements you noticed?
Was it beneficial? Do you recommend it?
We run Vsphere ESX 4.x w/ vcenter in the HQ site iscsi storage lefthand/HP, and a physical 2003R2 veeam box using iscsi initiator. The DR site also runs Vsphere ESX 4.0 (no vecenter) with iscsi lefthand/HP storage. The newly introduced veeam box at the dr site is a vm 2008r2 vm.

Any thoughts/tips you could provide would be appreciated!

Thanks!

[Vitaliy S.]

Hello Mark,

Please search for existing topics regarding Netex HyperIP, but I've merged your topic to this one because of the similar performance issues

The first thing that comes to my mind is that you may want to move your backup server back to HQ, specify service console crendetials for the destination ESX host and use Virtual Appliance mode or SAN mode to replicate your VMs to the DR site.

We recommend placing backup server on the DR site mostly if you have ESXi on your DR site, and not full blown ESX host. By the way, what is the current CPU load on your backup server?

Thanks.

[gploosley]

[merged into existing discussion]

I tried searching but couldn't find anyone with my setup so here goes...
Thanks in advance for reading and any kind suggestions to solve my problem(s).
One Live and one DR server at main office plus one DR server at off site DR location
WAN to DR is an ipsec VPN over a 4 Mbit leased line to the Internet
All servers ESXi 4.1 with on board storage - about 1.5 TB
Veeam is installed in a VM on the Live server with 4 vCPUs and 20 GB memory
Live server is 1 month old with dual 6 core Xeons and 64 GB
On VM running Veeam task manager reports CPU usage as 25% - in Replication job setup selected WAN - smallest blocks/highest compression
Backup and Replication Live to DR over LAN runs OK
Trying to replicate to DR location pver WAN - very slow
Did initial replication to NAS, shipped NAS to DR location, copied files to specified location on DR datastore
Trying to replicate the first VM over the WAN, 500 GB, after 31 hours it's reporting 210 GB processed at a rate of 2 MB/s
The delta vrb file copied so far is only 7.8 GB but I think 4 Mbit can copy up to 1.8 GB per hour?
When I check network traffic on the DR location router I see about 1200 Kbps transmit and 2 Kbps receive
Why is the DR location transmitting at more than 50% of the receive rate? Shouldn't it just be receiving the delta?
If the delta file is 7.8 GB why has it taken 31 hours so far - shouldn't 7.8 GB take around 4 to 5 hours at 4 Mbps?
My Datastore has 8 MB block size - would a smaller block size be faster?
Why is it so slow? What can I do to speed it up?
Thank you.

[Vitaliy S.]

Trying to replicate the first VM over the WAN, 500 GB, after 31 hours it's reporting 210 GB processed at a rate of 2 MB/s
The delta vrb file copied so far is only 7.8 GB but I think 4 Mbit can copy up to 1.8 GB per hour?

Replication job is not the same as file copy job. During replication your target VM's image is being rebuilt/updated - all the changes are injected to virtual disks while previous data is pushed into the rollbacks (vrb).

Why is the DR location transmitting at more than 50% of the receive rate? Shouldn't it just be receiving the delta?
If the delta file is 7.8 GB why has it taken 31 hours so far - shouldn't 7.8 GB take around 4 to 5 hours at 4 Mbps?

It does receive only changes. Please note that your VM image gets updated with the new data and this operation also generates traffic between backup console and target ESXi host.

My Datastore has 8 MB block size - would a smaller block size be faster?

No, VMFS block size doesn't have any impact on the backup/replication performance.

Why is it so slow? What can I do to speed it up?

Provided that you're using ESXi host as your destination target, you need to deploy backup server on the DR site to minimize the traffic between backup server and the remote host. Should more info be required, please take a look at this topic: Veeam Server Location

[gploosley]

Thank you for the help, it's much appreciated.

I installed Veeam at the DR location to "pull" the replication as recommended.

I can't see how to create the initial replica to a NAS at the source location that I can then drive to the target location in this scenario?

Obviously it isn't feasible to pull the initial replication over the WAN.

Do I have to somehow create the initial replica to NAS in "push" mode using a Veeam installation on the source LAN and then somehow copy the replication job to the Veeam installation on the target LAN?

[Vitaliy S.]

I can't see how to create the initial replica to a NAS at the source location that I can then drive to the target location in this scenario?

Yes, let me be more specific here. There is no native functionality to perform initial replica seeding from target location, this functionality should be coming in the next release.

The only way to perform this right now, is to move Veeam SQL configuration database together with the initial replica files. Here is an existing topic showing how to do it: Migrating B&R to new/64-bit server?

Do I have to somehow create the initial replica to NAS in "push" mode using a Veeam installation on the source LAN and then somehow copy the replication job to the Veeam installation on the target LAN?

That's correct.

[skrohn]

[merged into existing discussion]

I know that some of my problem is my bandwidth which I am going through the steps to to fix. But I am trying to evaluate if increasing bandwidth will totally fix my problem or not. I have 2 VM's that I am replicating. ESXI 4 at each side. One is an Exchange server that I have given up on for now, the other is a file server and runs a couple apps. The appserver has 705 gig allocated but only 300 in use. I have a 3 meg MPLS line with HyperIP on it. During the day obviously i get worse bandwidth than evening but I'm not seeing alot of difference in times. The appserver will start next replication in about 20 minutes and it has 600 meg of changes but this will take close to 4 hours minimum to run (sometimes more). The phone company tells me that with my contract and bundle I can go up to a 6MB MPLS line is my next jump. Will this handle both my servers with decent rep times or not?

I use Direct SAN access for the replication jobs, with WAN optimized and CBT.

[Vitaliy S.]

With ESXi host chosen as a destination target generally we recommend placing backup server on the DR site, this should give you better replication job performance rates. Thanks.

[skrohn]

I use Veeam for backup to so I would leave a Veeam at the remote office if I wanted to for backup but put a server at the DR site and replicate that way. Will that allow me to also start replicating Exchange?

[skrohn]

sorry another dumb question, so at the DR site I would not be using Direct SAN access anymore then, I would have to change it to _____ ?

[Vitaliy S.]

Yes, that's correct. As soon as you move your backup server to the DR site you will have to use Network mode. The benefits of moving backup server to the DR site in your case should outweigh all the pros of using SAN mode.

Besides, please note that you can continue using Veeam server on the main site for doing backups, while replication jobs can be handled by your remote Veeam backup server.

[xvendrell]

[merged into existing discussion]

Somebody have some experience with replication over WAN?
What WAN accelerators use for this?
How many bandwidth are necesari?

Thank you very much.

Francesc Xavier Vendrell

[Alexey D.]

Hello Francesc,

Please read through this topic, it should cover your questions. If you need more information, please kindly use forum search with these keywords, for example: http://www.veeam.com/forums/search.php? ... ccelerator