Slow WAN replication after seed

[StixForBrains]

I'm a Veeam noob with a question about things I should double-check.
I have a source FC-san at main office. Target SAS-san at remote site across a WAN. Link is 20 Mb fibre.
I did an initial replication job of 3 VMs to removeable media (10TB Drobo-FS) and took them to remote site. Copied to the target san in the lun location originally specified (confirmed in the text file the replica job creates on initial replica). was about 315Gig total original.
Then at the main site I started the original replication job a second time. A day and half later now it is still grinding at only about 25% done.
The job details shows 616 KB/s The backup mode is showing as NBD with change block tracking. I had the parameters set for Compression optimal, and storage WAN target optimized. And being a second pass with CBT active I would have expected much better throughput.

As a second test I just now started a file copy using the same veeam server at the same time as my current replica is still running, from same source san to same target san and I get a steady 2 MB/s. My veeam-VM doesn't seem to be working hard I gave it 4 CPUs and its barely over 25% and the esxi host its running on is not working hard either.

Is something wrong with my first replica or with the method I used with the removeable media. or any list of things I can check?

[Vitaliy S.]

Hello Tony,

File copy operation is a little bit different from replication job, so it's not quite fair to compare the results, though the speed you have is almost the same - very poor.

If I were you I would also try to upload any big file from backup server to your target datastore using vSphere Client -> Datastore Browser. That would be a good starting point for our troubleshooting. By the way, what are the servers you're replicating?

Thanks!

[StixForBrains]

Hey Vitaliy,
The 3 VMs I am testing with are all win2003R2 32 bit guests, one is sharepoint , one is exchange 2003, one is BES.
I am trying another test replication right now smaller and simpler VM right now just one VM same O/S but only service running is RSA authentications 20gig thick disk a single drive vmdk only. It is achieving 1MB/s from same source LUN to same target LUN at remote site. Which doesn't seem bad considering the first test is still running.
My theoretical total limit with a 20Mbs fibre lan extension would be about 2.5 MB/s, so I'm not sure why you thought 2 MB/s is slow for a file copy. I suppose compression would help some in the percieved processing rate? But this single VM test is a first pass so CBT will not help.
But I did expect my pre-seeded 3Vm test exch/sharpoint/bes job to be much faster than 616 KB/s with the CBT enabled.
One thing I wondered, should Veeam be automatically registering my seeded replica inside vcenter when it starts the CBT pass or not till its done the second pass?

My backup server is a veeam-VM at the source site ? Can you describe more fully the datastore test that you want me to try? My understanding is if I'm using my vi Client to upload to remote datastore it will be coming from my workstation not in anyway from Veeam guest. Or did you mean something else?

Thanks
Tony

[Vitaliy S.]

Tony,

I might have missed your statement about 20Mbs fibre lan, my bad. The reason why I've asked you about source VM type is that Exchange/SQL/Sharepoint VMs can produce a pretty large VIB/VRB files, as the applications inside those VMs are highly transactional. Besides the processing rate may differ from one VM to another due to a number of empty blocks on the virtual disks etc.

You may disregard my datastore test as I've missed your throughput total limit. But I was wondering what upload speed you would get while using vSphere Client from the backup console VM.

Actually, compression is not used for replication because an original, unchanged .vmdk should be presented to target host. And you do not need to register replicated VM manually, it would be done for you automatically at the next job run.

By the way, what replication job mode are you using? Have you chosen Virtual Appliance mode?

Thanks!

[StixForBrains]

Hi Vitaliy

My second test I described above with the single vm (win2k3 rsa srvr) ended up doing 2MB/s total on 1st pass and then 5MB/s on second pass. network mode. There would be have been very minimal difference on server from the two hours between the 1st and 2nd replications.

The original test of 3 vm's together is still processing slowly, I'm starting to think I must have done something wrong while copying my initial rep from the portable storage to the target maybe? My inital rep went to the NFS storage unit easy at about 25MB/s. But I did have some trouble figuring out how to get the files into the target datastore since I couldn;t get esxi to mount the MFS as a datastore. Maybe its ok since the 2nd pass replica seems to be working just very slow.

for your question, I'm using "network" not virtual applicance. I thought it was network to use for across a WAN? The session details says "NBD with change block tracking"

Is virtual appiance something I can change after this pass is finally done, before I try the next repication pass, or do I have to start over with a fresh seed copy?

The exchange VM is now done 155gig took over 68 hours running at 669KB/s, it did register the exch_replica now. The sharepoint is running now, and Blackberry server next. I thought I might get some dedupe benefits from 3 togther in same job since they were same o/s level, and I have tons of cpu on my Veeam-VM, but would it maybe be better to do such "transactional" vms each in separate job?

Thanks
Tony

[Vitaliy S.]

Hi Tony,

If you switch to Virtual Appliance mode, then traffic will go through ESX I/O stack to Veeam Backup server and then it will be transmitted over WAN in the same way it did with Network mode. If you happen to use on site destination targets then the speed performance would be higher with VA mode.

You may change the replication mode without recreating jobs from scratch. If you think that you might have done something wrong, feel free to contact our support team and ask for an assistance.

Reading back through my reponse above the processing rate might be different from one VM to another, moreover the processing rate is not the actual transfer speed. The processing rate is calculated using this formula: Total VM size/Time. That being said, no doubt that your connection link was fully saturated, however you may also try some WAN acceleration tools (such as HyperIP) to see if that helps you to improve replication performance.

Thanks!

[st0623]

Tony, I'm with NetEx, on the HyperIP team, and our product has the ability to rate limit replication by time of day so you don't saturate the link during peak times. We can also offload compression from Veeam Backup & Replication and push more data down the pipe at the HyperIP throughput rate limit. We design application throughput around the HyperIP key, how we sell and rate limit throughput, at around 90-95% of the link speed, then plan on getting at least 2:1 to 3:1 compression, depending on the data. Reach out to me and I'll talk about what we can do to help. emai: steve.thompson@netex.com. You can eval HyperIP for free for 30 days with Veeam Backup & Replication.