Veeam Replication Performance & Best Practices

[guillermo.lozano]

Hello Community,

I’m replicating VM's to different hardware ESXi servers, but same architecture Intel Processors.

Can anyone refer to the replication best practices to achieve best performance and short replication windows?

I'm running ESXi 5.1 on all servers
Source Servers: IBM x3550 on IBM DS3512 Storage RAID 10 & 5
Source Storage: IBM DS3512 Storage RAID 10 & 5

Destination Sever: IBM Pure Flex x220 nodes
Destination Storage: IBM v7000 Storage RAID 10 & 5

Veeam B&R Server is 24 GB ram, paging and replication meta data on different VHD's on RAID 10 on Source Hardware

Thank you.

Regards,
Guillermo.-

[Vitaliy S.]

Hi Guillermo,

The most important thing you should keep in mind is proxy server, make sure you have both of these on source and target ESX(i) hosts. With this setup VM traffic going across the link will be compressed at source proxy and decompressed at target proxy.

Thanks!

[veremin]

Hi Guillermo,

Furthermore, it might be worth deploying additional VB&R server at your DR site and letting it be responsible for only remote replication job. Thus, in disaster situation all functionality (Failover, Failback and etc.) can be performed by DR VBR Server itself without any issues.

Additionally, you might want to install Enterprise Manager to have visibility across two backup servers.

Hope this helps.
Thanks.

[guillermo.lozano]

Hello, thank you both for the quick reply.

I already have a proxy on the DR Site and it's the destination proxy for all the replication jobs.
I also have Veeam One, Veeam ONE Reports, Business View & Enterprise Manager installed.

v.Eremin, when you say "additional VB&R server at your DR site and letting it be responsible for only remote replication job" at what level do you mean to be responsible of the replication job? to let it act as the destination proxy?

Thanks in advanced.
Regards,
Guillermo.-

[veremin]

My original suggestion was to deploy additional VB&R server at DR site and create/manage remote replication jobs from there.

Such scenario would guarantee that in case of disaster, when, for instance, your production site with primal VB&R server goes down, all necessary steps, like Failover, Failback and etc. can be performed smoothly by DR VBR Server itself.

Hope this helps.
Thanks.

[guillermo.lozano]

Hi, thank you for the reply.

I'm not sure about the licensing for that design, Is there a limit for VB&R installations when i have enterprise license?

It’s enterprise but only for three hosts (Source hosts in 1st post, the ones i want to backup) and i'm replicating to other three hosts (destination hosts on 1st post).

Thanks.
Regards,

[foggy]

There are no limitations on the number of Veeam B&R installations as Veeam B&R is licensed per source host sockets.

[guillermo.lozano]

Great. thank you all for the help!
Regards,
Guillermo.-

[guillermo.lozano]

Hi Community,

Using v.Eremin's proposed solution, when i create the replication job on the destination DR VB&R, where should the repository for replica metadata be?
On the source or destination VB&R?

Thanks in advanced.
Regards,
Guillermo.-

[veremin]

It's recommended to specify source proxy to store replica metadata.

Hope this helps.
Thanks.

[guillermo.lozano]

Thank you!

[guillermo.lozano]

Hi Community,

After several tests, i've noticed that when a replication takes place, the first and last reconfiguration of the Source and Destination Proxies is really slow.

Does anyone what does this reconfiguration now how can I asses exactly whats happening at this stages

Thanks in advanced.

Regards,
Guillermo.-

[guillermo.lozano]

Hi,

Is the continiously option described in KB1522 http://www.veeam.com/kb1522 not available on version 6.5.0.128 64 bits?

Thanks.
Regards,
Guillermo.-

[veremin]

Is the continiously option described in KB1522 http://www.veeam.com/kb1522 not available on version 6.5.0.128 64 bits?

Nope. It’s still there.

Go to the replication job settings -> Schedule -> Run the job automatically -> Periodically every -> Hours -> Continuously

Hope this helps.
Thanks.

[guillermo.lozano]

Hi, Thanks again Vladimir.

Can anyone help with prior post?

After several tests, i've noticed that when a replication takes place, the first and last reconfiguration of the Source and Destination Proxies is really slow.
Does anyone what does this reconfiguration now how can I asses exactly whats happening at this stages ?

Thanks.
Guillermo.-

[foggy]

Guillermo, could you please clarify what are you referring to here?

the first and last reconfiguration of the Source and Destination Proxies is really slow.

Probably a screenshot will help.

[guillermo.lozano]

Hi,

During the replication when veeam takes the snapshot on the esxi server it also reconfigures the source proxy as the events on the esxi server describes:



this step occurs in the source and the destination proxy.

Thanks.
Regards,
Guillermo.-

[veremin]

It seems like it’s a hot-add method that is being used by both of the proxies.

In this mode, Backup proxy server uses SCSI Hot Add capability of VMware to attach disks of backed up VM directly to itself, and thus get direct access to data stored inside. The process of attaching/deattaging disks results in reconfiguring virtual machines.

Thus, it might be worth specifying Network proxy mode and seeing whether it helps.

Thanks.

[tsightler]

After several tests, i've noticed that when a replication takes place, the first and last reconfiguration of the Source and Destination Proxies is really slow.

Sorry, but "really slow" is a little vague, how long is that. This is typically the hotadd step, which would normally take 30-60 seconds. If you're sending data over a slow link or if the change rate of the VM is low then you might find that forcing network mode will give you better performance.

Does anyone what does this reconfiguration now how can I asses exactly whats happening at this stages

Typically this is the stage that's performing the hotadd of the source and target VM disks to the appropriate proxies. This requires a lot of communications with vCenter and from vCenter to the hosts. Typically you can determine exactly what's happening by reviewing the job logs on the Veeam server if you really want to dig into the guts of what's going on pretty much everything is logged there.

[guillermo.lozano]

Thank you both for the quick reply.

It's clear for me the action after your explanation. isn't network mode slower than virtual appliance?

The slowest step is the final reconfiguration, which takes around 2 -3 minutes.

i'm trying to close each gap so continuos replication is done as fast as possible.

Is there anyway to improve this time?

Thanks.
Regards,
Guillermo.-

[foggy]

isn't network mode slower than virtual appliance?

It is (typically), however it does not have this overhead required to hotadd disks.

Is there anyway to improve this time?

What is your primary bottleneck, according to the job stats?

[tsightler]

It's clear for me the action after your explanation. isn't network mode slower than virtual appliance?

In theory, virtual appliance mode can transfer data faster than network mode, which is important for full backups, however, as you have seen, when looking to replicate incremental data it's not all about speed of data transfer but rather the total time per job run includes all of the overhead in setting up and tearing down the proxies for replicaiton. Not only that, but in most replication cases link speeds between environments are the limiting factor for data transfer, although I didn't seen where you've mentioned anything about your link speeds.

For near-continuous replication there is usually not that much data transferred per-replication cycle, so the focus should be on reducing the amount of overhead per job run. Network mode has far less overhead since it doesn't require a reconfiguration of the proxies on job start/stop so even if it's slightly slower transferring the data, the job will probably take less time overall. Certainly this is very easy to test by simply setting the proxies to network mode and running a few job cycles.

For example, if it takes 2 minutes to do reconfiguration for hotadd in your environment, and those 2 minutes goes away completely, then it doesn't really matter if the transfer takes 15 seconds longer, you've still saved 95 seconds of total time.

[guillermo.lozano]

Great Answer, very clear.

Is the transport mode configured at proxy level or job level?

I've been seeing the logs, but they are too long for posting, i dont see any button to attach a file.

thanks for all!
Regards,
Guillermo.-

[guillermo.lozano]

My primary bottleneck depends on the moment, it can be source or target.
currently i'm testing on LAN with WAN dedup. so i can calculate data transfers since i'll be replicating over 4 Mbps uplink with 50-100 ms latency.

Thanks.
Regards,
Guillermo.-

[tsightler]

Is the transport mode configured at proxy level or job level?

Must be configured on the proxy.

I've been seeing the logs, but they are too long for posting, i dont see any button to attach a file.

The forum isn't really the place for log diving. If you want to get that deep and can't parse the logs yourself probably best bet is to open a support case. They can look at the logs and see where the delay actually occurs. Of course if this step actually takes vSphere 2 minutes to complete we're just waiting on it to complete.

since i'll be replicating over 4 Mbps uplink with 50-100 ms latency.

So network mode will not be your bottleneck since your link speed will be so slow. There's really no advantage to using virtual appliance modes at such low speeds since any mode will be able to transfer data at those speeds. Performance differences between transport modes don't really come into play until link speeds are over 100Mbps, and even then the differences are usually minimal until we get to gigabit speeds.

[guillermo.lozano]

Hi Tom, thank you again for a clear answer.

Right now i'm using a VB&R for backup on source and another on destination just like Vladimir Eremin suggested.

On each VB&R i have the local proxy and the remote proxy.

Should i configure Network mode in the remote proxy on the destination VB&R that does the replication?

I'll try it and get back with the results.

Regards,
Guillermo.-

[guillermo.lozano]

Hi Community,

After changing the transport mode on both local and remote proxy on the VB&R located on DR hardware which handles the replication I got much better results.

> 50% faster on network mode replicating on LAN speeds with Optimal Compression and WAN Target Dedup.

ESXi Operations takes only few seconds to complete, even though the bottleneck is the source.

I'm not sure if there is any other good practice to improve replication times under these conditions, but so far current replication performance meets the RPO and fulfills the expectations.

I'll be working on concurrent jobs to see the performance for both replication & Backup.
Thank you all for the help to accomplish this objective!!

Hardware and specs for the current tests:

I'm running ESXi 5.1 on all servers

Source Servers: IBM x3550 on IBM DS3512 Storage RAID 10 & 5
Source Storage: IBM DS3512 Storage RAID 10 & 5 - 1 Gbps ISCSI Links

Destination Sever: IBM Pure Flex x220 nodes
Destination Storage: IBM v7000 Storage RAID 10 & 5 - 10Gbps FCoE Links

Source & Destination Veeam B&R proxy Servers are 16 GB ram, 4 vCPU w/2 cores each, paging and replication meta data on different VHD's on RAID 10.

Note about Current Bottleneck:

Load: Source 98% > Proxy 10% > Network 0% > Target 75%

After the tests source & destination hardware will be inverted so i guess that bottleneck will be destination by then.

Best Regards,
Guillermo.-

[guillermo.lozano]

Hi Community,

I'm testing simultaneous replication jobs & I’m not sure about the Max Concurrent Tasks value.

My proxies have 4 vCpu's with 2 cores each and 16 GB Ram and Max Concurrent Tasks is configured at 8.

I’m replicating the test job with continuous sched and I’ve launched another test job (for the second time now), with the source proxy VM that also has VB&R, Enterprise Manager & Veeam ONE plus another VM.

The continuous replication is waiting with this message while the new job runs: Waiting for backup infrastructure resources availability.
Both source & Destination Proxies have very little LOAD.

I'm not sure if I understand the Veeam Concurrent Job Concept, because both jobs won't run together.

Thanks.
Regards,
Guillermo.-

[Gostev]

Hi Guillermo, this is unexpected. Looks like you should have plenty of available slots for concurrent jobs. Please open a support case for deeper investigation. Thanks!

[guillermo.lozano]

Hi Gostev, thanks for the information.

I'll open a case and post the results of the investigation.

Thanks.
Regards,
Guillermo.-