VMware vSphere full replication (initial replication) seem very slow?

[Simon_LBC]

I am currently performing my initial replication job of my brand new ESXi server to my secondary ESXi serveur on the local LAN (1 GPBS link). One of my VM (my main file server) is 12 TB large. My initial replication job is currently running for 25 HOURS and it only complete 7% of the 12 TB.

The replication speed is pretty stable at 20 MB/s, what appears pretty slow to me also... because both server are not in production right now and at linked at 1 GBPS over the local LAN.

According to Veeam, my bottleneck is TARGET and my job statistics are :

SOURCE 74% busy
PROXY 48% busy
NETWORK 28% busy
TARGET 97% busy

My TARGET server is not a brand new server, however it's a pretty high performance Dell PowerEdge T440 server with a local SAS RAID array as the target storage that should be pretty capable of decent R/W speeds, so I don't understand the actual issue of my setup...

Any suggestions??

[jorgedlcruz]

Hello,
What are the proxies utilized to perform the task? Any chance you can put a windows VM, or even a Linux Proxy per ESXi, on each of those storages per ESXi, and then on the job select source proxy the proxy from esx1, and target the proxy on esxi2.

If using HotAdd mode, you should see the maximum speed possible from both ESXi, on source the maximum read speed, and target the writing speed, always with the limit of that 1gbps.

Give it a try. If you can not deploy 2 proxies for some reason, I guess you are using your VBR as proxy as well on some of the sites, add another proxy to the esxi where you do not one.

I am guessing no SAN, just local storage per ESXi, right?

Let us know

[Simon_LBC]

Hi Jorge,

A single proxy is set on the Veeam server (VM) (that is running on the source EXSi as well). The source EXSi host is directly attached to a brand new Hitachi SAN storage (interface is FC 16 GBPS) and the destination host use local storage as I previously indicated. If this is recommended to set a backup proxy on both source and target host, I can surely create a second one on the target using a small Linux VM running on the target...

[jorgedlcruz]

Hello,
On source, as you are using virtual proxy we can not leverage the full DirectSAN, si o HotAdd will be enough, for the target. Yes, I will recommend a Linux proxy that you select on the replica job, living on that local storage.

Will be the fastest you can leverage with the environment you have.

Let us know.

[Simon_LBC]

Hello Jorge,

By setting a Linux Proxy (VM) on the destination host and set it as target proxy manually, I now see a substantial speed gain as my initial replication job ran for 20 hours and completed at 27% now and average speed is now 83 MB/s.

However, while I read the Veeam documentation about the "Direct Appliance (HotAdd)" mode VS DirectSAN mode, I can't figure out how do I set this parameter into my replication (or backup) jobs? Where is this parameter exactly? The exact location of the parameter setting is not indicated in Veeam documentation...

[jorgedlcruz]

Hello,
Glad it is working fine, to be honest, from the 1Gbps you said you have, you can not get any better than those 83MB/s you are maximizing all the bandwidth.

Yes, it could perhaps be a bit better, talking a few MB, using DirectSAN on the Source, as you have Fiber Channel SAN, but to do that you will need a physical proxy connected to same switching, etc. And I have ruled out from the equation reading the hardware you have.

I think the 83 MB/s is a victory already.

Thanks!

[Simon_LBC]

Hi Jorge,

Got it! Thanks.

One last clarification. For this test, I currently installed 2x dedicated Linux proxy, one is on the source (production) host and one is on the target host. Did 2x dedicated proxy are still recommended or did for the source host I could use the built-in proxy from the Veeam server VM that is located itself on the source host? I just want to ensure not having running VM that are unnecessary on the source (production) host.

[jorgedlcruz]

Hello,
If the environment is not big, you can use the VBR on the source as a proxy.
If you will use that VBR for backups, and more replicas, etc. I would recommend a separate Proxy that you can increase resources according to concurrent jobs, etc.

Have a great one!

[Simon_LBC]

Hello Jorge, looks like I'll need your help again... I am still trying hard to succeed on my initial backup job of my new production server using the proxies as you described and while I gain a good overall speed (around 100 MB/s average), since the initial full back is around 26 TB, it still requires pretty much 5 days to complete. But the thing is so far... it fails every single time for several reasons. Last attempt it fails after 3 days with this error :



Sorry my server is in French language... but apparently the failure is due to a certificate error somewhere... not sure if it have to do with PROXY1 machine or with the source VM.

Is it possible that my backup job run for too long (5 Days) and at some point some certificate somewhere expired??? It would be odd if it's the case.

Thanks again!

[foggy]

Hi Simon, please take a look at the following thread and feel free to open a case with our technical support for a closer look in case it doesn't help/isn't applicable. Thanks!

[Simon_LBC]

Hi foggy,

Well I just looked at this thread and it not similar as my case... because all my VMWare servers are brand new and fresh install on ESXi 7 and moreover, the certificat failed right DURING an active backup job.

[foggy]

Support will be able to identify the reason after reviewing the logs.