ESX 4 and ESX 4.1

[crichardson]

We are currently a mixed shop with many of our source ESX hosts running ESX 4.0 and many of our target hosts running 4.1. I was waiting for full 5.0 support before upgrading our source hosts but was recently advised there could be performance improvements by upgrading to 4.1, specifically with replication. Is this true?

[Gostev]

Depends on what processing mode you are using for your replication jobs. ESX 4.0 has a bug causing very slow network operations, but if you are using direct SAN or virtual appliance mode, then it does not affect you anyway.

[crichardson]

Hi Gostev,

Thanks for the prompt reply. The reason I ask is because we have two replication jobs. One has great throughput and the other does not. Both jobs repliacte VM's to our DR site over a 100mbps pipe (ethernet link, same lan/subnet).

Job 1: File Server
Our file server conssists of 1 VM with 5 virtual disks totalling 600GB. The replicaton job is setup using Direct SAN mode (network failover UNchecked) and application-aware image processing is checked. The initial replicaton took about 20 hours. Each repliaction since takes about 2 hours. The average throughput is well over 200MB/s. The source host is running ESX 4.0 and the target is running ESX 4.1. The DR SAN is an HP EVA 4400 Starter kit (fabric switches and HP drive shelves).

Job 2: Exchange Servers
Our Exchange farm consists of 2 VM's, 1 CAS/HUB (ex03) and 1 Mailbox server (ex04). ex03 has 2 virtual disks totalling 70GB. ex04 has 4 virtual disks totalling about 600GB. The replicaton job is setup using Direct SAN mode (network failover UNchecked) and application-aware image processing is checked. The Exchange logs are NOT truncated (backups take care of this). The intial replicaton of both VM's took about 30 hours. Each replication since takes anywhere from 8 to 20 hours total. EX03 takes about 2 hours, and EX04 takes anywhere from 6-20 hours. The average throughput is 20MB/s. The source host is running ESX 4.0 and the target is running ESXi 4.1. The DR SAN is an HP EVA 4400 Starter kit (fabric switches only) with an IBM 3500.

The only different between the two jobs is the target host ESX version (ESX 4.1 vs ESXi 4.1) and the drive shelves/controllers. I cannot seem to locate any issues with the IBM controllers or drive shelves. Both utilize the same fabric. Is there any way to determine in the Veeam logs which part of the replication is slowing down the process?

[Gostev]

Well that explains, because unlike "fat" ESX, ESXi is not an ideal target for replication with v5 > replication and ESXi

[crichardson]

Well frig, that explains a whole lot now doesn't it. I guess I should have searched a little harder before writting all this up. Well hopefully this can help anyone else out having the same problem. For now I guess I will rebuild the host using ESX 4.1 and wait for the next release of Veeam before switching over to ESXi 5.0.

Thanks for the help and sorry for wasting your time.

Cheers.

[crichardson]

Well, I switched the hosts over to ESX 4.1 update 2 a couple days ago and that didn't help. All other replications are runing fine except our Exchange replication job. It's taking almost 22 hours to run. Is there anyway to determine which part of the replication is taking so long?

[Gostev]

which part of the replication is taking so long?

Not sure what do you mean by this? "Part of the replication". Generally speaking, servers like SQL and Exchange produce a lot of disk changes, and they typically require more time to replicate than regular VMs.

Anyway, you should be able to boost the replication performance significantly (in a few times) by upgrading to v6 and using our new replication architecture.

[ThomasMc]

Anyway, you should be able to boost the replication performance significantly (in a few times) by upgrading to v6 and using our new replication architecture.

+1, my SQL replications are seeing a big improvement and not having to replicate the page file helps as well

[crichardson]

I just noticed V6 was released - awesome! I'll upgrade tomorrow.

By the way, we replicate 3 SQL servers as well as Exchange. They do take long, and are approxamtely 120GB in size each. They take 2 hours. Exchange is about 350GB in size (~650GB thick) and its taking 22 hours. I'll post again after I ugprade to V6. Thanks for the help!

[Gostev]

Please do, we would be very intersted to see the difference and raw numbers!

[crichardson]

I'm currently downloaded V6 as I type. I'll have it installed in a couple of minutes.

Our ESX hosts are still running ESX 4.1. Will I see even more improvement if I upgrade them to ESXi 5? Also, if I add vCenter to my v6 release, and the hosts are all version ESX 4.1, when I upgrade those hosts, will Veeam automatically pickup the changes of the host ESXi version?

[crichardson]

Ignore my question, it's answered in the What's New document, and here: Veeam v6 and vSphere versions and host types

[crichardson]

Massive improvement. The initial replication for Exchange took nearly 14 hours (down from 22). I have setup the job to run every hour between 9AM and 9PM. The job takes anywhere from 20 - 60 minutes to run. Amazing! I'm currently using a single proc physical box at our DR site as a proxy server. The job stats are indicating the proxy server is the bottleneck so I'm going to build a VM at our DR site to take on additional replication jobs.

[Gostev]

Nice, your improvement is inline with what I expected. Thanks for the update!

Please note that proxy bottleneck refers to the source proxy, not the target proxy. Target proxies generally consume very little CPU, as they do not need to do any heavy lifting. What you need here is more powerful proxy at the source site.

[crichardson]

Thanks Gostev.

Curently only one replication job is running. I assume my source proxy is the issue? My current souce proxy is a physcial machine connected directly to the SAN (fiber). It has a single E5405 Xeon and 4GB of memory. What if I was to create a second proxy in a virtual machine with 2vCPU's and 2-4 GB of ram? I've always been one for backing up via physical machines rather than virtual machines. But maybe using virtual machines as backup proxies is more productive than it used to be?

[Gostev]

Hi Corey, I would rather recommend upgrading CPU (or the host itself), as the current one is 5 years old and cannot provide adequate performance. As you may have noticed, we require "modern" CPU in the system requirements.

That said, you should don't treat bottleneck as the problem, but merely as indication of "weakest" spot in your backup infrastructure. For example, if you upgrade CPU on your physical proxy, bottleneck will simply shift somewhere else (although of course, you will see performance improvement along).

If you are happy with performance, you should not be changing anything. You can deploy additional proxies for redundancy, but you should only do that for redundancy, and if you are running multiple concurrent jobs and not fitting your backup window.

4 vCPU virtual proxies in Hot Add processing mode are usually as performant as physical proxies, however they put significant load on hosts running them, and thus may affect consolidation ratio. Physical proxies are usually cheaper from VMware licensing perspective, and they are not affected by virtual environment disasters (for example, you will always be able to restore VMs via physical proxy, while all of your virtual proxies might be lost due to disaster presenting recovery difficulties).