Performance expectations with EMC VNXe3100

[dfowler]

Hello,

Case ID#5177812

I have recently installed Veeam v6 to backup Vmare VMs hosted on ESXi v5 servers. Veeam is installed on physical server i.e. HP DL380 G5 (2 x Quad core CPUs, 12 GB RAM) with an iSCSI connection to the EMC (VNXe3100) SAN hosting the VMs. The Veeam backup proxy server is on the same physical server and configured to use 'Direct SAN access' as the Transport mode.

First ever backup was run against 3 VMs with 720 GB of thin provisioned disks (203 GB used) took 5 hrs, 52 minutes to complete at 9 MB/s. The Performance/Bottleneck statistics were: Source 99% > Proxy 51% > Network 4% > Target 0%.

As recommended by Veeam support I applied the latest patch (6.0.0.181). A second backup against 3 different VMs with 720 GB of thin provisioned disks (154 GB used) took 57 Minutes at 43 MB/s, which was a huge improvement. The Performance/Bottleneck statistics were: Source 72% > Proxy 42% > Network 37% > Target 5%.

I don't have any prior exposure to Veeam so I am unsure what to expect when it comes to performance; others have reported performance ranging from 60 MB/s to 1 GB/s, which is quite a range. Is 43 MB/s reasonable?

No problems with Veeam support just trying to figure out if the performance behaviour is 'normal'.

Thanks in advance for any responses.

Dave

[Jfmoots]

That's not bad at all. The number you're quoting sounds like the overall job performance, which is a good reference, but I'm interested in how fast we read the VMs off disk. If you right click on that job, and then select Realtime Statistics, you'll be able to view details on the VMs we backed up. Click on the name of the VM and you'll see even deeper details. How many MB/s are you seeing on each disk on those VMs? I would imagine it's a higher number... but maybe just a little higher.

What types of disks make up that VNX? How many disks are in the array?

[tfloor]

This setup looks a little bit the same as my: bottleneck source and proxy

[Gostev]

David, 45MB/s is what most people report out of box with 1Gb iSCSI SAN without any iSCSI traffic tuning. As a next step, you may want to read the sticky FAQ topic for some tips on improving iSCSI performance. Some Windows 2008 network stack tweaks, jumbo frames and good switches should bring you all the way up to the line speed. Thanks!

[dfowler]

Thanks for the responses.

The VNX solution has a DAP enclosure with 12 x 600 GB 15K SAS drives and a DAE expansion enclosure with 9 x 600 GB 15K SAS drives. Currently we have 2 x datastores at 1TB each. The iSCSI backbone comprises 2 x Brocade FastIron WS624G switches, the ESXi servers are HP DL380 G6s with 2 CPUs (Xeon E5540 6 cores/each) with 96 GB RAM, jumbo frames are enabled.

The VM statistics of the full backup job show read rates between 47 MB/s for the slowest and 56 MB/s for the fastest on the VM disks, which is slightly faster than the overall processing rate of 43 MB/s.

I will take a look at the tuning tips and post my results. Thanks again for the responses.

Dave

[cyberwarrior101]

[merged]

I've just installed VEEAM on an ESXi 5 Server hosted on an EMC VNXe3100. We have additional Volumes for D/R purposes (backups) hosted on a HP LeftHand iSCSI san.

I've started replication and notice that I only get about 30-37MB/S and it continues to show the bottleneck as the source. Any suggestions as to what to change/add that might increase performance? We're trying to replicate core production servers to our colo although at this rate it seems slow.

I should also mention the VM's and the VEEAM R/B server is on the same host both Virtual as I had previously read posts suggesting that the VEEAM server should be a VM also..

We are not using any proxy servers..

Thank You!
Ken

[Gostev]

Make sure you are using direct SAN or virtual appliance (hot add) processing mode?

[dellock6]

Ken, you wrote you are using replicas and you are not using any proxy servers (besides the Veeam Backup server ifself).
Best practices requires to have at least two proxies for remote replicas like you are doing, one in the production site, connected to the VNX (this can be the Veeam Backup server) and another one at the DR site. Speed differences using two proxies as recommended are usually great.

[Gostev]

Luca, he mentioned his bottleneck is source, while no-target-proxy issue would cause target being shown as a bottleneck... and he would be getting much slower speed anyway, probably.

[dellock6]

You are right, I've gone too fast as soon as I read replica+noproxy, it's becoming a sort of auto-reply

[cyberwarrior101]

Sounds good. I'll take a look at the proxy setup and see what's involved in the setup. Documentation available for this online?

One Update: When I originally setup VEEAM, I selected the Enable Application-Aware image processing option - as we're replicating SQL Server and Exchange I wanted to be sure no transactions were lost or corrupted during the copy process. When this was enabled, we saw roughly 20-35MB/Sec. I turned it off per our Vendor's recommendation - speed increased from 20-35MB/Sec to 200+/MB/SEC. In a scenerio where we're replication SQL or Exchange, is it recommended that this be on or off? I want to be sure no transactions are lost during the replication although also need to be sure we can replicate to the Co-location.

Appreciate everyone's input. My apologies if my questions are duplicated, I'm still a newbie
Ken

[Gostev]

You should always have it on for any Windows VM, and it does not affect backup performance anyway - only how guest quiescence is performed before the VM snapshot is created. While data transfer only starts once the VM snapshot is created.