Slow Throughput over 1G Network all the way with Jumbo Frame

[habibalby]

Hello,
I have setup a new veeam setup mix between physical and virtual servers as follows and max Rate Speed i can see is 27 MB/s with some VMs and some I can see even worse 16 and 18 MB/s

Physical Server: Dell 2850 Windows 2008 RT Enterprise 64bit
16 GB Ram
Inet Xeon CPU 3.20GHz Dual Core x 2
6 Network Interfaces 1Gbps

Proxy Server: VMware Virtual Machine Windows 2008 RT Enterprise 64bit
8 GB Ram
2 vCPUs qouad cores total 8 vCPUs
3 vNICs e1000

Repository Server: VMware Virtual Machine Windows 2008 RT Enterprise 64bit
8 GB Ram
2 vCPUs qouad cores total 8 vCPUs
3 vNICs e1000

Proxy configured as Network Mode.

Repository is IBM DS3512 1 TB Nearline SAS 7200 rpm

I have tried SAN Mode with the proxy VM connected to the EMC SAN where VMs are running, but no luck same speed rate.

Any help to fine tune this?

Thanks,

[dellock6]

Usual question, what are the bottleneck stats of the jobs?
Also, in the virtual proxies and repository I would prefer VMXNET3 vnics,since their drivers offload some tcp/ip operations so they are faster for iscsi access (I'm assuming is iscsi since you said you tried accessing in san mode via the VMs).

[habibalby]

Hello,
Bottleneck always shows Source
I was having all 10 Gbps VMXNet3 in the Proxy VM as well as on the Repository VM but I changed to e1000 I thought they were the bottleneck since my switches are 1 GB and the VMNICs are 1 GB.

What is your suggestion?

I changed from Network Mode Backup to Direct SAN Access, one Job contains three VMs each 20 GB Thin Provisioned, it reach up to 58 MB/s.

One job is running now on VM contains 2 Hard Disks, 1 40 GB Thick and one is 170 GB Thick. It resides on the IBM Storage where I'm offloading the backup. Speed Rate it reached on first disk 33 MB/s and now it's 28 MB/s

Thanks,

[dellock6]

I think one possible improvement is you can connect the DD3512 to the physical server (it's also the central Veeam server right?), and install repository role on it, and let run all the jobs in the two virtual proxies maybe in virtual appliance/hot add mode, and do not use the physical server as a proxy.

[habibalby]

Hello,
Here's the current utilization of both Proxy Machine and Repository Machine.

Proxy Utilization:


Repository Utilization:

[habibalby]

I think one possible improvement is you can connect the DD3512 to the physical server (it's also the central Veeam server right?), and install repository role on it, and let run all the jobs in the two virtual proxies maybe in virtual appliance/hot add mode, and do not use the physical server as a proxy.

Hello,
So, the Physical server will act as Backup Engine as well as Repository?

[dellock6]

No, backup engine (aka proxies) will be only the two virtual proxies, the physical server will act as Veeam Server (the console you connect to...) and as a repository.

[habibalby]

The 191 GB job just finished at final rate 27 MB/s. I will configure the physical server this way and will let you know the result.

Thanks,

[habibalby]

Hello,

Here's the result of the new setup;

[habibalby]

More progress result.

[habibalby]

Job ended 58 MB/s in 0:21:58 minutes.

I can see the vNIC of the proxy Machine is almost reaching 99% all the time while the backup is running.. I will change this interface to a 10 Gbps VMXNet3 and will report the result.

Any further suggestion?

Thanks,

[dellock6]

You talked about your environment but gave us no info about the EMC SAN, depending on its raid/disk/cache configuration its performances can highly vary. If the network is at 100% on a 1G speed, I would expect about 120 MBits since its the limit of this speed. vmxnet3 is not only about 10G but also offers some offloading network activities, so it can further increase your stats.

[habibalby]

hello,

VMs SAN:
EMC AX4-5i
SAS Disks 402 10K into two disk pools as RAID5

Repository SAN:
IBM DS3512 11 disks 1 TB Nearline SAS 7200 rpm into RAID 5 of two pools.

[habibalby]

Now proxy Machine with 10Gbps VMXNet3

[habibalby]

Worried, I'm getting Unable to Establish Direct SAN Connection while the LUNs are appearing in the Disk Manager

[Gostev]

Did you mount VMFS LUNs back to the backup proxy after you have changed VM networking to use VMXNET3?

[habibalby]

Yes, they are all mounted. VMFS LUNs from IBM as well as From EMC.

[dellock6]

Try to force the proxies to run in hotadd mode.

[habibalby]

Try to force the proxies to run in hotadd mode.

By unchecking Failover to Network Mode?

[dellock6]

Only for testing, to be sure you are using virtual appliance.
On production do not turn off network failover, since it's your save if something does not work with other modes.

[Gostev]

Yes, they are all mounted. VMFS LUNs from IBM as well as From EMC.

Then I am confused. This means you are trying to use Direct SAN Access mode, but then Luca is talking about hot add? I think you should really open a support case, as it is extremely ineffective to try and troubleshoot anything over the forum messages. We have already went through the whole page of screenshot in just a few hours, and this is really scary trend webex would be so much easier!

Something else to keep in mind though. After changing the backup proxy configuration, you need to be sure to perform storage rescan (or wait for it to happen automatically - once every 24 hours), otherwise the B&R engine will use the older data of connected/accessible volumes - which may not be good after the changes you have made to your backup proxy server.

[habibalby]

Hello,
Support case is: [ID#5193959]

I'm surprised when I backup a VM located in the EMC SAN LUN in Direct SAN Access the job success for that VM.

But, when I do the same with a VM located in the IBM SAN LUN, I got Unable to establish direct SAN Access.
Failed to Open VDDK Disk [is read-only mode] true.

Thanks,

[habibalby]

Got it working via SAN by specifying the datastore manually in the proxy server;

Hummmm, how the vCenter doesn't detect the Datastores on the IBM LUNs.... That's the question.