Host-based backup of VMware vSphere VMs.
Post Reply
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Slow Throughput over 1G Network all the way with Jumbo Frame

Post by habibalby »

Hello,
I have setup a new veeam setup mix between physical and virtual servers as follows and max Rate Speed i can see is 27 MB/s with some VMs and some I can see even worse 16 and 18 MB/s

Physical Server: Dell 2850 Windows 2008 RT Enterprise 64bit
16 GB Ram
Inet Xeon CPU 3.20GHz Dual Core x 2
6 Network Interfaces 1Gbps

Proxy Server: VMware Virtual Machine Windows 2008 RT Enterprise 64bit
8 GB Ram
2 vCPUs qouad cores total 8 vCPUs
3 vNICs e1000

Repository Server: VMware Virtual Machine Windows 2008 RT Enterprise 64bit
8 GB Ram
2 vCPUs qouad cores total 8 vCPUs
3 vNICs e1000

Proxy configured as Network Mode.

Repository is IBM DS3512 1 TB Nearline SAS 7200 rpm

I have tried SAN Mode with the proxy VM connected to the EMC SAN where VMs are running, but no luck same speed rate.

Any help to fine tune this?

Thanks,
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by dellock6 »

Usual question, what are the bottleneck stats of the jobs?
Also, in the virtual proxies and repository I would prefer VMXNET3 vnics,since their drivers offload some tcp/ip operations so they are faster for iscsi access (I'm assuming is iscsi since you said you tried accessing in san mode via the VMs).
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by habibalby »

Hello,
Bottleneck always shows Source :)
I was having all 10 Gbps VMXNet3 in the Proxy VM as well as on the Repository VM but I changed to e1000 I thought they were the bottleneck since my switches are 1 GB and the VMNICs are 1 GB.

What is your suggestion?

I changed from Network Mode Backup to Direct SAN Access, one Job contains three VMs each 20 GB Thin Provisioned, it reach up to 58 MB/s.

One job is running now on VM contains 2 Hard Disks, 1 40 GB Thick and one is 170 GB Thick. It resides on the IBM Storage where I'm offloading the backup. Speed Rate it reached on first disk 33 MB/s and now it's 28 MB/s

Thanks,
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by dellock6 » 1 person likes this post

I think one possible improvement is you can connect the DD3512 to the physical server (it's also the central Veeam server right?), and install repository role on it, and let run all the jobs in the two virtual proxies maybe in virtual appliance/hot add mode, and do not use the physical server as a proxy.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by habibalby »

Hello,
Here's the current utilization of both Proxy Machine and Repository Machine.

Proxy Utilization:
Image

Repository Utilization:

Image
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by habibalby »

dellock6 wrote:I think one possible improvement is you can connect the DD3512 to the physical server (it's also the central Veeam server right?), and install repository role on it, and let run all the jobs in the two virtual proxies maybe in virtual appliance/hot add mode, and do not use the physical server as a proxy.
Hello,
So, the Physical server will act as Backup Engine as well as Repository?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by dellock6 »

No, backup engine (aka proxies) will be only the two virtual proxies, the physical server will act as Veeam Server (the console you connect to...) and as a repository.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by habibalby »

The 191 GB job just finished at final rate 27 MB/s. I will configure the physical server this way and will let you know the result.

Thanks,
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by habibalby »

Hello,

Here's the result of the new setup;

Image
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by habibalby »

More progress result.

Image
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by habibalby »

Job ended 58 MB/s in 0:21:58 minutes.

I can see the vNIC of the proxy Machine is almost reaching 99% all the time while the backup is running.. I will change this interface to a 10 Gbps VMXNet3 and will report the result.

Any further suggestion?

Thanks,
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by dellock6 »

You talked about your environment but gave us no info about the EMC SAN, depending on its raid/disk/cache configuration its performances can highly vary. If the network is at 100% on a 1G speed, I would expect about 120 MBits since its the limit of this speed. vmxnet3 is not only about 10G but also offers some offloading network activities, so it can further increase your stats.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by habibalby »

hello,

VMs SAN:
EMC AX4-5i
SAS Disks 402 10K into two disk pools as RAID5

Repository SAN:
IBM DS3512 11 disks 1 TB Nearline SAS 7200 rpm into RAID 5 of two pools.
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by habibalby »

Now proxy Machine with 10Gbps VMXNet3
Image
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by habibalby »

Worried, I'm getting Unable to Establish Direct SAN Connection while the LUNs are appearing in the Disk Manager :)
Image
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by Gostev »

Did you mount VMFS LUNs back to the backup proxy after you have changed VM networking to use VMXNET3?
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by habibalby »

Yes, they are all mounted. VMFS LUNs from IBM as well as From EMC.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by dellock6 »

Try to force the proxies to run in hotadd mode.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by habibalby »

dellock6 wrote:Try to force the proxies to run in hotadd mode.
By unchecking Failover to Network Mode?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by dellock6 »

Only for testing, to be sure you are using virtual appliance.
On production do not turn off network failover, since it's your save if something does not work with other modes.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by Gostev »

habibalby wrote:Yes, they are all mounted. VMFS LUNs from IBM as well as From EMC.
Then I am confused. This means you are trying to use Direct SAN Access mode, but then Luca is talking about hot add? I think you should really open a support case, as it is extremely ineffective to try and troubleshoot anything over the forum messages. We have already went through the whole page of screenshot in just a few hours, and this is really scary trend :D webex would be so much easier!

Something else to keep in mind though. After changing the backup proxy configuration, you need to be sure to perform storage rescan (or wait for it to happen automatically - once every 24 hours), otherwise the B&R engine will use the older data of connected/accessible volumes - which may not be good after the changes you have made to your backup proxy server.
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by habibalby »

Hello,
Support case is: [ID#5193959]

I'm surprised when I backup a VM located in the EMC SAN LUN in Direct SAN Access the job success for that VM.

But, when I do the same with a VM located in the IBM SAN LUN, I got Unable to establish direct SAN Access.
Failed to Open VDDK Disk [is read-only mode] true.

Thanks,
habibalby
Veteran
Posts: 391
Liked: 32 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Slow Throughput over 1G Network all the way with Jumbo F

Post by habibalby »

Got it working via SAN by specifying the datastore manually in the proxy server;

Hummmm, how the vCenter doesn't detect the Datastores on the IBM LUNs.... That's the question.

Image

Image
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 72 guests