bottleneck source and proxy

[tfloor]

Hello,

Using Veeam V6 P3.
Physical Windows 2008 R2,SP1 2x Quadcore E5335 @ 2 Ghz, 14GB mem
Forward incrementals Mo-Thu
Synthetic full on Friday
4x Gigabit Nics in a trunk to a HP 2848 switch
Mode: Direct SAN access
Concurrent jobs: 1
Snapshot: 14

Source:
EMC VNX5300

Target:
QNAP TS-EC1279U-RP
4x Gigabit Nics in a trunk to the same HP 2848 switch.

Hypervisor:
VMware ESXi 4.1


We have bought the QNAP because we had a Datadomain DD610, but that was a bottleneck (target) for about 90%.
So we placed the QNAP and we saw about the same speeds as before. A little bit more. Around the 100MB/s
Now the bottlneck status = Source=90%, Proxy=85%, Network=25%, Target=10%

What can i do to get a better performance, is it the VNX or is it the Proxy.
How can i test that?
I don't have more physical servers with a HBA to the SAN , so i can't try to run more jobs concurrent with another proxy.
I've tried a VM as proxy one time, but the speed was around 25 MB/s ,,, so that was a very fast test with useless results.
I've bought veeam to get rid of backup exec, because that very slow.
It's faster than BE, but i think i can get more out of these specs.

Please give me some good advice.
Thanks.

[dellock6]

Have you tested to extract other files from the VNX, to be sure that value of 100MB/s is not the limit the VNX can do? Have never used a VNX so I do not know how it can be configured, sata or sas, which raid...

Datadomain would have been (almost sure...) the bottleneck because of the inline deduplication it was applying to data it was receiving.

[tfloor]

Luca,
The vnx has several raid 5 groups with Luns.
It has SAS and NLSAS raid groups. So 10K and 7.5K i think
I only have vmfs volumes on it. So who has a method to test the speed?
The vnx is a brand new system from emc. The server uses emc powerpath for lun access.
Please let me know how to perform tests for my bottlenecks.

[tfloor]

Luca,
The vnx has several raid 5 groups with Luns.
It has SAS and NLSAS raid groups.15K SAS/7K2 NL-SAS
I only have vmfs volumes on it. So who has a method to test the speed?
The vnx is a brand new system from emc. The server uses emc powerpath for lun access.
Please let me know how to perform tests for my bottlenecks.

[dellock6]

I'm not saying the VNX is slow, never used one but I know it's a good product. I'm only finding a way to remove some variables from your problem
The speed can be tested with some IOmeter test, but it would not be relevant to Veeam activities, the tests would be eventually fast on local activities but slower going outside.
A "quick and dirty" test I would do would be to create a zeroeager-thick disk on the VNX, and trying to copy it using FastSCP. Knowing the size and measuring the time you can find out a first speed value. It's not a definite answer but we can see if you got the same speed even outside Veaam Backup.

[tsightler]

Physical Windows 2008 R2,SP1 2x Quadcore E5335 @ 2 Ghz, 14GB mem

So a 5335 and 2Ghz is pretty low-end hardware at this point and you will not get maximum performance from it.

Now the bottlneck status = Source=90%, Proxy=85%, Network=25%, Target=10%

Notice how close you are to maximizing the Proxy. I'd suggest testing with compression set to "Low" and see if you get a performance improvement.

I wouldn't expect this hardware to do much more that 100-150MB/s, although perhaps you could stretch it a little by running multiple jobs, although not to many as you can see that a single job is already using a good bit of CPU.

[tfloor]

I'm not saying the VNX is slow, never used one but I know it's a good product. I'm only finding a way to remove some variables from your problem
The speed can be tested with some IOmeter test, but it would not be relevant to Veeam activities, the tests would be eventually fast on local activities but slower going outside.
A "quick and dirty" test I would do would be to create a zeroeager-thick disk on the VNX, and trying to copy it using FastSCP. Knowing the size and measuring the time you can find out a first speed value. It's not a definite answer but we can see if you got the same speed even outside Veaam Backup.

I've not the knowledge for this.
Can you explain me how i can create a zeroeager-thick disk on the VNX ?

[tfloor]

So a 5335 and 2Ghz is pretty low-end hardware at this point and you will not get maximum performance from it.
Notice how close you are to maximizing the Proxy. I'd suggest testing with compression set to "Low" and see if you get a performance improvement.

I wouldn't expect this hardware to do much more that 100-150MB/s, although perhaps you could stretch it a little by running multiple jobs, although not to many as you can see that a single job is already using a good bit of CPU.

Hi,
Ok so my hardware is a little bit too old for the high performance i will expect.

I've done the test with the compression.
Here are the results:

NONE: = 122MB/s Load: Source 63% > Proxy 39% > Network 60% > Target 67%
LOW: = 124 MB/s Load: Source 78% > Proxy 50% > Network 45% > Target 38%
OPTIMAL: = 123 MB/s Load: Source 91% > Proxy 77% > Network 18% > Target 8%
BEST: = 94MB/s Load: Source 54% > Proxy 88% > Network 9% > Target 5%


Can someone explain me the bottleneck for this results?.
I don't see much differents with lower compression.
Thanks.

[dellock6]

read the results in this way: as long as you are using the lowest compression methods, the bottleneck is the VNX, that can send out data at a maximum rate of about 120MB/s. In fact you see similar results in the first three methods.
Once you raise the compression to best, the proxy is receving too much data given its CPU power, so it becomes the new bottleneck. But the first three results seems to confirm the maximum speed of the VNX is 120MB/s. This number sounds really familiar for a 1 GB/s connection, do not know how the VNX works, but maybe even having multiple gigabit connections, it uses once at a time?

[tfloor]

Luca,

The vnx is a San that is connected with fiber. There are only Ethernet cables for management.
s for file level support. Without an add-on FC module, there are four FC ports, two on each SP. The add-on FC module does include four usable FC ports on each SP. So, with the add-on, there are 12 usable FC ports, 6 on each SP.
We don't have the addon. Connected Hba's with 4gb FC from San switch inside hp blade chassis.

What do you mean with " sounds familiar with 1 GB/s ?

[Gostev]

It almost looks like the data transfer speed is limit to 1Gb LAN throughtput, that's what he means.

Do you have Patch #3 installed, since it bring some changes to bottleneck stats?
Just so that I understand what product build generated the stats above.

[dellock6]

Sorry, I read on the first post about 4 gbits ethernet in trunk mode so I thought it was iscsi, forget about it. Anton got exactly what I meant.

How is the physical Veeam Server connected to the FC fabric? You use an HBA on it right? Have you checked the job is effectivly running in san mode, or is it failing over to network?

[tfloor]

Gostev,

Yep ,p3 is installed from the beginning.
I also have the idea that I always have the max of 1 GB THROUGPUT.

[tfloor]

Sorry, I read on the first post about 4 gbits ethernet in trunk mode so I thought it was iscsi, forget about it. Anton got exactly what I meant.

How is the physical Veeam Server connected to the FC fabric? You use an HBA on it right? Have you checked the job is effectivly running in san mode, or is it failing over to network?

The physical server has an FC card with 1 pair FC 4 GB connected to the FC switch integrated in blade chassis.
The physical server has also a 4 GB static lacp trunk to a 2848
The qnap has the same 4 GB static lacp trunk on 4 other ports of the same switch

[Gostev]

Target:
QNAP TS-EC1279U-RP
4x Gigabit Nics in a trunk to the same HP 2848 switch.

Would it be possible to somehow monitor the traffic on each of those 4 NICs when the backup is running?
Because this looks to be the only place where you have 1Gb in the picture, so worth a closer look.

[Gostev]

The physical server has also a 4 GB static lacp trunk to a 2848

Can you look at the networking tab of the Windows Task Manager on this server?
I assume it should show all those 4 NICs separately, so this might be easiest.

[tfloor]

Most of the time only 1 nic is blowing.
But in my opinion that's why you need multiple jobs to load balance and get more speed.
But that's not working. Speed is the same

But if i forget the target and backup to the local backup disk I get the same speeds. That will say something I think. Don' t you think ?

[Gostev]

No, because that's exactly the speed I would expect from a modern hard disk as well. If you have SSD handy, try testing with one.

[tfloor]

It's hard to understand the conclusion.
Doe you mean the speed of 120MB is the speed of modern hard disk. Also the vnx 15K SAS ?
Sorry but I can't conclude anything at the moment.
I have no Ssd here. But beside testing that's not a solution

[Gostev]

Yes, I mean the speed of your local hard drive is about 120 MB/s. I just tried to copy a file from SSD to a hard drive in my home computer, and I am getting just a little over 100 MB/s. This was in response to yours "if i forget the target and backup to the local backup disk I get the same speeds. That will say something I think".

[tfloor]

Anton,

Aha right I understand that
So the local harddrive isnt a good test because that has also around the Same speed
What can I do do test the several stages. I still don't know if the vnx 5300 or the proxy is the proble
But I know with low compression I have the same speed as optimal compression. So that would say the vnx.
But it's hard to imagine that that is a problem. So I want to know how to do some reliable tests. Between FC San and backup server.

[dellock6]

Tristan,
try to run the IO analyzer from VMware on the VNX. Deploy it as a virtual machine from here:

http://labs.vmware.com/flings/io-analyzer

Let's see what is going to come out. I'm too doubtful a VNX can do "only" 120mb/s, when also there is no potential limit by the FC fabric.

[tfloor]

Luca,

Great. Finally a nice thing to test the speed.
I've look at the tutorial. Looks like this will test the speed between a vm and a ESX host.
So that's cool. I don't know what's a THROUGPUT for one pair 4gb FC exactly, but I think that must be more than I see

Thanks. I will put the results here!

[dellock6]

Right, the tool tests the IO between the vm and the ESXi storage, but is a first method to understand the speed of the VNX. Let's say it will return "again" 120 mb/s, it could be the effective speed of the VNX.
About the theoretical limit of FC, assuming a 95% efficiency (that's the usual FC efficiency), in a 4 Gbit/s FC you can transfer 512 MB/s, at 95% is 486.4 MB/s limit. Way over the 120 MB/s you are seeing. (also, consider disk latency and I/O randomness).

[tfloor]

Thanks for the theory. No I know the theoretical speed of the data flow.
I can download and run the vm appliance tomorrow to test the vnx speed of the vm.
I will post the results here as soon as I have it.
Thanks

[tfloor]

Luca,

I've deployed the Vmware I/O Analyzer.
I've add a second harddisk to the vm, with 4GB (4 times the memory). And i thought i can migrate that HD to the storage LUN i want to test. (is that correct?), i've looked at different tutorials but don't see anything about that.
Further i have to specify "Workload Spec"( Dropdown button with a lot of choices). So which one do i have to choose? and how long do i have to run the test.
Can you tell me how to configure/run the tool?

Regards,
Tristan.

[dellock6]

Hi tristan,
sorry but honestly I've seen that tool on VMware but never had the opportunity to run ti, only seen some articles on internet about it. Consider "flings" from VMware are tools developed and realeased "as is", so there is no official support from VMware.
Anyway, there is a tutorial on the dedicated section of the VMware Community:
http://communities.vmware.com/community ... =documents

[tfloor]

Luca,

Then for me it's maybe still useless. There are so much options for the workload specs, and those are not specified in the tutorial.
I have done 1 test from 1 tutorial, and this is a results... Don't know what i can do with it (that's the annoying of all test), i can't do anything with it. Grrr



Is there anyone who can say something about it...

[dellock6]

Uhm, this is a start.
Try to push the storage to its limit by doing a workload with:
- 64k block size (VNX default size from EMC forums https://community.emc.com/thread/130731)
- 100% read (we are simulating Veeam activity, and it only reads from the san, does not writes)
- randomness try different values, this depends on how data are fragmented in the san. 100% would give you the ideal best perfomance, but is not so "real world"

[tfloor]

Luca,

I've follow your specs, and here are the results for the same lun where i do always a testjob (vm) with.



and



What do you think about this.
With my knowledge i see that the lun can do around 5000 IOPS at about 356 MegaByte per second.
Using only VMHBA2 and the rest is doing nothing.