NetApp Source as bottleneck

[lightsout]

I've got a case open with NetApp on this, but just wanted to see if anyone else has the same issue.

NetApp - FAS 3220 with flash cache on 7-mode 8.2.3.
Veeam - Physical server on 2012 R2 with v8 patch 1. I don't have Enterprise Plus so these are not SAN snapshots, just normal VMware ones.

Connectivity is via 8Gb FC using direct SAN backups. I see the "[san]" tags so I know it is working.

Here is the issue, my per VMDK throughput is no more than 100MB/sec. If I run parallel disks at once, I can get 200-300MB/sec so the SAN, FC fabric, Backup Proxy, Repository, etc can all handle more throughput, but even in this case each disk is no faster than 100MB/sec. The bottleneck is always listed as the source with this 100MB/sec issue./

I don't have any throttling on the FC fabric, Veeam itself nor on the NetApp that I know of that could be causing it.

Has anyone else seen this issue, or is just me?

[Vitaliy S.]

Hello,

Do you have this job performance on full or incremental job pass?

Thank you!

[lightsout]

It happens on both types, although the effect is much more visible on the active full backups.

[Vitaliy S.]

If source is a bottleneck, then you can try to update all the firmware and make sure you're using latest storage drivers. Probably other community members with the same configuration can chime into our conversation.

[lightsout]

That is what I am hoping for.

Looking at the bottleneck "source", is there any sort of details you can tell me from that? How does Veeam determine that is the bottleneck?

[Vitaliy S.]

This is explained in the topic I've referenced above

[emachabert]

Do you have deduplication enabled on those volumes ?

[lightsout]

Yes. I've got NetApp SIS on and I've got Windows 2012 R2 deduplication on too.

My Veeam server and ESXi servers are both on 10Gb. So I just tried NBD too, and that was also pegged to 100MB/sec.

Even when the disk is zeroed out and empty, and I'm getting all deduplicated data, it is still 100MB/sec.

So is the suggestion to turn SIS off on the NetApp?

[emachabert]

Give it a try. Test a new LUN without dedupe (SIS) enabled and see if performance increase.

[lightsout]

I tried two things:

1. New volume without SIS.
2. Turning "flash cache" off on the NetApp.

#1 made no difference, and #2 slowed it down (surprise!).

[emachabert]

is the MPIO on the proxy configured as netapp recomend it to be ?

[lightsout]

I was using the Microsoft MPIO driver, so I just installed the MPIO NetApp driver for 2012 R2. Confirmed AULA is on.

No change in performance. I'm not looked too hard, but running the NetApp Powershell cmdlets look like everything ok to me. I can see 4 active paths and they'll all showing activity.

Is there anything I should be looking for?

I'm thinking of just trying HotAdd to see if that suffers from the same speed. My main concern is that my VMware environment is also limited to 100MB/sec.

[emachabert]

Pretty strange.
Looks like everything is setup correctly, did you check your SaN switch ? Like speed, error counters etc.
Did you open a support case with Netapp ?

[Butha]

Hi,

Also running Netapp - NFS though - but my first comment would be that 100MB/s is actually pretty good speed. Remember there are many things that happens wrt compressions/dedupe etc in Veeam that implies it will not come close to the actual speed that the san can do while for example doing file copies etc. (we are running higher end Netapps - 6250 that can sustain 400MB/s +) I often average out at around 30Mb on jobs.

With regards to bottlenecks - remember it will always identify one - if you hover on the bottleneck - what % is source? What I've seen before is that when using 2012 + dedupe is OS the "write" speed seems incredible because of high dedupe ratios - so the software seems to thing the source is the problem, as it can write at much faster speed because of de-dupe - so it's not "Real"..

Only thing that sortof stands out - is the 100MB/s - are you perhaps running 1Gig network switches? Remember that often the transport between source - proxy - target happens at network layer - so 1 Gig switch somewhere in the middle would cause that bottleneck. If you are running 100% Direct San mode you should be fine, but if any proxy is network enabled it might be your issue.

You could test 2 options - setup a VM and add some space to it and configure it as a repository in Veeam - also make sure you have a VM as a proxy on the same host as the source VM and the Virtual repository server - this will rule out any physical networking.

Not sure how much effort you would want to spend - but as an interesting test I would perhaps setup a quick VM with Veeam V7 on it - some repository space and see what happens if you write to that - That would compare the speed of the veeamagent.exe between versions to see if anything shows up or just stick

B

[HannesK]

Hi,
maybe the source simply is not faster?

I had the same situation at a customer with an other storage vendor where the customer complained that Veeam is so slow. After investigating the 99% source load we found out that his current backup software works at exactly the same speed as Veeam because the source storage was under heavy load and could not serve more than 80MByte/s.

You could try the following:
- create a LUN on the Netapp
- connect with a Linux box (I suggest Linux because I'm not aware of /dev/null on Windows)
- create a "some gigabyte" file
- copy the file from Netapp to /dev/null


200-300 MB/s with parallel processing sounds ok for a FAS 3220 for me. How many and which disks do you have?


Best regards,
Hannes

[Andreas Neufert]

Hi...

I saw a lot of storage systems that scale better by using multiple parallel streams instead of a single one.

Even I saw some SSD Storage Systems that operate on 300MB/s for a single VM disk but scale 10x or more when using parallel processing.

100MB/s is a fair throughput for a single disk (not that good but not that bad either).

Also important is that you measure throughput with Active Fulls, that's the only way to get real numbers. Incremental runs use random read and performance will varies.

[lightsout]

Thanks for all the feedback, I thought getting on the digest would get some more people involved! Ok, to try to address all the questions I've had:

1. Yes, I've got a case open with NetApp. At this point, I am convinced it is a problem on their side.
2. It is Active Full performance I am looking at. Incremental performance is not an issue.
3. When sitting at 100MB/sec, the source is 99% the bottleneck still!
4. The main array I'm testing is 48 900GB SAS disks with 512GB of flash cache.
5. My FC network is at 8Gb and my Veeam has a 10Gb network to the ESXi hosts. Both run at 100MB/sec, there is no 1Gb network anywhere.
6. Running HD Speed in VMware, I am also limited to 100MB/sec. So this isn't a Veeam issue!
7. The big issue I see is, if I have a backup running two disks in parallel, I'll get 180-200MB/sec. When one disk finishes, the speed then drops to 100MB/sec. If it can do two disks at 200MB/sec, then why it can't it do one disk at something close to that speed?

See this image to show it appears in Veeam:

[emachabert]

I mainly deploy 3par arrays so I can't compare with many other netapp arrays, and I have a bunch of 7200 that look like similar to your config in terms of spindles.
For example, with a 7200 with 48*600g 10k, I can have a sustained rate of 840MB/s in active full using 4 streams (direct SAN 8gb) . I get arround 20minutes per terabyte.
I only have one customer with a FAS6220 (1tb flash cache and 270 spindles), never got more that 680MB/s whatever the number of streams...

[Gostev]

The big issue I see is, if I have a backup running two disks in parallel, I'll get 180-200MB/sec. When one disk finishes, the speed then drops to 100MB/sec. If it can do two disks at 200MB/sec, then why it can't it do one disk at something close to that speed?

Because the bottleneck is at source (actual storage, or connectivity to one) - and not somewhere in the Veeam engine. NetApp simply would not "give" us individual disk's data faster than 100 MB/s.

[loelly]

Welcome to the world of NetApp, where everything is slow, complex and expensive. Kidding aside: you will often see performance increases when using more streams. You will notice the same behavior when firing up a couple of VMs and running a dd/HP Speed against the array.

[lightsout]

Because the bottleneck is at source (actual storage, or connectivity to one) - and not somewhere in the Veeam engine. NetApp simply would not "give" us individual disk's data faster than 100 MB/s.

Yes, I agree this is not a Veeam issue, but the above is really the heart of my problem.

The reason I know this is an issue is we just migrated to a NetApp from an old HP EVA. The old EVA was able to push 200-300MB/sec per disk on the same infrastructure, so something on the NetApp side is just wrong.

[b.lagace]

Hey,

Are you 100% sure about your network.
Cause it really sounds like you're hitting a 1Gbit wall.

Put a file on a lun on the ESXI host and initiate the command from there to copy that file towards the C-drive of your Veeam Server.
Most disks are capable of doing more than 100MB write/sec so see what's going on.
Open your taskmanager and monitor you're network from ESXI point of view.

Regards,
Bjorn

[RGijsen]

We aren't nearly as big to justify a Netapp. Our main storage is a HP MSA2040, with one 16x450GB 10krpm RAID6 volume and one 6x900GB 10krpm RAID6 volume. The first one holds regular filedata like homedrives, profiles and regular fileshares. The second volume holds the VM's (Hyper-V in our case). Our backup device is a HP P2000 with 16x 1TB 7200rpm in RAID5. The backup volume has 2012R2 dedup enabled. I am using an offhost proxy.

While a complete different setup, Hyper-V with HP entry-level SAN's and just 4Gbps fiber (although dual path with round robin - actual rate is about 600MB/sec) my full jobs sustain about 200-300MB/sec. That's with only one job running at a time in order not to congest the storage, but with three or four volumes at a time. The MSA2040 has 4GB cache (hardly usefull when reading), the P2000 has 2GB of cache but no addition flash-cache or anything. Our storage can't do dedupe by itself, but as said we DO have 2012R2 dedupe enabled. 2012R2 dedupe is post-process though, not inline. That means when writing data there is absolutely no impact of using 2012R2 dedupe.

I can't believe our puny HP MSA2040 performs so much better than your Netapp with additional flashboards. To be short I think your issue is, or atleast should not be the netapp. As stated before here, check your network topology, are there 1Gb links in there between your hypervisors and proxy and / or proxy and storage (if not on the same box)? Do you see high cpu load on specific processes on your Veeam box? If so, do you have compression or dedupe enabled in Veeam as well? If the latter is true, note that compression is not beneficial to dedupe, and deduping on three levels (netapp, 2012R2 and veeam) defeats the purpose of it somehow.

[lightsout]

Hey,

Are you 100% sure about your network.
Cause it really sounds like you're hitting a 1Gbit wall.

Put a file on a lun on the ESXI host and initiate the command from there to copy that file towards the C-drive of your Veeam Server.
Most disks are capable of doing more than 100MB write/sec so see what's going on.
Open your taskmanager and monitor you're network from ESXI point of view.

Regards,
Bjorn

I agree, but I've got two Veeam servers and two NetApp SANs on different networks and they behave the same. One NetApp SAN replaced a HP EVA SAN, and over the same FC network, it is slower after I used storage vMotion to migrate the machines over.

I ran HD speed as per here:

http://kb.vmware.com/selfservice/micros ... Id=1006821

It was interesting. The first pass ran at 100MB/sec. The 2nd pass ran at 300MB/sec, as it was now running from the NetApp flash cache. That to me suggests the network between the NetApp + VMware is capable of more than 100MB/sec, but there is something else (internal to the NetApp perhaps?!) that is limited to 100MB/sec.

[Andreas Neufert]

Is the Proxy and Repository on same Veeam Server?

[lightsout]

Yes they are.

[Andreas Neufert]

Maybe it do not help as the Source is the bottleneck, but we can give it a try.
On some installations I saw that the local windows stack together with some drivers and the implemented .net network drivers/filters reduced the possible speed between our components.
Please install v8 Patch 1 and create the following registry key on B&R Server and restart Veeam Backup Service:

DataMoverLocalFastPath (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to the following values:
0: Default behavior (no optimizations)
1: Data exchange through TCP socket on the loopback interface (faster)
2: Data exchange through shared memory (fastest)

Use option 1 or 2

Again this maybe has 0 effect but wen can give it a try.

There is also an option that our support can test a special VMware Kit against the storage to see if a native VMware read is in the same throughput area than our software. Just to find out if Veeam isn´t the bottleneck. But you alread said that you had the same speed in other situations without Veeam.

As the others said. Work with the Storage Vendor on the general speed to VMware. If it increases, likely our software speed will increase as well.

[lightsout]

Thanks for the good idea. I gave it a try, and no difference.

Right now I'll let the case play out with NetApp. I may get VMware involved as well, as yes, once that is resolved I suspect it will resolve it for Veeam as well.

[Andreas Neufert]

Thanks for the feedback. Please let us know if you find the root cause.

[vee789]

Any update on this case?
Would be great to learn something