NetApp Source as bottleneck

Availability for the Always-On Enterprise

Re: NetApp Source as bottleneck

Veeam Logoby Andreas Neufert » Mon Mar 30, 2015 7:30 am

Hi...

I saw a lot of storage systems that scale better by using multiple parallel streams instead of a single one.

Even I saw some SSD Storage Systems that operate on 300MB/s for a single VM disk but scale 10x or more when using parallel processing.

100MB/s is a fair throughput for a single disk (not that good but not that bad either).

Also important is that you measure throughput with Active Fulls, that's the only way to get real numbers. Incremental runs use random read and performance will varies.
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Mon Mar 30, 2015 2:38 pm

Thanks for all the feedback, I thought getting on the digest would get some more people involved! Ok, to try to address all the questions I've had:

  1. Yes, I've got a case open with NetApp. At this point, I am convinced it is a problem on their side.
  2. It is Active Full performance I am looking at. Incremental performance is not an issue.
  3. When sitting at 100MB/sec, the source is 99% the bottleneck still!
  4. The main array I'm testing is 48 900GB SAS disks with 512GB of flash cache.
  5. My FC network is at 8Gb and my Veeam has a 10Gb network to the ESXi hosts. Both run at 100MB/sec, there is no 1Gb network anywhere.
  6. Running HD Speed in VMware, I am also limited to 100MB/sec. So this isn't a Veeam issue!
  7. The big issue I see is, if I have a backup running two disks in parallel, I'll get 180-200MB/sec. When one disk finishes, the speed then drops to 100MB/sec. If it can do two disks at 200MB/sec, then why it can't it do one disk at something close to that speed?

See this image to show it appears in Veeam:

Image
lightsout
Expert
 
Posts: 185
Liked: 47 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby emachabert » Mon Mar 30, 2015 6:25 pm 1 person likes this post

I mainly deploy 3par arrays so I can't compare with many other netapp arrays, and I have a bunch of 7200 that look like similar to your config in terms of spindles.
For example, with a 7200 with 48*600g 10k, I can have a sustained rate of 840MB/s in active full using 4 streams (direct SAN 8gb) . I get arround 20minutes per terabyte.
I only have one customer with a FAS6220 (1tb flash cache and 270 spindles), never got more that 680MB/s whatever the number of streams...
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017
emachabert
Veeam Vanguard
 
Posts: 354
Liked: 163 times
Joined: Wed Nov 17, 2010 11:42 am
Location: France
Full Name: Eric Machabert

Re: NetApp Source as bottleneck

Veeam Logoby Gostev » Tue Mar 31, 2015 12:58 am

lightsout wrote:The big issue I see is, if I have a backup running two disks in parallel, I'll get 180-200MB/sec. When one disk finishes, the speed then drops to 100MB/sec. If it can do two disks at 200MB/sec, then why it can't it do one disk at something close to that speed?

Because the bottleneck is at source (actual storage, or connectivity to one) - and not somewhere in the Veeam engine. NetApp simply would not "give" us individual disk's data faster than 100 MB/s.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: NetApp Source as bottleneck

Veeam Logoby loelly » Tue Mar 31, 2015 8:36 am 2 people like this post

Welcome to the world of NetApp, where everything is slow, complex and expensive. Kidding aside: you will often see performance increases when using more streams. You will notice the same behavior when firing up a couple of VMs and running a dd/HP Speed against the array.
loelly
Enthusiast
 
Posts: 49
Liked: 10 times
Joined: Thu Apr 17, 2014 8:25 am
Full Name: Jens Siegmann

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Tue Mar 31, 2015 1:36 pm 1 person likes this post

Gostev wrote:Because the bottleneck is at source (actual storage, or connectivity to one) - and not somewhere in the Veeam engine. NetApp simply would not "give" us individual disk's data faster than 100 MB/s.

Yes, I agree this is not a Veeam issue, but the above is really the heart of my problem.

The reason I know this is an issue is we just migrated to a NetApp from an old HP EVA. The old EVA was able to push 200-300MB/sec per disk on the same infrastructure, so something on the NetApp side is just wrong.
lightsout
Expert
 
Posts: 185
Liked: 47 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby b.lagace » Wed Apr 01, 2015 9:58 am 1 person likes this post

Hey,

Are you 100% sure about your network.
Cause it really sounds like you're hitting a 1Gbit wall.

Put a file on a lun on the ESXI host and initiate the command from there to copy that file towards the C-drive of your Veeam Server.
Most disks are capable of doing more than 100MB write/sec so see what's going on.
Open your taskmanager and monitor you're network from ESXI point of view.

Regards,
Bjorn
b.lagace
Enthusiast
 
Posts: 31
Liked: 17 times
Joined: Wed Jul 02, 2014 1:23 pm
Full Name: Bjorn Lagace

Re: NetApp Source as bottleneck

Veeam Logoby RGijsen » Wed Apr 01, 2015 10:08 am

We aren't nearly as big to justify a Netapp. Our main storage is a HP MSA2040, with one 16x450GB 10krpm RAID6 volume and one 6x900GB 10krpm RAID6 volume. The first one holds regular filedata like homedrives, profiles and regular fileshares. The second volume holds the VM's (Hyper-V in our case). Our backup device is a HP P2000 with 16x 1TB 7200rpm in RAID5. The backup volume has 2012R2 dedup enabled. I am using an offhost proxy.

While a complete different setup, Hyper-V with HP entry-level SAN's and just 4Gbps fiber (although dual path with round robin - actual rate is about 600MB/sec) my full jobs sustain about 200-300MB/sec. That's with only one job running at a time in order not to congest the storage, but with three or four volumes at a time. The MSA2040 has 4GB cache (hardly usefull when reading), the P2000 has 2GB of cache but no addition flash-cache or anything. Our storage can't do dedupe by itself, but as said we DO have 2012R2 dedupe enabled. 2012R2 dedupe is post-process though, not inline. That means when writing data there is absolutely no impact of using 2012R2 dedupe.

I can't believe our puny HP MSA2040 performs so much better than your Netapp with additional flashboards. To be short I think your issue is, or atleast should not be the netapp. As stated before here, check your network topology, are there 1Gb links in there between your hypervisors and proxy and / or proxy and storage (if not on the same box)? Do you see high cpu load on specific processes on your Veeam box? If so, do you have compression or dedupe enabled in Veeam as well? If the latter is true, note that compression is not beneficial to dedupe, and deduping on three levels (netapp, 2012R2 and veeam) defeats the purpose of it somehow.
RGijsen
Enthusiast
 
Posts: 55
Liked: 4 times
Joined: Fri Oct 10, 2014 2:06 pm

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Wed Apr 01, 2015 1:29 pm

b.lagace wrote:Hey,

Are you 100% sure about your network.
Cause it really sounds like you're hitting a 1Gbit wall.

Put a file on a lun on the ESXI host and initiate the command from there to copy that file towards the C-drive of your Veeam Server.
Most disks are capable of doing more than 100MB write/sec so see what's going on.
Open your taskmanager and monitor you're network from ESXI point of view.

Regards,
Bjorn


I agree, but I've got two Veeam servers and two NetApp SANs on different networks and they behave the same. One NetApp SAN replaced a HP EVA SAN, and over the same FC network, it is slower after I used storage vMotion to migrate the machines over.

I ran HD speed as per here:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006821

It was interesting. The first pass ran at 100MB/sec. The 2nd pass ran at 300MB/sec, as it was now running from the NetApp flash cache. That to me suggests the network between the NetApp + VMware is capable of more than 100MB/sec, but there is something else (internal to the NetApp perhaps?!) that is limited to 100MB/sec.
lightsout
Expert
 
Posts: 185
Liked: 47 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby Andreas Neufert » Wed Apr 01, 2015 1:52 pm

Is the Proxy and Repository on same Veeam Server?
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Wed Apr 01, 2015 2:35 pm

Yes they are.
lightsout
Expert
 
Posts: 185
Liked: 47 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby Andreas Neufert » Wed Apr 01, 2015 2:59 pm 1 person likes this post

Maybe it do not help as the Source is the bottleneck, but we can give it a try.
On some installations I saw that the local windows stack together with some drivers and the implemented .net network drivers/filters reduced the possible speed between our components.
Please install v8 Patch 1 and create the following registry key on B&R Server and restart Veeam Backup Service:

DataMoverLocalFastPath (DWORD) registry value under HKLM\SOFTWARE\Veeam\Veeam Backup and Replication, and set it to the following values:
0: Default behavior (no optimizations)
1: Data exchange through TCP socket on the loopback interface (faster)
2: Data exchange through shared memory (fastest)

Use option 1 or 2

Again this maybe has 0 effect but wen can give it a try.

There is also an option that our support can test a special VMware Kit against the storage to see if a native VMware read is in the same throughput area than our software. Just to find out if Veeam isn´t the bottleneck. But you alread said that you had the same speed in other situations without Veeam.

As the others said. Work with the Storage Vendor on the general speed to VMware. If it increases, likely our software speed will increase as well.
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Wed Apr 01, 2015 7:52 pm

Thanks for the good idea. I gave it a try, and no difference.

Right now I'll let the case play out with NetApp. I may get VMware involved as well, as yes, once that is resolved I suspect it will resolve it for Veeam as well.
lightsout
Expert
 
Posts: 185
Liked: 47 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby Andreas Neufert » Wed Apr 01, 2015 8:04 pm

Thanks for the feedback. Please let us know if you find the root cause.
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: NetApp Source as bottleneck

Veeam Logoby vee789 » Mon Dec 14, 2015 3:28 pm

Any update on this case?
Would be great to learn something :)
vee789
Novice
 
Posts: 5
Liked: never
Joined: Mon Oct 12, 2015 2:53 pm

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: chrisit, nokogerra, v.Eremin and 54 guests