NetApp Source as bottleneck

Availability for the Always-On Enterprise

NetApp Source as bottleneck

Veeam Logoby lightsout » Sun Mar 22, 2015 11:54 pm

I've got a case open with NetApp on this, but just wanted to see if anyone else has the same issue.

NetApp - FAS 3220 with flash cache on 7-mode 8.2.3.
Veeam - Physical server on 2012 R2 with v8 patch 1. I don't have Enterprise Plus so these are not SAN snapshots, just normal VMware ones.

Connectivity is via 8Gb FC using direct SAN backups. I see the "[san]" tags so I know it is working.

Here is the issue, my per VMDK throughput is no more than 100MB/sec. If I run parallel disks at once, I can get 200-300MB/sec so the SAN, FC fabric, Backup Proxy, Repository, etc can all handle more throughput, but even in this case each disk is no faster than 100MB/sec. The bottleneck is always listed as the source with this 100MB/sec issue./

I don't have any throttling on the FC fabric, Veeam itself nor on the NetApp that I know of that could be causing it.

Has anyone else seen this issue, or is just me?
lightsout
Expert
 
Posts: 183
Liked: 47 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby Vitaliy S. » Mon Mar 23, 2015 10:31 am

Hello,

Do you have this job performance on full or incremental job pass?

Thank you!
Vitaliy S.
Veeam Software
 
Posts: 19450
Liked: 1092 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Mon Mar 23, 2015 2:15 pm

It happens on both types, although the effect is much more visible on the active full backups.
lightsout
Expert
 
Posts: 183
Liked: 47 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby Vitaliy S. » Mon Mar 23, 2015 2:29 pm

If source is a bottleneck, then you can try to update all the firmware and make sure you're using latest storage drivers. Probably other community members with the same configuration can chime into our conversation.
Vitaliy S.
Veeam Software
 
Posts: 19450
Liked: 1092 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Mon Mar 23, 2015 2:37 pm

That is what I am hoping for. :)

Looking at the bottleneck "source", is there any sort of details you can tell me from that? How does Veeam determine that is the bottleneck?
lightsout
Expert
 
Posts: 183
Liked: 47 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby Vitaliy S. » Mon Mar 23, 2015 2:59 pm 1 person likes this post

This is explained in the topic I've referenced above ;)
Vitaliy S.
Veeam Software
 
Posts: 19450
Liked: 1092 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: NetApp Source as bottleneck

Veeam Logoby emachabert » Tue Mar 24, 2015 5:55 pm 1 person likes this post

Do you have deduplication enabled on those volumes ?
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017
emachabert
Veeam Vanguard
 
Posts: 353
Liked: 163 times
Joined: Wed Nov 17, 2010 11:42 am
Location: France
Full Name: Eric Machabert

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Tue Mar 24, 2015 6:13 pm

Yes. I've got NetApp SIS on and I've got Windows 2012 R2 deduplication on too. :)

My Veeam server and ESXi servers are both on 10Gb. So I just tried NBD too, and that was also pegged to 100MB/sec.

Even when the disk is zeroed out and empty, and I'm getting all deduplicated data, it is still 100MB/sec.

So is the suggestion to turn SIS off on the NetApp?
lightsout
Expert
 
Posts: 183
Liked: 47 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby emachabert » Tue Mar 24, 2015 7:07 pm 1 person likes this post

Give it a try. Test a new LUN without dedupe (SIS) enabled and see if performance increase.
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017
emachabert
Veeam Vanguard
 
Posts: 353
Liked: 163 times
Joined: Wed Nov 17, 2010 11:42 am
Location: France
Full Name: Eric Machabert

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Fri Mar 27, 2015 7:00 pm

I tried two things:

1. New volume without SIS.
2. Turning "flash cache" off on the NetApp.

#1 made no difference, and #2 slowed it down (surprise!).
lightsout
Expert
 
Posts: 183
Liked: 47 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby emachabert » Fri Mar 27, 2015 10:35 pm

is the MPIO on the proxy configured as netapp recomend it to be ?
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017
emachabert
Veeam Vanguard
 
Posts: 353
Liked: 163 times
Joined: Wed Nov 17, 2010 11:42 am
Location: France
Full Name: Eric Machabert

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Sat Mar 28, 2015 2:37 pm

I was using the Microsoft MPIO driver, so I just installed the MPIO NetApp driver for 2012 R2. Confirmed AULA is on.

No change in performance. I'm not looked too hard, but running the NetApp Powershell cmdlets look like everything ok to me. I can see 4 active paths and they'll all showing activity.

Is there anything I should be looking for?

I'm thinking of just trying HotAdd to see if that suffers from the same speed. My main concern is that my VMware environment is also limited to 100MB/sec.
lightsout
Expert
 
Posts: 183
Liked: 47 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby emachabert » Sun Mar 29, 2015 9:40 pm

Pretty strange.
Looks like everything is setup correctly, did you check your SaN switch ? Like speed, error counters etc.
Did you open a support case with Netapp ?
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017
emachabert
Veeam Vanguard
 
Posts: 353
Liked: 163 times
Joined: Wed Nov 17, 2010 11:42 am
Location: France
Full Name: Eric Machabert

Re: NetApp Source as bottleneck

Veeam Logoby Butha » Mon Mar 30, 2015 5:25 am

Hi,

Also running Netapp - NFS though - but my first comment would be that 100MB/s is actually pretty good speed. Remember there are many things that happens wrt compressions/dedupe etc in Veeam that implies it will not come close to the actual speed that the san can do while for example doing file copies etc. (we are running higher end Netapps - 6250 that can sustain 400MB/s +) I often average out at around 30Mb on jobs.

With regards to bottlenecks - remember it will always identify one - if you hover on the bottleneck - what % is source? What I've seen before is that when using 2012 + dedupe is OS the "write" speed seems incredible because of high dedupe ratios - so the software seems to thing the source is the problem, as it can write at much faster speed because of de-dupe - so it's not "Real"..

Only thing that sortof stands out - is the 100MB/s - are you perhaps running 1Gig network switches? Remember that often the transport between source - proxy - target happens at network layer - so 1 Gig switch somewhere in the middle would cause that bottleneck. If you are running 100% Direct San mode you should be fine, but if any proxy is network enabled it might be your issue.

You could test 2 options - setup a VM and add some space to it and configure it as a repository in Veeam - also make sure you have a VM as a proxy on the same host as the source VM and the Virtual repository server - this will rule out any physical networking.

Not sure how much effort you would want to spend - but as an interesting test I would perhaps setup a quick VM with Veeam V7 on it - some repository space and see what happens if you write to that - That would compare the speed of the veeamagent.exe between versions to see if anything shows up or just stick

B
Butha
Enthusiast
 
Posts: 28
Liked: 11 times
Joined: Wed Oct 03, 2012 10:59 am
Full Name: Butha van der Merwe

Re: NetApp Source as bottleneck

Veeam Logoby HannesK » Mon Mar 30, 2015 6:22 am

Hi,
maybe the source simply is not faster?

I had the same situation at a customer with an other storage vendor where the customer complained that Veeam is so slow. After investigating the 99% source load we found out that his current backup software works at exactly the same speed as Veeam because the source storage was under heavy load and could not serve more than 80MByte/s.

You could try the following:
- create a LUN on the Netapp
- connect with a Linux box (I suggest Linux because I'm not aware of /dev/null on Windows)
- create a "some gigabyte" file
- copy the file from Netapp to /dev/null


200-300 MB/s with parallel processing sounds ok for a FAS 3220 for me. How many and which disks do you have?


Best regards,
Hannes
HannesK
Veeam Software
 
Posts: 1112
Liked: 140 times
Joined: Mon Sep 01, 2014 11:46 am

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google [Bot] and 15 guests