NetAPP backup performance

VMware specific discussions

Re: NetAPP backup performance

Veeam Logoby NightBird » Wed Mar 11, 2015 6:44 pm

Just a little question, if you were happy with your CX3-20, why did you choose netapp instead of a EMC VNX box ?
NightBird
Service Provider
 
Posts: 148
Liked: 20 times
Joined: Tue Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE

Re: NetAPP backup performance

Veeam Logoby daniel.negru » Wed Mar 11, 2015 7:55 pm

Why?
1. my genuine stupidity I believe.
2. actually I have asked the solution provider for a VNX quote. Twice. Somehow he even forgot to acknowledge my request and I let it go being under the impression it may not fit our budget. I though EMC is generally more expensive than others. Plus I was aiming for DeDupe and NetAPP seems to be better, or so some sites claim.

In the end, why: point 1 is probably 80% of the reason.

Daniel.
daniel.negru
Influencer
 
Posts: 17
Liked: 1 time
Joined: Wed Feb 13, 2013 5:36 pm
Full Name: Daniel Negru

Re: NetAPP backup performance

Veeam Logoby Delo123 » Wed Mar 11, 2015 8:40 pm

Oh boy....
I would seriously get that Netapp sales guy over.... :(
If they are not able to do Dedupe the right way (inline) they shouldn't offer it at all for primary storage...

Alternativly get EMC in and let them buy your Netapp stuff and sell you something else with a good discount...
Delo123
Expert
 
Posts: 348
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: NetAPP backup performance

Veeam Logoby daniel.negru » Wed Mar 11, 2015 9:34 pm

Again, I am looking if anyone has similar stories.
I am trying to convince myself it must be something else in the environment at hand. So far only Jon has com forward with a similar story though not identical as his problems were SATA performance related.

Currently I would not bash NetAPP just yet. They come forward, involving some of their resources to troubleshoot what is happening. They assured me what I experience is not typical for their users and in the following days or weeks I will be troubleshooting this with their support. It may be something in the ESXi environment or who knows what else. so far no one was able to find the smoking gun and I have exhausted all my 'google engineer' knowledge.

If this will end up in just 'take it as it is' attitude / there is nothing that can be done, I will be super mad but for the time being I am still hopeful.

My general feeling is that I should have gone for 1 extra SAS shelf and for the FAS 8000 series... but that ship has sailed, no budget now.

Thank you,
Daniel.
daniel.negru
Influencer
 
Posts: 17
Liked: 1 time
Joined: Wed Feb 13, 2013 5:36 pm
Full Name: Daniel Negru

Re: NetAPP backup performance

Veeam Logoby Delo123 » Mon Mar 16, 2015 8:14 am

Hi Daniel,

Please do keep us updated...
Hope you will find a solution
Delo123
Expert
 
Posts: 348
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: NetAPP backup performance

Veeam Logoby Joshue » Mon Mar 16, 2015 9:13 am

Hi.

We are in the process of configuring a FAS2552 (20x900GB SAS, 4x200GB SSD) with 10Gb NFS (we were told that the performarce would be better than with iSCSI), Clustered Data ONTAP 8.3 and Dedup, for our Vmware 5.1

I'm a little scared..., previously we were using Nexenta with iSCSI+SATA+SSD and the performance (Vmware and Veeam) was sufficient for our needs.
Joshue
Lurker
 
Posts: 1
Liked: never
Joined: Tue Dec 18, 2012 5:47 pm
Full Name: Joshue Martin

Re: NetAPP backup performance

Veeam Logoby jveerd1 » Mon Mar 16, 2015 1:25 pm

We are troubleshooting a NetApp 3250 8.2 cDOT backup performance issue with 10G. We have Nexus 5000 switches and some C7000 enclosures. Storage integration is enabled on the backup jobs. Veeam throttling kicks in as soon as the jobs start, because latency increases on the NetApp.
Expected backup throughput is 200+ MB/s, but we rarely see half.
Please keep this thread updated with your findings.
jveerd1
Service Provider
 
Posts: 35
Liked: 8 times
Joined: Tue Mar 12, 2013 9:12 am
Full Name: Joeri van Eerd

Re: NetAPP backup performance

Veeam Logoby jb1095 » Mon Mar 16, 2015 2:17 pm

Can you please check vSphere for your disk read & write latency while the backup jobs are running? Specifically during snapshot creation and removal? From what I understand, anything over 15ms on either is very bad.
jb1095
Enthusiast
 
Posts: 35
Liked: 10 times
Joined: Tue Mar 03, 2015 9:32 pm
Full Name: Jon Brite

Re: NetAPP backup performance

Veeam Logoby daniel.negru » Fri Mar 20, 2015 4:41 pm

Hi everyone,

Sorry for missing in the thread, I'm on vacation.
No real updates so far, we wait for a meeting to figure out next course of action.

Joshue: I have seen no real difference with NFS versus iSCSI, we tested to see if the bottleneck may be iSCSI initiator. Yes, iSCSI is another layer over file system, NFS is a more direct access to the file system, so NFS should be at least as fast if not remotely faster. Depending on your needs and how you configure your aggregates, it may be enough for you. I don't think performance wise C mode versus 7 mode is any better.
I would be curious of your experience with it. Your setup seem to be smaller then mine but fairly similar. Please come back with your findings.
What are your IOPs you require and read/write ratio? Any idea?

I will check the latency next time backups kick in. From memory I remember seeing latency easily going to double digits during intensive IO and the CPU on FAS going to 90%-100%.
daniel.negru
Influencer
 
Posts: 17
Liked: 1 time
Joined: Wed Feb 13, 2013 5:36 pm
Full Name: Daniel Negru

Re: NetAPP backup performance

Veeam Logoby jb1095 » Fri Mar 20, 2015 4:55 pm

jveerd1 wrote:We are troubleshooting a NetApp 3250 8.2 cDOT backup performance issue with 10G. We have Nexus 5000 switches and some C7000 enclosures. Storage integration is enabled on the backup jobs. Veeam throttling kicks in as soon as the jobs start, because latency increases on the NetApp.
Expected backup throughput is 200+ MB/s, but we rarely see half.
Please keep this thread updated with your findings.

The only time we see over 100MB/s on our NetAPP is on a few small incremental. The fulls usually range betwwen 35MB/s and 70MB/s and we are full 10GBE everywhere....we are just waiting to get a POC from Nimble & Pure at this point and will see how they stack up(which I bet a years salary will be twice as fast at least) in terms of backup speed, and overall Exchange & SQL performance. Good luck with your meeting today Daniel, hopefully they find something mis-configured and you won't have to dive deep into a 58756325679 mile long rabbit hole just to find out your NetApp was undersized from the get go and your only option is a total forklife.
jb1095
Enthusiast
 
Posts: 35
Liked: 10 times
Joined: Tue Mar 03, 2015 9:32 pm
Full Name: Jon Brite

Re: NetAPP backup performance

Veeam Logoby Delo123 » Mon Mar 23, 2015 11:25 am 1 person likes this post

JB, please keep us updated on Nimble / Pure. I guess everybody here is interessted in Backup / Restore performance on those babies... :)
Delo123
Expert
 
Posts: 348
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: NetAPP backup performance

Veeam Logoby jb1095 » Tue Mar 24, 2015 2:54 pm 1 person likes this post

Delo123 wrote:JB, please keep us updated on Nimble / Pure. I guess everybody here is interessted in Backup / Restore performance on those babies... :)

Oh I definitely will. I can't wait to get our POC devices in house to test them out. The key will be to test them under normal load we see every day. I will be sure to post the results as we have a full history over the past year of our servers performance.
jb1095
Enthusiast
 
Posts: 35
Liked: 10 times
Joined: Tue Mar 03, 2015 9:32 pm
Full Name: Jon Brite

Re: NetAPP backup performance

Veeam Logoby nbctcp » Tue Mar 24, 2015 7:06 pm

INFO:
-NetApp FAS3240 SAS 72 disks
-UCS B 96GB RAM
-Nexus 5K+2K
-Citrix XenServer 6.2
-Citrix XenDesktop 7.1

PROBLEM: slow performance of my XenDesktop+XenApp
App icon become generic after sometime, so I can't launch app in RDS

VM boot improved from 20s to 12s after change vm format in NetApp from XenServer to Linux
Because I use SAN, so I didn't enable jumbo frame in Citrix PVS and MS RDS

After reading this thread, I start to suspect NetApp is the source of the problem.
But I am no longer working to my previous company since May last year.
I don't know the status now

My tech lead and boss following FlexPod 100%.
After test myself, IMO FlexPod only accurate in Cisco part not the rest

QUESTIONS:
1. How do you pinpoint NetApp is the culprit.
You mention CPU CoStop. Is that NetApp command?
Do you run iperf in Windows server? If yes, how fast IO do you expect?

tq
nbctcp
Lurker
 
Posts: 2
Liked: never
Joined: Fri Jul 30, 2010 3:49 am
Full Name: Nawir

Re: NetAPP backup performance

Veeam Logoby jb1095 » Wed Apr 08, 2015 3:26 pm 1 person likes this post

nbctcp wrote:INFO:
-NetApp FAS3240 SAS 72 disks
-UCS B 96GB RAM
-Nexus 5K+2K
-Citrix XenServer 6.2
-Citrix XenDesktop 7.1

PROBLEM: slow performance of my XenDesktop+XenApp
App icon become generic after sometime, so I can't launch app in RDS

VM boot improved from 20s to 12s after change vm format in NetApp from XenServer to Linux
Because I use SAN, so I didn't enable jumbo frame in Citrix PVS and MS RDS

After reading this thread, I start to suspect NetApp is the source of the problem.
But I am no longer working to my previous company since May last year.
I don't know the status now

My tech lead and boss following FlexPod 100%.
After test myself, IMO FlexPod only accurate in Cisco part not the rest

QUESTIONS:
1. How do you pinpoint NetApp is the culprit.
You mention CPU CoStop. Is that NetApp command?
Do you run iperf in Windows server? If yes, how fast IO do you expect?

tq



I am so sorry but I did not see an email claiming there was a reply to this post. CPU CoStop is checked in your VMware environment. You can do it via command line, or in the VSphere client. You can also check for disk read/write latency from the same page(just click advanced and change data sets).

Update on the SAN shopping situation:

After getting all the quotes in from NetApp, Pure, and Nimble, we were so impressed with Nimbles quote that we decided to go with a bigger controller. Instead of the CS300(30k IOPS), we decided to go with the CS500(90k IOPS) instead. This unit has 2.4TB flash and 36TB raw storage.

We also were able to reach out to a few existing Nimble customers(Not recommended by our Nimble Sales team). We found them via various forums and reached out through private messages to set up calls. We got some very good information. I will share one of their stories.

An IT head at a company in Boston, MA was a NetApp shop for 6-7 years and started SAN shopping last year. He ended up doing a POC on 2 devices. The first was an All Flash NetApp array. The second was a Nimble CS400. He stated that the Nimble CS400 was literally 4 times faster than the All Flash Netapp array and the decision to go with Nimble over Netapp became a no brainer...especially as the cost was significantly cheaper. He ultimately purchased the CS500 which has been in production with zero issues since late last summer.

We also asked about having multiple plugins into VSphere seeing how we are adding to our environment and not removing the NetApp at this time and he was doing the same thing without issue. This something the Nimble team said "should not" be a problem but that they were not sure and would get back to us. We will be starting our POC the first week in May. We will also be doing a local replication to our second CS500 and once done, will ship it to our London office.

I will keep you all posted on our progress as soon as the POC starts. My goal is to put so much info on here, that anyone looking for info will get their fill. I will also be providing real time data benchmark results from our production environment for everything(VM boot time. SQL, Exchange, disk read/write latency, CPU Co-Stop, etc...) from both our current NetApp and our new Nimble CS500. If any of you want more info, please feel free to PM me.
jb1095
Enthusiast
 
Posts: 35
Liked: 10 times
Joined: Tue Mar 03, 2015 9:32 pm
Full Name: Jon Brite

Re: NetAPP backup performance

Veeam Logoby daniel.negru » Mon Apr 13, 2015 3:29 pm

Hi everyone,

So far the news for me: it is obvious the CPUs on 25xx controllers are the bottleneck, we are asked to fork some more $ for 8000 series controllers, we are in the process of begging for budget and/or cut into other budget items to make this one fly, it will take about a month I believe.

The solution was supposed to give 10k iops easily and it seems that depending of the nature of IOPs, it crumbles under only 2 to 3 k iops/controller with high latency.
Since no one can explain why that is, I am reluctantly optimistic of the upgrade outcome.
daniel.negru
Influencer
 
Posts: 17
Liked: 1 time
Joined: Wed Feb 13, 2013 5:36 pm
Full Name: Daniel Negru

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: Bing [Bot] and 7 guests