NetAPP backup performance

[bbibber]

Did you consider something like PernixData?

Sincerely
Guy

[vee789]

Any update on this case?
Could you find the real cause for the slow backup?
Would also be interesting so see if the 8xxx head made any difference...?

[larry]

I have two sites with FAS2552 8.3.1P1, NFS 4.1, VMware 6. I do have the SSD cache and all SAS disks in the FAS2552s. My backup speeds are very good. Doing a full backup I see between 200 – 500 MBS. I have seen as little as 70 meg running backups during the day but the bottle neck was the destination. When I changed CBT off because of VM 6 bug I would see reads at 700 MPS. Which was mostly reads. I have 10 gig using Nexus 3K switches. My numbers were so different that I felt I needed to post. I even have some volumes with dedupe and compression on. My dedupe numbers are over 50 percent which is better than sales told me they would be.

[daniel.negru]

Upgrading to FAS 8000 helped very little. At least the FAS CPUs are not hammered and not a bottleneck after, but the speeds are still under a DELL MD toy iSCSI storage. Single streams linger somewhere around 60-100 MB while MD is capable of double of that. Not a disaster, just a big disappointment.

NetAPP employee finally said is a known issue and it is normal for single stream reads to be slow on FAS. Too bad I had no call recording to have it on file.
That makes Larry's experience very puzzling... I tried NFS as well while on FAS 2552, speeds were identical with what I have on iSCSI.
From what Larry says, either NFS + ESXi 6 is magic or I have a big issue somewhere that I, Veeam and several NetAPP gurus @ support failed in see during months and months of troubleshooting.

I do use dedupe on some special volumes, no compression. Still dedupe after the fact (post) is not the best. I see up to 87% reduction on one of the volumes (ex: 12 TB saved) but that has nothing to do with backup speeds anyways.

I already did put this issue behind me. It is what it is and next NAS/SAN refresh cycle, if I am still around, it will be what it will be.

[jbennett]

Here's my late addition

We have a 72x 10k NLSAS disks FAS2240 in 7-Mode with HA running and we're using only the dual 8Gbps fibre channel connectivity.

It was painfully slow with our Hyper-V 2012 cluster. We're migrating the hosts to VMware Cluster with the latest version ESXi. We're seeing much better write performance than we're used too seeing. Note, we're not using dedupe and cut down to just one main VMware LUN per proc for our cluster - it has seemed to help.



I feel your pain daniel - we were left for months to fend for ourselves during initial configuration of this system, since Hyper-V wasn't their support's customer demographic at the time. I cringe when I hear NetApp's name muttered - like a beaten dog. I've come to the same conclusion of just planning on replacing it at the next interval.

[jveerd1]

For all NetApp customers, please deploy NetApp Harvest and Graphite to gain complete visibility into your NetApp systems. Once we looked at the Graphite data it was pretty clear the NetApp systems of our customer were running at their maximum performance capabilities during the backup window. True, the performance is still not great, but at least now we know what the root cause is.

[acidpaintball]

I am so sorry but I did not see an email claiming there was a reply to this post. CPU CoStop is checked in your VMware environment. You can do it via command line, or in the VSphere client. You can also check for disk read/write latency from the same page(just click advanced and change data sets).

Update on the SAN shopping situation:

After getting all the quotes in from NetApp, Pure, and Nimble, we were so impressed with Nimbles quote that we decided to go with a bigger controller. Instead of the CS300(30k IOPS), we decided to go with the CS500(90k IOPS) instead. This unit has 2.4TB flash and 36TB raw storage.

We also were able to reach out to a few existing Nimble customers(Not recommended by our Nimble Sales team). We found them via various forums and reached out through private messages to set up calls. We got some very good information. I will share one of their stories.

An IT head at a company in Boston, MA was a NetApp shop for 6-7 years and started SAN shopping last year. He ended up doing a POC on 2 devices. The first was an All Flash NetApp array. The second was a Nimble CS400. He stated that the Nimble CS400 was literally 4 times faster than the All Flash Netapp array and the decision to go with Nimble over Netapp became a no brainer...especially as the cost was significantly cheaper. He ultimately purchased the CS500 which has been in production with zero issues since late last summer.

We also asked about having multiple plugins into VSphere seeing how we are adding to our environment and not removing the NetApp at this time and he was doing the same thing without issue. This something the Nimble team said "should not" be a problem but that they were not sure and would get back to us. We will be starting our POC the first week in May. We will also be doing a local replication to our second CS500 and once done, will ship it to our London office.

I will keep you all posted on our progress as soon as the POC starts. My goal is to put so much info on here, that anyone looking for info will get their fill. I will also be providing real time data benchmark results from our production environment for everything(VM boot time. SQL, Exchange, disk read/write latency, CPU Co-Stop, etc...) from both our current NetApp and our new Nimble CS500. If any of you want more info, please feel free to PM me.

My word. You actually picked a Nimble? We have the same device in-house and we have been regretting the purchase from day one. Backups are slow, device is slow, writes are extremely slow, and we have massive lag. Our nimble is running about 100% cache all the time @ 25+ms lag. Backups take hours to perform too. The device is good for read speeds IF the data lives in the cache pool only. If you do any writes the device fails at this. It uses 7200 rpm drives for its backend drives.

Exchange will pause from time to time and SQL... I dont even want to talk about that. Lets just say my ESXTOP shows wait time for the datastores a loooong time. My UCS is typically waiting for the stores to perform the IO before it moves on. I have 136 running VM's on a 7 blade UCS system.

Veeam - when it starts a backup the VM pauses for a very long time and once the backup is completed the clean up process can take hours on a SQL server release.

Conclusion.. Nimble is terrible, which it isnt near end of life yet and we are looking to purchase a new device from EMC or Netapp to replace it..

[markangus]

acidpaintball - Conclusion.. Nimble is terrible, which it isnt near end of life yet and we are looking to purchase a new device from EMC or Netapp to replace it..

I have to give my two cents... It seems like your system is over worked or hardware issue, IMO. I had some issues early on with my backups and found out one of my switches was defective, drove me nuts for a couple months trying to pinpoint it. VMware was taking 20 hours to release my SQL server snapshot. I replaced the switch and couldn't be happier with my Nimble and Veeam setup. I have been running Nimble for just under a year. I went back and forth on Nimble and an EMC. Best decision in terms of IT hardware I have made. I have worked with IBM, Compellent, HP, and Dell storage prior to Nimble. Nimble blows them away in performance and ease of use. I currently work at a smaller shop, we use Veeam to backup our 15 servers in under 2 hours, just over 2.5TB of data . The bottleneck for us is the QNap as we don't have enough spindles in it to keep up, planning on adding more later this year. I think we would get it under 1 hour. We are talking 2.5TB of data on those servers running, two SQL databases, Exchange, SharePoint, and various other applications. We then replicate it over to an offsite location using Veeam and it works like a charm. The ease of use for Veaam and Nimble for backups and recovery is outstanding. It is exactly what I need for an IT manager wearing multiple hats.

[Gostev]

I must say I was quite surprised with the above feedback as well, simply because I've never heard a single negative feedback from a Nimble customer before

[vbakalov]

Hello, we have been running Nimble for 4.5 years. We have 4 Nimble C440 and for the backup we use Nimble CS260.
We are running 200+ servers on one Nimble CS440 15-25000 IOPs no sweat at all. We use the other one to replicate. Same for the VDI.
VDI 900 Desktops on another CS440 5000 IOPs . Desktops are always on and I reboot 200+ desktops and no one notice it. Nimble spike to 70K IOPs for 1 min.
We are running VEEAM for backup on Nimble CS260. All 10GB connections. Same storage is use for Video recording.
You have a hardware issue. If you want to stay in business and not go broke this is the storage you want to have.

[vbakalov]

You should buy the NetApp and in about a year you will come back to Nimble to get a replacement for the NetApp. Same thing happen to another department where I work. They did not listen and got NetApp and now the NetApp is host, and they are looking to get a Nimble.

[Encode7]

Conclusion.. Nimble is terrible, which it isnt near end of life yet and we are looking to purchase a new device from EMC or Netapp to replace it..

Nimble is brilliant. If you are getting 100% cache utilisation from day 1 it wasn't sized properly - it doesn't have enough SSD cache & you need to upgrade it to have more.

Being mis-specced from day 1 is pretty much inexcusable so talk to the original supplier (they should have offered a proof of concept or worked with you to understand your workload to size it appropriately) & if not a positive outcome talk to Nimble direct.

Regardless of hardware choice, a mis-specced solution will always cause issues, which is why I often highlight the importance of vendor choice, and why it also pays to form a relationship with the manufacturer (who usually have system engineers) and not just the vendor.

I've seen this happen with manufacturers and suppliers of all sizes, of all partner status levels, old and new.

It is worst for those systems that can't be upgraded in the field and to fix requires replacing, so you have to be even more careful when sizing those. You are fortunate that Nimble is one that can be upgraded with no downtime in any direction.

[cbrasga]

@acidpaintball - Did you contact Nimble support (which is fantastic BTW), what did they say? Your experience definitely sounds like a very rare exception considering the 99% positive feedback I've received in doing my research both online and in person, and from my stellar experience with our Nimble CS300 units (operating at around 50% of cache).

Obviously if you're running at 100% on any system you're going to have issues regardless of the product, whether it's NetApp or EMC. It sounds like your system wasn't spec'd properly, not really a problem with the Nimble product/technology.

[rollergirl]

For all NetApp customers, please deploy NetApp Harvest and Graphite to gain complete visibility into your NetApp systems. Once we looked at the Graphite data it was pretty clear the NetApp systems of our customer were running at their maximum performance capabilities during the backup window. True, the performance is still not great, but at least now we know what the root cause is.

Why does Netapp need this? It seems netapp doesn't have the necessary tools to monitor the system properly and relies on open source tools. Who supports Harvest and Graphite? What are the requirements? Do I need additional tools beyond these two?

Wow!

[enoyksir]

@acidpaintball, I agree with some of the users that have stated if you don’t have your solution properly sized, another manufacturer won’t matter. I’ve had two arrays running two environments for the past two years without issues. I replaced an EMC VNX with the CS260 series in both locations. Before purchasing the arrays I visited customers with the arrays and performed some performance testing and was impressed with Nimble’s performance. In my own environment, which is now undersized because of the change in our business, I’ve yet to experience any performance hit, in fact when I switched SANs, I didn’t tell the end users and practically all of them noticed improved performance.

Before wasting time and efforts on looking at another solution you should reach out to Nimble Support, and use InfoSight to properly size your array.

[norm.oneal]

@acidpaintball I think these guys have been being nice to you. The information you posted about Nimble is as far from being true as anything I have ever read. We have deployed Nimble arrays all over the midwest with nothing but happy customers. We have downsized data centers, performed SAN consolidation by several racks in some cases and improved performance at every customer site. Please get with the Nimble Channel & or Support Team, they are second to none as well....-Norm O'Neal