Comprehensive data protection for all workloads
lightsout
Expert
Posts: 227
Liked: 62 times
Joined: Apr 10, 2014 4:13 pm
Contact:

Re: NetApp Source as bottleneck

Post by lightsout »

No, I never got to the bottom of it to be honest. However it remains on my list, but I want to do some performance testing to see if my VMs can pull more than 100MB/sec.

Changes to ESXi 6 hasn't made any difference.
lightsout
Expert
Posts: 227
Liked: 62 times
Joined: Apr 10, 2014 4:13 pm
Contact:

Re: NetApp Source as bottleneck

Post by lightsout » 1 person likes this post

So I finally found something that made a difference.

Rather than using FC, I tried a DirectNFS backup instead via 10Gb ethernet. For a job with one small VM, I got 94MB/sec via FC and after migrating it to a NFS datastore I backed up it again, and then I got 229MB/sec for the job instead.
lightsout
Expert
Posts: 227
Liked: 62 times
Joined: Apr 10, 2014 4:13 pm
Contact:

Re: NetApp Source as bottleneck

Post by lightsout »

And then I ran this util to benchmark my FC/NFS VM performance.

http://www.heraflux.com/resources/utili ... spd-batch/

They turned up almost identical, with FC being a few % faster at the high end. From my point of view, this means my FC & NFS infrastructure is good, so why is Veeam so slow over FC? I will say despite the speed difference in FC/SAN vs DirectNFS, source is still 99% the bottleneck according to Veeam.
larry
Veteran
Posts: 387
Liked: 97 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

Re: NetApp Source as bottleneck

Post by larry »

I have two sites with FAS2552 8.3.1P1, NFS 4.1, VMware 6. I do have the SSD cache and all SAS disks in the FAS2552s. My backup speeds are very good. Doing a full backup I see between 200 – 500 MBS. I have seen as little as 70 meg running backups during the day but the bottle neck was the destination. When I changed CBT off because of VM 6 bug I would see reads at 700 MPS. Which was mostly reads. I have 10 gig using Nexus 3K switches. My numbers were so different that I felt I needed to post. I even have some volumes with dedupe and compression on.

posted this last year in thread
vmware-vsphere-f24/netapp-backup-perfor ... 26635.html

I didnt add that I use all 10g "ethernet", which is most likly why my numbers were way different.

Info to add to lightsout

these numbers are when doing full or new backups. The more data the better the numbers.
Busy source moves around even inside same jobs different VMs. Source 75, proxy 53 network 47 target 31 then Source 33 proxy 94 Network 9 target 15 - numbers never stay the same. Most times source is the highest.
lightsout
Expert
Posts: 227
Liked: 62 times
Joined: Apr 10, 2014 4:13 pm
Contact:

Re: NetApp Source as bottleneck

Post by lightsout »

larry - what transport mode are you using for Veeam now? DirectNFS backups are plenty fast for me, it is the SAN/FC backups which are oddly slow (like half the speed of DirectNFS).
larry
Veteran
Posts: 387
Liked: 97 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

Re: NetApp Source as bottleneck

Post by larry »

I have different settings. One site I switched to all Network to see if I could stop the vm lock where you need to reset the mgmt service. I don't get it very often so I wont know for a month. They were all set to auto before the change. Switching to network didn't effect the throughput at all as we are all ten G. We have no FC.
lightsout
Expert
Posts: 227
Liked: 62 times
Joined: Apr 10, 2014 4:13 pm
Contact:

Re: NetApp Source as bottleneck

Post by lightsout » 1 person likes this post

OK so I have found a way to speed up my FC backups.

I was having terrible trouble using storage snapshots for my backups. Each backup job would run fine individually, but if too many ran at once the filer would become overloaded (due to zombie blocks on the NetApp) and would timeout in response to Veeam. I had both Veeam & NetApp support looking at it - both said the configuration on their side was fine. The solution was, to delete the FlexClone license from the NetApp! This should make it less efficient:

https://helpcenter.veeam.com/backup/vsp ... clone.html

But for me, it made it way more efficient, and now I'm getting 300MB/sec+ via FC. And my storage snapshots no longer impact my NetApp.
Pat490
Expert
Posts: 170
Liked: 29 times
Joined: Apr 28, 2015 7:18 am
Full Name: Patrick
Location: Germany
Contact:

Re: NetApp Source as bottleneck

Post by Pat490 »

what FAS & OnTap version are you using? We had to add exactly a FlexClone license to be able to use storage Snapshots from our NetApp after migration to cDot. What backup mode do use now?
lightsout
Expert
Posts: 227
Liked: 62 times
Joined: Apr 10, 2014 4:13 pm
Contact:

Re: NetApp Source as bottleneck

Post by lightsout »

Its a FAS3220 7-mode 8.2.4. We're about to change to cDOT 9.1 so we'll see if that makes any difference, I expect it will make a big one!
Post Reply

Who is online

Users browsing this forum: a.sim, Amazon [Bot] and 141 guests