NetApp Source as bottleneck

Availability for the Always-On Enterprise

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Mon Dec 14, 2015 4:34 pm

No, I never got to the bottom of it to be honest. However it remains on my list, but I want to do some performance testing to see if my VMs can pull more than 100MB/sec.

Changes to ESXi 6 hasn't made any difference.
lightsout
Expert
 
Posts: 188
Liked: 48 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Tue Sep 27, 2016 1:27 pm 1 person likes this post

So I finally found something that made a difference.

Rather than using FC, I tried a DirectNFS backup instead via 10Gb ethernet. For a job with one small VM, I got 94MB/sec via FC and after migrating it to a NFS datastore I backed up it again, and then I got 229MB/sec for the job instead.
lightsout
Expert
 
Posts: 188
Liked: 48 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Mon Oct 03, 2016 12:59 pm

And then I ran this util to benchmark my FC/NFS VM performance.

http://www.heraflux.com/resources/utili ... spd-batch/

They turned up almost identical, with FC being a few % faster at the high end. From my point of view, this means my FC & NFS infrastructure is good, so why is Veeam so slow over FC? I will say despite the speed difference in FC/SAN vs DirectNFS, source is still 99% the bottleneck according to Veeam.
lightsout
Expert
 
Posts: 188
Liked: 48 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby larry » Mon Oct 03, 2016 9:06 pm

I have two sites with FAS2552 8.3.1P1, NFS 4.1, VMware 6. I do have the SSD cache and all SAS disks in the FAS2552s. My backup speeds are very good. Doing a full backup I see between 200 – 500 MBS. I have seen as little as 70 meg running backups during the day but the bottle neck was the destination. When I changed CBT off because of VM 6 bug I would see reads at 700 MPS. Which was mostly reads. I have 10 gig using Nexus 3K switches. My numbers were so different that I felt I needed to post. I even have some volumes with dedupe and compression on.

posted this last year in thread
vmware-vsphere-f24/netapp-backup-performance-t26635.html

I didnt add that I use all 10g "ethernet", which is most likly why my numbers were way different.

Info to add to lightsout

these numbers are when doing full or new backups. The more data the better the numbers.
Busy source moves around even inside same jobs different VMs. Source 75, proxy 53 network 47 target 31 then Source 33 proxy 94 Network 9 target 15 - numbers never stay the same. Most times source is the highest.
larry
Expert
 
Posts: 384
Liked: 90 times
Joined: Wed Mar 24, 2010 5:47 pm
Full Name: Larry Walker

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Mon Oct 03, 2016 11:50 pm

larry - what transport mode are you using for Veeam now? DirectNFS backups are plenty fast for me, it is the SAN/FC backups which are oddly slow (like half the speed of DirectNFS).
lightsout
Expert
 
Posts: 188
Liked: 48 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby larry » Tue Oct 04, 2016 2:55 pm

I have different settings. One site I switched to all Network to see if I could stop the vm lock where you need to reset the mgmt service. I don't get it very often so I wont know for a month. They were all set to auto before the change. Switching to network didn't effect the throughput at all as we are all ten G. We have no FC.
larry
Expert
 
Posts: 384
Liked: 90 times
Joined: Wed Mar 24, 2010 5:47 pm
Full Name: Larry Walker

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Mon Feb 13, 2017 2:26 pm 1 person likes this post

OK so I have found a way to speed up my FC backups.

I was having terrible trouble using storage snapshots for my backups. Each backup job would run fine individually, but if too many ran at once the filer would become overloaded (due to zombie blocks on the NetApp) and would timeout in response to Veeam. I had both Veeam & NetApp support looking at it - both said the configuration on their side was fine. The solution was, to delete the FlexClone license from the NetApp! This should make it less efficient:

https://helpcenter.veeam.com/backup/vsp ... clone.html

But for me, it made it way more efficient, and now I'm getting 300MB/sec+ via FC. And my storage snapshots no longer impact my NetApp.
lightsout
Expert
 
Posts: 188
Liked: 48 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: NetApp Source as bottleneck

Veeam Logoby Pat490 » Tue Feb 14, 2017 10:29 am

what FAS & OnTap version are you using? We had to add exactly a FlexClone license to be able to use storage Snapshots from our NetApp after migration to cDot. What backup mode do use now?
Pat490
Expert
 
Posts: 142
Liked: 24 times
Joined: Tue Apr 28, 2015 7:18 am
Location: Germany
Full Name: Patrick

Re: NetApp Source as bottleneck

Veeam Logoby lightsout » Fri Feb 17, 2017 3:02 pm

Its a FAS3220 7-mode 8.2.4. We're about to change to cDOT 9.1 so we'll see if that makes any difference, I expect it will make a big one!
lightsout
Expert
 
Posts: 188
Liked: 48 times
Joined: Thu Apr 10, 2014 4:13 pm

Previous

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: baber, Google [Bot], tdewin and 1 guest