NetApp Source as bottleneck

[lightsout]

No, I never got to the bottom of it to be honest. However it remains on my list, but I want to do some performance testing to see if my VMs can pull more than 100MB/sec.

Changes to ESXi 6 hasn't made any difference.

[lightsout]

So I finally found something that made a difference.

Rather than using FC, I tried a DirectNFS backup instead via 10Gb ethernet. For a job with one small VM, I got 94MB/sec via FC and after migrating it to a NFS datastore I backed up it again, and then I got 229MB/sec for the job instead.

[lightsout]

And then I ran this util to benchmark my FC/NFS VM performance.

http://www.heraflux.com/resources/utili ... spd-batch/

They turned up almost identical, with FC being a few % faster at the high end. From my point of view, this means my FC & NFS infrastructure is good, so why is Veeam so slow over FC? I will say despite the speed difference in FC/SAN vs DirectNFS, source is still 99% the bottleneck according to Veeam.

[larry]

I have two sites with FAS2552 8.3.1P1, NFS 4.1, VMware 6. I do have the SSD cache and all SAS disks in the FAS2552s. My backup speeds are very good. Doing a full backup I see between 200 – 500 MBS. I have seen as little as 70 meg running backups during the day but the bottle neck was the destination. When I changed CBT off because of VM 6 bug I would see reads at 700 MPS. Which was mostly reads. I have 10 gig using Nexus 3K switches. My numbers were so different that I felt I needed to post. I even have some volumes with dedupe and compression on.

posted this last year in thread
vmware-vsphere-f24/netapp-backup-perfor ... 26635.html

I didnt add that I use all 10g "ethernet", which is most likly why my numbers were way different.

Info to add to lightsout

these numbers are when doing full or new backups. The more data the better the numbers.
Busy source moves around even inside same jobs different VMs. Source 75, proxy 53 network 47 target 31 then Source 33 proxy 94 Network 9 target 15 - numbers never stay the same. Most times source is the highest.

[lightsout]

larry - what transport mode are you using for Veeam now? DirectNFS backups are plenty fast for me, it is the SAN/FC backups which are oddly slow (like half the speed of DirectNFS).

[larry]

I have different settings. One site I switched to all Network to see if I could stop the vm lock where you need to reset the mgmt service. I don't get it very often so I wont know for a month. They were all set to auto before the change. Switching to network didn't effect the throughput at all as we are all ten G. We have no FC.

[lightsout]

OK so I have found a way to speed up my FC backups.

I was having terrible trouble using storage snapshots for my backups. Each backup job would run fine individually, but if too many ran at once the filer would become overloaded (due to zombie blocks on the NetApp) and would timeout in response to Veeam. I had both Veeam & NetApp support looking at it - both said the configuration on their side was fine. The solution was, to delete the FlexClone license from the NetApp! This should make it less efficient:

https://helpcenter.veeam.com/backup/vsp ... clone.html

But for me, it made it way more efficient, and now I'm getting 300MB/sec+ via FC. And my storage snapshots no longer impact my NetApp.

[Pat490]

what FAS & OnTap version are you using? We had to add exactly a FlexClone license to be able to use storage Snapshots from our NetApp after migration to cDot. What backup mode do use now?

[lightsout]

Its a FAS3220 7-mode 8.2.4. We're about to change to cDOT 9.1 so we'll see if that makes any difference, I expect it will make a big one!