Direct Storage Access FC - slow

VMware specific discussions

Re: Direct Storage Access FC - slow

Veeam Logoby foggy » Mon Mar 07, 2016 3:00 pm

I would appreciate if you keep us posted about the results of your investigation.

You're correct regarding the license requirement.
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Direct Storage Access FC - slow

Veeam Logoby m1kkel » Mon Mar 07, 2016 3:06 pm

We have a deal.
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

Re: Direct Storage Access FC - slow

Veeam Logoby Delo123 » Tue Mar 15, 2016 3:01 pm

Hi Mikkel, any update on this?
I also sometimes wonder why some jobs run with 1GB/s and others with "only" 200MB/s...
Delo123
Expert
 
Posts: 348
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Direct Storage Access FC - slow

Veeam Logoby m1kkel » Tue Mar 15, 2016 7:57 pm

Well yes.

Veeam support had me run the test with diskspd.exe on disk id #17 (wich is a testlun on my san) and that test gave me 300 MB / Sec.
Testing with large block size 4096 KB gave me over 400 MB / Sec.

However they wanted to run a test using VDDK library, which is through the vmware stack, and that is what veeam utilizes when we run backup. That test gave me the same speeds as Veeam - so the support engineer pointed me at vmware, and that's who i am talking to right now, started yestoday, and we do not have a conclusion yet, they are asking silly questions like: "did you talk to the storage team to see if you have latency issues" - I AM THE STORAGE TEAM, AND DISKSPD.EXE GIVES ME OVER 400 MB / SEC
Jesus!

Which version of vmware are you using?
I am on 5.5 ..
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

Re: Direct Storage Access FC - slow

Veeam Logoby Delo123 » Tue Mar 15, 2016 8:49 pm

WHat is the syntax they wanted you to run?
diskspd.exe -c1G -b4 -t4 -d20 -a0,1 test1.dat test2.dat (or b4096K for 4MBI)
Total IO 4K
thread | bytes | I/Os | MB/s | I/O per s |
total: 39896186880 | 9740280 | 1841.94 | 471537.28
Read IO 4096K
thread | bytes | I/Os | MB/s | I/O per s |
------------------------------------------------------------------------------
total: 122771472384| 29271 | 5675.64 | 1418.91

We are mixed, these results are from latest 5.5, didn't test on 6 yet

Did you ever try a vm with multiple disks? This is a backup job of a single VM, processing rate is 627MB/s but with multiple disks (same SAN)
15.03.2016 21:28:25 :: Hard disk 3 (1,3 TB) 81,2 GB read at 173 MB/s [CBT]
15.03.2016 21:28:25 :: Hard disk 4 (1000,0 GB) 143,2 GB read at 199 MB/s [CBT]
15.03.2016 21:28:26 :: Hard disk 2 (1,3 TB) 81,1 GB read at 178 MB/s [CBT]
15.03.2016 21:28:26 :: Hard disk 5 (1000,0 GB) 143,1 GB read at 208 MB/s [CBT]
15.03.2016 21:28:26 :: Hard disk 1 (40,0 GB) 2,6 GB read at 120 MB/s [CBT]
Delo123
Expert
 
Posts: 348
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Direct Storage Access FC - slow

Veeam Logoby m1kkel » Wed Mar 16, 2016 1:55 pm

How can you reach 5675 MB / Sec? I think you need to test with cache disabled. the "-H" option.
If you are using direct san access, you need to test your lun and not a disk you've presented to your proxy. Like this:

C:\Users\Administrator\Desktop\Diskspd-v2.0.15\amd64fre>diskspd.exe -b4096K -h -d60 #17
#17 is my disk number
-d60 is duration 60 seconds
-h is to disable host caching.



This is the command they ran for me, on my proxy:
c:\tmp\vix\bin>VixDiskLibSample -readbench 4096 -host 10.1.8.2 -user veeam -password XXXXXX -mode san -vm "moref=vm-33299" -ssmoref "snapshot-37207" -initex "C:\tmp\vix\initex.txt" -libdir "C:Program Files (x86)\Veeam\Backup Transport\x64\vddk_6_0" -thumb "ac:1a:b4:8a:92:2d:92:d5:8e:ca:f3:42:03:06:01:cc:a5:bb:73:67" "[Test] HostingTest/HostingTest.vmdk"

It outputs the speed pr. second, and it is around 150 to 200 MB / Sec.

I just created a new job with 2 servers in it, total 2 disk's, one for each VM.
16-03-2016 12:16:46 :: Hard disk 1 (95,0 GB) 49,1 GB read at 145 MB/s [CBT]
16-03-2016 12:16:46 :: Hard disk 1 (81,0 GB) 73,2 GB read at 179 MB/s [CBT]

Processing rate is 299 MB / Sec.
Maybe i can push that with a job with vm's with more disks.
I will try, hold on.
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

Re: Direct Storage Access FC - slow

Veeam Logoby m1kkel » Wed Mar 16, 2016 1:59 pm

Are your disks thin, or thick eager or thick lazy?
Which san, and how many disks?
Do you have one or 2 FC ports on your proxy?
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

Re: Direct Storage Access FC - slow

Veeam Logoby Delo123 » Wed Mar 16, 2016 2:39 pm

Just tested a physical Lun, I think it's capped by the HBA's,

Total IO
thread | bytes | I/Os | MB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 26734493696 | 6374 | 1274.71 | 318.68 | #49 (49GB)
------------------------------------------------------------------------------
total: 26734493696 | 6374 | 1274.71 | 318.68

We use DataCore with a mixture of 48 ssd's and 4 nvme's per node. The Veeam proxy is somewhat limited since it "only" has 2 dual 8Gbit FC, 2 ports for each node, this test was for a single node, so 2x8GB FC. All luns are thick eager zeroed
Delo123
Expert
 
Posts: 348
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Direct Storage Access FC - slow

Veeam Logoby emachabert » Wed Mar 16, 2016 2:42 pm 2 people like this post

Mikkel, I have dozens of 3Par arrays backuped by Veeam, using DirectSAN over FC.
They all give high numbers when it comes to full backup with Veeam. The smallest is a 7200 with 40*450 10K and active full runs between 700 and 1200 MB/s (from R5 5+1 CPG vluns), with 6 concurrent threads.

Just follow best practices :
- Windows MPIO configured for 3ParVV
- Fillword set to 3 on Brocade 8Gb fabrics
- All VM disks are Eager thick zeroed
- VVs are thin provisoinned
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017
emachabert
Veeam Vanguard
 
Posts: 354
Liked: 163 times
Joined: Wed Nov 17, 2010 11:42 am
Location: France
Full Name: Eric Machabert

Re: Direct Storage Access FC - slow

Veeam Logoby m1kkel » Wed Mar 16, 2016 3:00 pm

Great info! THanks!
Well i have a 7200 with 40 FC 10K 600 GB Disks + SSD layer. All connected to the same switch, and all vm's are on CPG: R5 5+1 step size 128 KB, preferred chunklets: fast. It is only possible for me to run with 4 concurrent threads, i could try to create a new job with 4 servers with total 4 disks to monitor the speed. I just testet with a new job with 2 servers with 2 disks. Speeds:
16-03-2016 12:16:46 :: Hard disk 1 (95,0 GB) 49,1 GB read at 145 MB/s [CBT]
16-03-2016 12:16:46 :: Hard disk 1 (81,0 GB) 73,2 GB read at 179 MB/s [CBT]
324 MB / Sec and processing rate was 299.

However, also you also states, i should get much more.

- Windows MPIO configured for 3ParVV
Already done, and from 3par cli i can see that i am using all 4 paths to the storage.

- Fillword set to 3 on Brocade 8Gb fabrics
Here we have a problem. FIllword is set to 3 for my VMWARE hosts and my proxy, however ports connecting to 3par is set to fillword 0... Should i set it to 3 everywhere?
I had a LOT of errors in my port logs, but i do not know when i last reset them, maybe it was before we got the 3par system so i just cleared them.
ESX03 had er_bad_os count 581.321 - that's a lot...


- All VM disks are Eager thick zeroed
Here we have a problem. All my vmware disks are thin provisioned, and it is a hell to manage. Is 3par best practises, to use Eager Thick?

- VVs are thin provisoinned[/quote]
So are mine.
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

Re: Direct Storage Access FC - slow

Veeam Logoby m1kkel » Wed Mar 16, 2016 3:04 pm

I can see from portstatsshow that i also have lots of er_bad_os on my 3par ports. 4.153.070.745 Invalid ordered set
Those ports have only been in use for 3par. Will clear them now and watch them.
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

Re: Direct Storage Access FC - slow

Veeam Logoby m1kkel » Wed Mar 16, 2016 3:06 pm

Okay, er_bad_os is just rising and rising constantly. Is this a problem?
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

Re: Direct Storage Access FC - slow

Veeam Logoby Delo123 » Wed Mar 16, 2016 3:35 pm

On all the ports? That doesn't sound good... Not sure if you are able to test 4GB to see if that's fine? When not I hope somebody with brocade's can help you out, we are qlogic only
Delo123
Expert
 
Posts: 348
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Direct Storage Access FC - slow

Veeam Logoby emachabert » Wed Mar 16, 2016 4:35 pm 1 person likes this post

This is a known issue with Brocade fabric @8gb/s.
FIllword should be set to 3 (if ARBF/ARBF fails, use IDLE/ARBF), if not you get bad_os error increasing continuously.

Beware, configuring the fillword will disable/enable the port, so do one port at a time with 5 min pause within each.

Regarding the eager thick zeroed, you should definitively look at the litterature about Thin on Thin, Thin on Thick, Thick on Thick and thick on thin :D
When dealing with a 3par, having hardware assisted thin provisionnig and global wide stripping, you should really consider using Thick on Thin (Eager Zeroed).

One Veeam value about using Thick VM disk is DirectSAN restore and CBT restore !! Think about it !

:D
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017
emachabert
Veeam Vanguard
 
Posts: 354
Liked: 163 times
Joined: Wed Nov 17, 2010 11:42 am
Location: France
Full Name: Eric Machabert

Re: Direct Storage Access FC - slow

Veeam Logoby BrandonH » Wed Mar 16, 2016 10:03 pm

Interesting, I'm seeing the same numbers as you (130-180MB/s). I'm using Brocade Condor3/16G (No Fillword), so that's not in my options.

I have two 7400's, 24 SSD, 148 FC, 60 NL.

I have two Proxies, HP DL380G9's (Dual 12 Core, 32G ram) with Brocade/QLogic 1862's with 16GFC/10G Ethernet.

I use Thin VV's with Thick Eager Zeroed.

My 3PAR's are seperated by roughly 66k of DWDM. Two 10g for Ethernet, two 10G FC (Roughly 500ns return). <--This is my only complicating factor

I back up to FC Storage, with a copy job that then moved the job to a StoreOnce appliance for longer retention.

All of my hosts reside on 1862 Brocade's as well, same 16g/10g setup. I use storage snapshots for my backups. My hosts are also running 5.5.

When speaking to support a year or so ago, I was told the speeds I'm getting are normal, that I wouldn't see any faster. I don't have any fabric errors or port errors, everything seems to be running very clean. An active full will net me 188 average. I also get 99% Source, 24% Proxy, 1% Network, 1% Target (That math seems a bit off too rofl).

I'm running Flash Caching, I was taking a large amount of hits. I tend to stay around 25-30% during my peek traffic times.
BrandonH
Influencer
 
Posts: 21
Liked: 2 times
Joined: Wed Jul 10, 2013 4:37 pm
Full Name: Brandon Hogue

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: ilya.konopak and 20 guests