Direct Storage Access FC - slow

[m1kkel]

Hello.

I am running a phusical server 2012R2 with 12 2 TB disks connected, using storage spaces mirror with 4 coloums, interleave at 262144.
This server is the proxy.
Deduplication is not enabled.
FIlesystem is NTFS formatted for large volume. 10,9 TB 64 KB chunk size.
2 SSD disks for write cache at 100 GB total (mirror)

I attached a qlogic FC adapter to the system, and in our fc switch, where our 3par 7200 array is connected.

Some time ago, before i took this into production i ran a test backup with a test server on a test lun, the first full backup i got over 250 MB / Sec write speec. (forever forward incremental)

After this, i started using the system for all our backups, and im currently using 9,5 TB.

However if i create a new backup job of the same test server, located on the same test lun, my speed have dropped to 150 MB / Sec. For other jobs speeds are varying between 80 MB / Sec and 130 MB / Sec, however theese backups are forever forward, and i assume the first full backup will be fastest because of sequential IO.

I can cot understand why the speed of a fresh full backup job from the same server have dropped so much.

I should be getting a lot more than 150 MB / Sec, and maybe also more than 247 MB / Sec The test server i'm using might have been thin provisioned at the first run a long time ago, and now it is thick provisioned lazy zeroed.
I am talking about processing rate when i am talking about the speed, and not throughput all time.

Run today:
03-03-2016 15:42:10 :: Using backup proxy 10.1.50.2 for disk Hard disk 1 [san]
03-03-2016 15:42:11 :: Hard disk 1 (95,0 GB) 49,1 GB read at 134 MB/s [CBT]
03-03-2016 15:32:46 :: Load: Source 99% > Proxy 82% > Network 18% > Target 0%

First run on 12/12-2015:
12-12-2015 23:19:29 :: Using backup proxy 10.1.50.2 for disk Hard disk 1 [san]
12-12-2015 23:19:31 :: Hard disk 1 (95,0 GB) 63,6 GB read at 247 MB/s [CBT]
12-12-2015 23:24:55 :: Load: Source 99% > Proxy 85% > Network 41% > Target 1%

It shows that veem's been waiting the longest for source, which is 3par system. It doesnt make any sense to me...

Please comment

[m1kkel]

Hi there.

Having some speed issues with full backup from 3par 7200 / 40 FC 10 Disks into psysical reposiroty and proxy server with FB 8 Gbit hba.

Veeam processing rate is 153 MB / Sec

3Par console show me that i am reading data from my test lun with my test server at 175 MB / Sec at the lowest, and 400 MB / Sec at the highest.
Screenshot from 3par: http://imgur.com/VsBQK3N

As you can see, i will assume the average is 300 MB / Sec, yet veeam shows me 153 MB. There is no other activity on that lun.

06-03-2016 21:51:13 :: Hard disk 1 (95,0 GB) 49,0 GB read at 153 MB/s [CBT]
06-03-2016 21:57:30 :: Busy: Source 99% > Proxy 86% > Network 18% > Target 0%

Am i missing something?

[Gostev]

Hi. I recommend that you check with HP on their reporting, since math says numbers in Veeam statistics are correct: 49GB at 153 MB/s needs about 6 minutes to complete (not exactly 6:17 as per your log, because initializing and finalizing the transfer also takes some time).

[m1kkel]

And right you are - that leads me to the next question, why am i not having higher speeds on full backup via FC?
The proxy server only uses 1 Ghz at one core so it is not doing anything, consuming 2 GB ram total of 32..

[Gostev]

According to bottleneck stats above, either 3PAR itself is very busy serving other workloads, or there is some kind of fabric issue.

[m1kkel]

3Par is doning nothing at that time. Fabric issue, hmm do you have any suggestion on how to find out whitch part of my fabric issue is the problem?

Would it help to determine if 3åpar is the issue, by simply creating a test lun there, initializing it on our proxy, placing a VBK file there, and copying it back to the proxy repository? DOes that simulate a full backup? .. sequintal IO ..

[Vitaliy S.]

Hi Mikkel,

Bases on the stats both source and backup proxy affect your job performance, so try to update your FC adapter firmware, might help. BTW, what is the CPU usage on the proxy server when your backup job is running?

Thanks!

[m1kkel]

Yeah that's the funny part, Backup proxy which is also the repository is doing nothing. Cpu usage 900 Mhz, 2 GB ram used. It's a 3.1 Ghz quad core Xeon 1220 Cpu with 32 GB memory.
I already updated my HBA firmware, was thinking of replacing it with a brocade instead, just to try something.
Now i've rebooted the server, and configured it's bios to let the CPU work full throttle all the time. Ran a new job, and speed increased to 206 MB / Sec, bottleneck stats:
07-03-2016 13:18:26 :: Load: Source 99% > Proxy 50% > Network 3% > Target 0%

Funny thing i noticed, when i ran this job in December, it was prior to V9 upgrade. Don't know if that have anything to say?

206 MB / Sec is still not enough for active full on incremental backup from 3par storage on fibre channel..,

[Vitaliy S.]

Just for performance testing purposes, can you run a test backup job using hotadd proxy located on this storage? It would interesting to compare the stats.

[Vitaliy S.]

Would it help to determine if 3åpar is the issue, by simply creating a test lun there, initializing it on our proxy, placing a VBK file there, and copying it back to the proxy repository? DOes that simulate a full backup?

Not exactly, since copying data between the LUNs and retrieving it via vStorage API is a bit different.

[m1kkel]

Ok.

[m1kkel]

Jep.

Here are the results from active full of the same server, hot add, virtuel proxy.
07-03-2016 13:39:44 :: Hard disk 1 (95,0 GB) 49,0 GB read at 143 MB/s [CBT]
07-03-2016 13:46:33 :: Busy: Source 97% > Proxy 41% > Network 13% > Target 0%

...

[Gostev]

Merged duplicate discussions.

[foggy]

Mikkel, I'd recommend opening a ticket with a support team for a closer investigation, including your deployment review and probably running some performance tests. Involving storage vendor might also be a good step.

Also, haven't you thought of using storage snapshots to backup from?

[m1kkel]

Ok, i will open a ticket.

I have thought of storage snapshots, unfortunately that requires the entreprise plus license as far as i am concerned..

[foggy]

I would appreciate if you keep us posted about the results of your investigation.

You're correct regarding the license requirement.

[m1kkel]

We have a deal.

[Delo123]

Hi Mikkel, any update on this?
I also sometimes wonder why some jobs run with 1GB/s and others with "only" 200MB/s...

[m1kkel]

Well yes.

Veeam support had me run the test with diskspd.exe on disk id #17 (wich is a testlun on my san) and that test gave me 300 MB / Sec.
Testing with large block size 4096 KB gave me over 400 MB / Sec.

However they wanted to run a test using VDDK library, which is through the vmware stack, and that is what veeam utilizes when we run backup. That test gave me the same speeds as Veeam - so the support engineer pointed me at vmware, and that's who i am talking to right now, started yestoday, and we do not have a conclusion yet, they are asking silly questions like: "did you talk to the storage team to see if you have latency issues" - I AM THE STORAGE TEAM, AND DISKSPD.EXE GIVES ME OVER 400 MB / SEC
Jesus!

Which version of vmware are you using?
I am on 5.5 ..

[Delo123]

WHat is the syntax they wanted you to run?
diskspd.exe -c1G -b4 -t4 -d20 -a0,1 test1.dat test2.dat (or b4096K for 4MBI)
Total IO 4K
thread | bytes | I/Os | MB/s | I/O per s |
total: 39896186880 | 9740280 | 1841.94 | 471537.28
Read IO 4096K
thread | bytes | I/Os | MB/s | I/O per s |
------------------------------------------------------------------------------
total: 122771472384| 29271 | 5675.64 | 1418.91

We are mixed, these results are from latest 5.5, didn't test on 6 yet

Did you ever try a vm with multiple disks? This is a backup job of a single VM, processing rate is 627MB/s but with multiple disks (same SAN)
15.03.2016 21:28:25 :: Hard disk 3 (1,3 TB) 81,2 GB read at 173 MB/s [CBT]
15.03.2016 21:28:25 :: Hard disk 4 (1000,0 GB) 143,2 GB read at 199 MB/s [CBT]
15.03.2016 21:28:26 :: Hard disk 2 (1,3 TB) 81,1 GB read at 178 MB/s [CBT]
15.03.2016 21:28:26 :: Hard disk 5 (1000,0 GB) 143,1 GB read at 208 MB/s [CBT]
15.03.2016 21:28:26 :: Hard disk 1 (40,0 GB) 2,6 GB read at 120 MB/s [CBT]

[m1kkel]

How can you reach 5675 MB / Sec? I think you need to test with cache disabled. the "-H" option.
If you are using direct san access, you need to test your lun and not a disk you've presented to your proxy. Like this:

C:\Users\Administrator\Desktop\Diskspd-v2.0.15\amd64fre>diskspd.exe -b4096K -h -d60 #17
#17 is my disk number
-d60 is duration 60 seconds
-h is to disable host caching.



This is the command they ran for me, on my proxy:
c:\tmp\vix\bin>VixDiskLibSample -readbench 4096 -host 10.1.8.2 -user veeam -password XXXXXX -mode san -vm "moref=vm-33299" -ssmoref "snapshot-37207" -initex "C:\tmp\vix\initex.txt" -libdir "C:Program Files (x86)\Veeam\Backup Transport\x64\vddk_6_0" -thumb "ac:1a:b4:8a:92:2d:92:d5:8e:ca:f3:42:03:06:01:cc:a5:bb:73:67" "[Test] HostingTest/HostingTest.vmdk"

It outputs the speed pr. second, and it is around 150 to 200 MB / Sec.

I just created a new job with 2 servers in it, total 2 disk's, one for each VM.
16-03-2016 12:16:46 :: Hard disk 1 (95,0 GB) 49,1 GB read at 145 MB/s [CBT]
16-03-2016 12:16:46 :: Hard disk 1 (81,0 GB) 73,2 GB read at 179 MB/s [CBT]

Processing rate is 299 MB / Sec.
Maybe i can push that with a job with vm's with more disks.
I will try, hold on.

[m1kkel]

Are your disks thin, or thick eager or thick lazy?
Which san, and how many disks?
Do you have one or 2 FC ports on your proxy?

[Delo123]

Just tested a physical Lun, I think it's capped by the HBA's,

Total IO
thread | bytes | I/Os | MB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 26734493696 | 6374 | 1274.71 | 318.68 | #49 (49GB)
------------------------------------------------------------------------------
total: 26734493696 | 6374 | 1274.71 | 318.68

We use DataCore with a mixture of 48 ssd's and 4 nvme's per node. The Veeam proxy is somewhat limited since it "only" has 2 dual 8Gbit FC, 2 ports for each node, this test was for a single node, so 2x8GB FC. All luns are thick eager zeroed

[emachabert]

Mikkel, I have dozens of 3Par arrays backuped by Veeam, using DirectSAN over FC.
They all give high numbers when it comes to full backup with Veeam. The smallest is a 7200 with 40*450 10K and active full runs between 700 and 1200 MB/s (from R5 5+1 CPG vluns), with 6 concurrent threads.

Just follow best practices :
- Windows MPIO configured for 3ParVV
- Fillword set to 3 on Brocade 8Gb fabrics
- All VM disks are Eager thick zeroed
- VVs are thin provisoinned

[m1kkel]

Great info! THanks!
Well i have a 7200 with 40 FC 10K 600 GB Disks + SSD layer. All connected to the same switch, and all vm's are on CPG: R5 5+1 step size 128 KB, preferred chunklets: fast. It is only possible for me to run with 4 concurrent threads, i could try to create a new job with 4 servers with total 4 disks to monitor the speed. I just testet with a new job with 2 servers with 2 disks. Speeds:
16-03-2016 12:16:46 :: Hard disk 1 (95,0 GB) 49,1 GB read at 145 MB/s [CBT]
16-03-2016 12:16:46 :: Hard disk 1 (81,0 GB) 73,2 GB read at 179 MB/s [CBT]
324 MB / Sec and processing rate was 299.

However, also you also states, i should get much more.

- Windows MPIO configured for 3ParVV
Already done, and from 3par cli i can see that i am using all 4 paths to the storage.

- Fillword set to 3 on Brocade 8Gb fabrics
Here we have a problem. FIllword is set to 3 for my VMWARE hosts and my proxy, however ports connecting to 3par is set to fillword 0... Should i set it to 3 everywhere?
I had a LOT of errors in my port logs, but i do not know when i last reset them, maybe it was before we got the 3par system so i just cleared them.
ESX03 had er_bad_os count 581.321 - that's a lot...


- All VM disks are Eager thick zeroed
Here we have a problem. All my vmware disks are thin provisioned, and it is a hell to manage. Is 3par best practises, to use Eager Thick?

- VVs are thin provisoinned[/quote]
So are mine.

[m1kkel]

I can see from portstatsshow that i also have lots of er_bad_os on my 3par ports. 4.153.070.745 Invalid ordered set
Those ports have only been in use for 3par. Will clear them now and watch them.

[m1kkel]

Okay, er_bad_os is just rising and rising constantly. Is this a problem?

[Delo123]

On all the ports? That doesn't sound good... Not sure if you are able to test 4GB to see if that's fine? When not I hope somebody with brocade's can help you out, we are qlogic only

[emachabert]

This is a known issue with Brocade fabric @8gb/s.
FIllword should be set to 3 (if ARBF/ARBF fails, use IDLE/ARBF), if not you get bad_os error increasing continuously.

Beware, configuring the fillword will disable/enable the port, so do one port at a time with 5 min pause within each.

Regarding the eager thick zeroed, you should definitively look at the litterature about Thin on Thin, Thin on Thick, Thick on Thick and thick on thin
When dealing with a 3par, having hardware assisted thin provisionnig and global wide stripping, you should really consider using Thick on Thin (Eager Zeroed).

One Veeam value about using Thick VM disk is DirectSAN restore and CBT restore !! Think about it !

[BrandonH]

Interesting, I'm seeing the same numbers as you (130-180MB/s). I'm using Brocade Condor3/16G (No Fillword), so that's not in my options.

I have two 7400's, 24 SSD, 148 FC, 60 NL.

I have two Proxies, HP DL380G9's (Dual 12 Core, 32G ram) with Brocade/QLogic 1862's with 16GFC/10G Ethernet.

I use Thin VV's with Thick Eager Zeroed.

My 3PAR's are seperated by roughly 66k of DWDM. Two 10g for Ethernet, two 10G FC (Roughly 500ns return). <--This is my only complicating factor

I back up to FC Storage, with a copy job that then moved the job to a StoreOnce appliance for longer retention.

All of my hosts reside on 1862 Brocade's as well, same 16g/10g setup. I use storage snapshots for my backups. My hosts are also running 5.5.

When speaking to support a year or so ago, I was told the speeds I'm getting are normal, that I wouldn't see any faster. I don't have any fabric errors or port errors, everything seems to be running very clean. An active full will net me 188 average. I also get 99% Source, 24% Proxy, 1% Network, 1% Target (That math seems a bit off too rofl).

I'm running Flash Caching, I was taking a large amount of hits. I tend to stay around 25-30% during my peek traffic times.