Direct Storage Access FC - slow

VMware specific discussions

Direct Storage Access FC - slow

Veeam Logoby m1kkel » Thu Mar 03, 2016 3:12 pm

Hello.

I am running a phusical server 2012R2 with 12 2 TB disks connected, using storage spaces mirror with 4 coloums, interleave at 262144.
This server is the proxy.
Deduplication is not enabled.
FIlesystem is NTFS formatted for large volume. 10,9 TB 64 KB chunk size.
2 SSD disks for write cache at 100 GB total (mirror)

I attached a qlogic FC adapter to the system, and in our fc switch, where our 3par 7200 array is connected.

Some time ago, before i took this into production i ran a test backup with a test server on a test lun, the first full backup i got over 250 MB / Sec write speec. (forever forward incremental)

After this, i started using the system for all our backups, and im currently using 9,5 TB.

However if i create a new backup job of the same test server, located on the same test lun, my speed have dropped to 150 MB / Sec. For other jobs speeds are varying between 80 MB / Sec and 130 MB / Sec, however theese backups are forever forward, and i assume the first full backup will be fastest because of sequential IO.

I can cot understand why the speed of a fresh full backup job from the same server have dropped so much.

I should be getting a lot more than 150 MB / Sec, and maybe also more than 247 MB / Sec The test server i'm using might have been thin provisioned at the first run a long time ago, and now it is thick provisioned lazy zeroed.
I am talking about processing rate when i am talking about the speed, and not throughput all time.

Run today:
03-03-2016 15:42:10 :: Using backup proxy 10.1.50.2 for disk Hard disk 1 [san]
03-03-2016 15:42:11 :: Hard disk 1 (95,0 GB) 49,1 GB read at 134 MB/s [CBT]
03-03-2016 15:32:46 :: Load: Source 99% > Proxy 82% > Network 18% > Target 0%

First run on 12/12-2015:
12-12-2015 23:19:29 :: Using backup proxy 10.1.50.2 for disk Hard disk 1 [san]
12-12-2015 23:19:31 :: Hard disk 1 (95,0 GB) 63,6 GB read at 247 MB/s [CBT]
12-12-2015 23:24:55 :: Load: Source 99% > Proxy 85% > Network 41% > Target 1%

It shows that veem's been waiting the longest for source, which is 3par system. It doesnt make any sense to me...

Please comment :)
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

[MERGED] Direct Access VEEAM speed vs. 3par

Veeam Logoby m1kkel » Sun Mar 06, 2016 9:13 pm

Hi there.

Having some speed issues with full backup from 3par 7200 / 40 FC 10 Disks into psysical reposiroty and proxy server with FB 8 Gbit hba.

Veeam processing rate is 153 MB / Sec

3Par console show me that i am reading data from my test lun with my test server at 175 MB / Sec at the lowest, and 400 MB / Sec at the highest.
Screenshot from 3par: http://imgur.com/VsBQK3N

As you can see, i will assume the average is 300 MB / Sec, yet veeam shows me 153 MB. There is no other activity on that lun.

06-03-2016 21:51:13 :: Hard disk 1 (95,0 GB) 49,0 GB read at 153 MB/s [CBT]
06-03-2016 21:57:30 :: Busy: Source 99% > Proxy 86% > Network 18% > Target 0%

Am i missing something?
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

Re: Direct Access VEEAM speed vs. 3par

Veeam Logoby Gostev » Sun Mar 06, 2016 9:29 pm

Hi. I recommend that you check with HP on their reporting, since math says numbers in Veeam statistics are correct: 49GB at 153 MB/s needs about 6 minutes to complete (not exactly 6:17 as per your log, because initializing and finalizing the transfer also takes some time).
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Direct Access VEEAM speed vs. 3par

Veeam Logoby m1kkel » Sun Mar 06, 2016 9:41 pm

And right you are - that leads me to the next question, why am i not having higher speeds on full backup via FC? :-)
The proxy server only uses 1 Ghz at one core so it is not doing anything, consuming 2 GB ram total of 32..
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

Re: Direct Access VEEAM speed vs. 3par

Veeam Logoby Gostev » Sun Mar 06, 2016 10:36 pm

According to bottleneck stats above, either 3PAR itself is very busy serving other workloads, or there is some kind of fabric issue.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Direct Access VEEAM speed vs. 3par

Veeam Logoby m1kkel » Mon Mar 07, 2016 8:51 am

3Par is doning nothing at that time. Fabric issue, hmm do you have any suggestion on how to find out whitch part of my fabric issue is the problem?

Would it help to determine if 3åpar is the issue, by simply creating a test lun there, initializing it on our proxy, placing a VBK file there, and copying it back to the proxy repository? DOes that simulate a full backup? .. sequintal IO ..
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

Re: Direct Storage Access FC - slow

Veeam Logoby Vitaliy S. » Mon Mar 07, 2016 12:07 pm

Hi Mikkel,

Bases on the stats both source and backup proxy affect your job performance, so try to update your FC adapter firmware, might help. BTW, what is the CPU usage on the proxy server when your backup job is running?

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19570
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Direct Storage Access FC - slow

Veeam Logoby m1kkel » Mon Mar 07, 2016 12:28 pm

Yeah that's the funny part, Backup proxy which is also the repository is doing nothing. Cpu usage 900 Mhz, 2 GB ram used. It's a 3.1 Ghz quad core Xeon 1220 Cpu with 32 GB memory.
I already updated my HBA firmware, was thinking of replacing it with a brocade instead, just to try something.
Now i've rebooted the server, and configured it's bios to let the CPU work full throttle all the time. Ran a new job, and speed increased to 206 MB / Sec, bottleneck stats:
07-03-2016 13:18:26 :: Load: Source 99% > Proxy 50% > Network 3% > Target 0%

Funny thing i noticed, when i ran this job in December, it was prior to V9 upgrade. Don't know if that have anything to say?

206 MB / Sec is still not enough for active full on incremental backup from 3par storage on fibre channel..,
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

Re: Direct Storage Access FC - slow

Veeam Logoby Vitaliy S. » Mon Mar 07, 2016 12:37 pm

Just for performance testing purposes, can you run a test backup job using hotadd proxy located on this storage? It would interesting to compare the stats.
Vitaliy S.
Veeam Software
 
Posts: 19570
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Direct Access VEEAM speed vs. 3par

Veeam Logoby Vitaliy S. » Mon Mar 07, 2016 12:58 pm

m1kkel wrote:Would it help to determine if 3åpar is the issue, by simply creating a test lun there, initializing it on our proxy, placing a VBK file there, and copying it back to the proxy repository? DOes that simulate a full backup?

Not exactly, since copying data between the LUNs and retrieving it via vStorage API is a bit different.
Vitaliy S.
Veeam Software
 
Posts: 19570
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Direct Access VEEAM speed vs. 3par

Veeam Logoby m1kkel » Mon Mar 07, 2016 1:04 pm

Ok.
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

Re: Direct Storage Access FC - slow

Veeam Logoby m1kkel » Mon Mar 07, 2016 1:05 pm

Jep.

Here are the results from active full of the same server, hot add, virtuel proxy.
07-03-2016 13:39:44 :: Hard disk 1 (95,0 GB) 49,0 GB read at 143 MB/s [CBT]
07-03-2016 13:46:33 :: Busy: Source 97% > Proxy 41% > Network 13% > Target 0%

...
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

Re: Direct Storage Access FC - slow

Veeam Logoby Gostev » Mon Mar 07, 2016 1:48 pm

Merged duplicate discussions.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Direct Storage Access FC - slow

Veeam Logoby foggy » Mon Mar 07, 2016 2:20 pm

Mikkel, I'd recommend opening a ticket with a support team for a closer investigation, including your deployment review and probably running some performance tests. Involving storage vendor might also be a good step.

Also, haven't you thought of using storage snapshots to backup from?
foggy
Veeam Software
 
Posts: 14752
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Direct Storage Access FC - slow

Veeam Logoby m1kkel » Mon Mar 07, 2016 2:26 pm

Ok, i will open a ticket.

I have thought of storage snapshots, unfortunately that requires the entreprise plus license as far as i am concerned..
m1kkel
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Thu Nov 06, 2014 8:01 pm
Full Name: Mikkel Nielsen

Next

Return to VMware vSphere



Who is online

Users browsing this forum: alanbolte and 9 guests

cron