Comprehensive data protection for all workloads
trafsta
Enthusiast
Posts: 43
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by trafsta » Jul 09, 2012 1:56 am

Hmmm, 14 hours for latest transform on Friday :( Still seems way too long for a 400GB to 600GB file.

J1mbo
Expert
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by J1mbo » Jul 10, 2012 8:19 am

What's your storage subsystem consist of?

trafsta
Enthusiast
Posts: 43
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by trafsta » Jul 11, 2012 5:54 am

512mb DDR3 cache RAID6 12 x 3.5" 7krpm 2TB drives (local storage on a Dell PE box). 24GB ram on server.

Very little I/O occurs during the 14 to 30+ hours... it's not saturated at all. As a test during the 14+ hours I can copy hundreds of gigabytes of data on this server manually and the copy goes through speedily. Looking at performance monitors shows very little I/O during the vast majority of the transform. We're talking about a 400GB VBK file and about 250-275GB of VRB files from the previous 6 days that gets transformed, it shouldn't take 14-30+ hours. The latest patch did decrease it down to < 20 hours but its still 14-20 hours, whereas it was 22-35ish hours before the latest custom patch if memory serves...

J1mbo
Expert
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by J1mbo » Jul 11, 2012 7:21 am

Bear in mind that the transform process involves masses of small, random writes. With RAID-6 and SATA drives, it's not going to be quick. Transferring big files is completely different since the controllers caching almost eliminates the write penalty (mostly, by committing full stripes).

trafsta
Enthusiast
Posts: 43
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by trafsta » Jul 13, 2012 1:25 pm

I've been told that many times, but honestly it's not what is happening here.

Also, yesterday support indicated that they have found a way to speed it up more with a different algorithm. I have installed their new custom 6.1 agent on the server and have a transform scheduled for tonight. Hopefully it works like they think it will :)

trafsta
Enthusiast
Posts: 43
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by trafsta » Jul 17, 2012 12:41 pm

Darn, 18.5hrs was the latest transform duration :(

J1mbo
Expert
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by J1mbo » Jul 17, 2012 1:19 pm

trafsta wrote:Very little I/O occurs during the 14 to 30+ hours...
I just wondered how that IO rate was being measured?

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Gostev » Jul 17, 2012 2:13 pm

trafsta wrote:Darn, 18.5hrs was the latest transform duration :(
So, down up to 2 times from your original timings of 30 hours? Pretty nice impovement, if you ask me :D

trafsta
Enthusiast
Posts: 43
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by trafsta » Jul 17, 2012 4:31 pm

Not if you ask me...

I used perfmon to monitor I/O.

Going to just stop using transforms I think. It is the only thing that is slow.

Cokovic
Expert
Posts: 295
Liked: 59 times
Joined: Sep 06, 2011 8:45 am
Full Name: Haris Cokovic
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Cokovic » Jul 17, 2012 10:50 pm

trafsta wrote:512mb DDR3 cache RAID6 12 x 3.5" 7krpm 2TB drives (local storage on a Dell PE box). 24GB ram on server.

Very little I/O occurs during the 14 to 30+ hours... it's not saturated at all. As a test during the 14+ hours I can copy hundreds of gigabytes of data on this server manually and the copy goes through speedily. Looking at performance monitors shows very little I/O during the vast majority of the transform. We're talking about a 400GB VBK file and about 250-275GB of VRB files from the previous 6 days that gets transformed, it shouldn't take 14-30+ hours. The latest patch did decrease it down to < 20 hours but its still 14-20 hours, whereas it was 22-35ish hours before the latest custom patch if memory serves...
Just a little question regarding your controller. Does it have a battery backed up write cache? Cause makes a huge difference in performance. If not you can usually override it in the controller settings (but just for testing purposes as it is not recommended to activate it without at least an UPS to prevent data loss in case of a power outage).

J1mbo
Expert
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by J1mbo » Jul 18, 2012 7:15 am

To second that re write cache, but also the counters being used on perfmon are important; helpful will be physical disk % Disk Time, Avg. Disk Bytes/Transfer, Current Disk Queue Length and Disk Transfers/Sec as it's a random workload.

Since RAID-6 has a 6 IO write penalty, with 12 drives the array should be able to sustain something like 150 to 300 IOPS in writes and maybe 750 to 1500 in reads (with a queue depth of at least 10), so I'd expect transfers/sec sustained to be something like 300-400 during the transform.

Hope that helps.

trafsta
Enthusiast
Posts: 43
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by trafsta » Jul 18, 2012 8:35 pm

Yes it has a battery backed up write cache and it is enabled.

I'm on vacation right now but I'll try setting up the perfmon ahead of time with your specified counters to record the timeframe that the transforms takes place and we'll see what it comes up with.

averylarry
Expert
Posts: 261
Liked: 28 times
Joined: Mar 22, 2011 7:43 pm
Full Name: Ted
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by averylarry » Jul 19, 2012 4:08 pm

trafsta wrote:Not if you ask me...
I used perfmon to monitor I/O.

Going to just stop using transforms I think. It is the only thing that is slow.
I'll support you on this.

I gave up on transforms a long time ago.

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Gostev » Jul 22, 2012 9:04 pm

averylarry wrote:I gave up on transforms a long time ago.
Try it once Patch 1 is out - we are seeing performance improvement of up to a few times depending on the storage!

baa
Novice
Posts: 9
Liked: never
Joined: May 15, 2012 7:16 am
Full Name: Albert
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by baa » May 07, 2013 6:03 am

Good day!
I had a similar situation. Incremental backup.

http://www.freeimagehosting.net/p28q5

Vitaliy S.
Product Manager
Posts: 22777
Liked: 1526 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Vitaliy S. » May 07, 2013 9:48 am

Hi Albert, what version of Veeam B&R are you running? What is the target repository type?

baa
Novice
Posts: 9
Liked: never
Joined: May 15, 2012 7:16 am
Full Name: Albert
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by baa » May 07, 2013 9:59 am

Veeam BR - Ver 6.5.0.128
Backup to local disk D:
Now set up a full backup. Let's see what happens.

foggy
Veeam Software
Posts: 18037
Liked: 1533 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by foggy » May 07, 2013 10:11 am

Is it a local disk of a Veeam B&R machine? Is it physical or virtual? Tell us a bit more on your setup, please.

Post Reply

Who is online

Users browsing this forum: No registered users and 17 guests