Comprehensive data protection for all workloads
trafsta
Enthusiast
Posts: 45
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by trafsta »

Hmmm, 14 hours for latest transform on Friday :( Still seems way too long for a 400GB to 600GB file.
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by J1mbo »

What's your storage subsystem consist of?
trafsta
Enthusiast
Posts: 45
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by trafsta »

512mb DDR3 cache RAID6 12 x 3.5" 7krpm 2TB drives (local storage on a Dell PE box). 24GB ram on server.

Very little I/O occurs during the 14 to 30+ hours... it's not saturated at all. As a test during the 14+ hours I can copy hundreds of gigabytes of data on this server manually and the copy goes through speedily. Looking at performance monitors shows very little I/O during the vast majority of the transform. We're talking about a 400GB VBK file and about 250-275GB of VRB files from the previous 6 days that gets transformed, it shouldn't take 14-30+ hours. The latest patch did decrease it down to < 20 hours but its still 14-20 hours, whereas it was 22-35ish hours before the latest custom patch if memory serves...
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by J1mbo »

Bear in mind that the transform process involves masses of small, random writes. With RAID-6 and SATA drives, it's not going to be quick. Transferring big files is completely different since the controllers caching almost eliminates the write penalty (mostly, by committing full stripes).
trafsta
Enthusiast
Posts: 45
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by trafsta »

I've been told that many times, but honestly it's not what is happening here.

Also, yesterday support indicated that they have found a way to speed it up more with a different algorithm. I have installed their new custom 6.1 agent on the server and have a transform scheduled for tonight. Hopefully it works like they think it will :)
trafsta
Enthusiast
Posts: 45
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by trafsta »

Darn, 18.5hrs was the latest transform duration :(
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by J1mbo »

trafsta wrote:Very little I/O occurs during the 14 to 30+ hours...
I just wondered how that IO rate was being measured?
Gostev
Chief Product Officer
Posts: 31753
Liked: 7258 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Gostev »

trafsta wrote:Darn, 18.5hrs was the latest transform duration :(
So, down up to 2 times from your original timings of 30 hours? Pretty nice impovement, if you ask me :D
trafsta
Enthusiast
Posts: 45
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by trafsta »

Not if you ask me...

I used perfmon to monitor I/O.

Going to just stop using transforms I think. It is the only thing that is slow.
Cokovic
Veteran
Posts: 295
Liked: 59 times
Joined: Sep 06, 2011 8:45 am
Full Name: Haris Cokovic
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Cokovic »

trafsta wrote:512mb DDR3 cache RAID6 12 x 3.5" 7krpm 2TB drives (local storage on a Dell PE box). 24GB ram on server.

Very little I/O occurs during the 14 to 30+ hours... it's not saturated at all. As a test during the 14+ hours I can copy hundreds of gigabytes of data on this server manually and the copy goes through speedily. Looking at performance monitors shows very little I/O during the vast majority of the transform. We're talking about a 400GB VBK file and about 250-275GB of VRB files from the previous 6 days that gets transformed, it shouldn't take 14-30+ hours. The latest patch did decrease it down to < 20 hours but its still 14-20 hours, whereas it was 22-35ish hours before the latest custom patch if memory serves...
Just a little question regarding your controller. Does it have a battery backed up write cache? Cause makes a huge difference in performance. If not you can usually override it in the controller settings (but just for testing purposes as it is not recommended to activate it without at least an UPS to prevent data loss in case of a power outage).
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by J1mbo »

To second that re write cache, but also the counters being used on perfmon are important; helpful will be physical disk % Disk Time, Avg. Disk Bytes/Transfer, Current Disk Queue Length and Disk Transfers/Sec as it's a random workload.

Since RAID-6 has a 6 IO write penalty, with 12 drives the array should be able to sustain something like 150 to 300 IOPS in writes and maybe 750 to 1500 in reads (with a queue depth of at least 10), so I'd expect transfers/sec sustained to be something like 300-400 during the transform.

Hope that helps.
trafsta
Enthusiast
Posts: 45
Liked: 2 times
Joined: Apr 10, 2012 5:50 pm
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by trafsta »

Yes it has a battery backed up write cache and it is enabled.

I'm on vacation right now but I'll try setting up the perfmon ahead of time with your specified counters to record the timeframe that the transforms takes place and we'll see what it comes up with.
averylarry
Veteran
Posts: 264
Liked: 30 times
Joined: Mar 22, 2011 7:43 pm
Full Name: Ted
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by averylarry »

trafsta wrote:Not if you ask me...
I used perfmon to monitor I/O.

Going to just stop using transforms I think. It is the only thing that is slow.
I'll support you on this.

I gave up on transforms a long time ago.
Gostev
Chief Product Officer
Posts: 31753
Liked: 7258 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Gostev »

averylarry wrote:I gave up on transforms a long time ago.
Try it once Patch 1 is out - we are seeing performance improvement of up to a few times depending on the storage!
baa
Novice
Posts: 9
Liked: never
Joined: May 15, 2012 7:16 am
Full Name: Albert
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by baa »

Good day!
I had a similar situation. Incremental backup.

http://www.freeimagehosting.net/p28q5
Vitaliy S.
VP, Product Management
Posts: 27347
Liked: 2785 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by Vitaliy S. »

Hi Albert, what version of Veeam B&R are you running? What is the target repository type?
baa
Novice
Posts: 9
Liked: never
Joined: May 15, 2012 7:16 am
Full Name: Albert
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by baa »

Veeam BR - Ver 6.5.0.128
Backup to local disk D:
Now set up a full backup. Let's see what happens.
foggy
Veeam Software
Posts: 21133
Liked: 2140 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Incremental to rollback transform: possible bottlenecks?

Post by foggy »

Is it a local disk of a Veeam B&R machine? Is it physical or virtual? Tell us a bit more on your setup, please.
Post Reply

Who is online

Users browsing this forum: Bing [Bot], chris.childerhose, Google [Bot], mkretzer, nathanrsafti and 75 guests