-
- Enthusiast
- Posts: 45
- Liked: 2 times
- Joined: Apr 10, 2012 5:50 pm
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Hmmm, 14 hours for latest transform on Friday Still seems way too long for a 400GB to 600GB file.
-
- Veteran
- Posts: 261
- Liked: 29 times
- Joined: May 03, 2011 12:51 pm
- Full Name: James Pearce
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
What's your storage subsystem consist of?
-
- Enthusiast
- Posts: 45
- Liked: 2 times
- Joined: Apr 10, 2012 5:50 pm
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
512mb DDR3 cache RAID6 12 x 3.5" 7krpm 2TB drives (local storage on a Dell PE box). 24GB ram on server.
Very little I/O occurs during the 14 to 30+ hours... it's not saturated at all. As a test during the 14+ hours I can copy hundreds of gigabytes of data on this server manually and the copy goes through speedily. Looking at performance monitors shows very little I/O during the vast majority of the transform. We're talking about a 400GB VBK file and about 250-275GB of VRB files from the previous 6 days that gets transformed, it shouldn't take 14-30+ hours. The latest patch did decrease it down to < 20 hours but its still 14-20 hours, whereas it was 22-35ish hours before the latest custom patch if memory serves...
Very little I/O occurs during the 14 to 30+ hours... it's not saturated at all. As a test during the 14+ hours I can copy hundreds of gigabytes of data on this server manually and the copy goes through speedily. Looking at performance monitors shows very little I/O during the vast majority of the transform. We're talking about a 400GB VBK file and about 250-275GB of VRB files from the previous 6 days that gets transformed, it shouldn't take 14-30+ hours. The latest patch did decrease it down to < 20 hours but its still 14-20 hours, whereas it was 22-35ish hours before the latest custom patch if memory serves...
-
- Veteran
- Posts: 261
- Liked: 29 times
- Joined: May 03, 2011 12:51 pm
- Full Name: James Pearce
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Bear in mind that the transform process involves masses of small, random writes. With RAID-6 and SATA drives, it's not going to be quick. Transferring big files is completely different since the controllers caching almost eliminates the write penalty (mostly, by committing full stripes).
-
- Enthusiast
- Posts: 45
- Liked: 2 times
- Joined: Apr 10, 2012 5:50 pm
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
I've been told that many times, but honestly it's not what is happening here.
Also, yesterday support indicated that they have found a way to speed it up more with a different algorithm. I have installed their new custom 6.1 agent on the server and have a transform scheduled for tonight. Hopefully it works like they think it will
Also, yesterday support indicated that they have found a way to speed it up more with a different algorithm. I have installed their new custom 6.1 agent on the server and have a transform scheduled for tonight. Hopefully it works like they think it will
-
- Enthusiast
- Posts: 45
- Liked: 2 times
- Joined: Apr 10, 2012 5:50 pm
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Darn, 18.5hrs was the latest transform duration
-
- Veteran
- Posts: 261
- Liked: 29 times
- Joined: May 03, 2011 12:51 pm
- Full Name: James Pearce
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
I just wondered how that IO rate was being measured?trafsta wrote:Very little I/O occurs during the 14 to 30+ hours...
-
- Chief Product Officer
- Posts: 31816
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
So, down up to 2 times from your original timings of 30 hours? Pretty nice impovement, if you ask metrafsta wrote:Darn, 18.5hrs was the latest transform duration
-
- Enthusiast
- Posts: 45
- Liked: 2 times
- Joined: Apr 10, 2012 5:50 pm
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Not if you ask me...
I used perfmon to monitor I/O.
Going to just stop using transforms I think. It is the only thing that is slow.
I used perfmon to monitor I/O.
Going to just stop using transforms I think. It is the only thing that is slow.
-
- Veteran
- Posts: 295
- Liked: 59 times
- Joined: Sep 06, 2011 8:45 am
- Full Name: Haris Cokovic
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Just a little question regarding your controller. Does it have a battery backed up write cache? Cause makes a huge difference in performance. If not you can usually override it in the controller settings (but just for testing purposes as it is not recommended to activate it without at least an UPS to prevent data loss in case of a power outage).trafsta wrote:512mb DDR3 cache RAID6 12 x 3.5" 7krpm 2TB drives (local storage on a Dell PE box). 24GB ram on server.
Very little I/O occurs during the 14 to 30+ hours... it's not saturated at all. As a test during the 14+ hours I can copy hundreds of gigabytes of data on this server manually and the copy goes through speedily. Looking at performance monitors shows very little I/O during the vast majority of the transform. We're talking about a 400GB VBK file and about 250-275GB of VRB files from the previous 6 days that gets transformed, it shouldn't take 14-30+ hours. The latest patch did decrease it down to < 20 hours but its still 14-20 hours, whereas it was 22-35ish hours before the latest custom patch if memory serves...
-
- Veteran
- Posts: 261
- Liked: 29 times
- Joined: May 03, 2011 12:51 pm
- Full Name: James Pearce
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
To second that re write cache, but also the counters being used on perfmon are important; helpful will be physical disk % Disk Time, Avg. Disk Bytes/Transfer, Current Disk Queue Length and Disk Transfers/Sec as it's a random workload.
Since RAID-6 has a 6 IO write penalty, with 12 drives the array should be able to sustain something like 150 to 300 IOPS in writes and maybe 750 to 1500 in reads (with a queue depth of at least 10), so I'd expect transfers/sec sustained to be something like 300-400 during the transform.
Hope that helps.
Since RAID-6 has a 6 IO write penalty, with 12 drives the array should be able to sustain something like 150 to 300 IOPS in writes and maybe 750 to 1500 in reads (with a queue depth of at least 10), so I'd expect transfers/sec sustained to be something like 300-400 during the transform.
Hope that helps.
-
- Enthusiast
- Posts: 45
- Liked: 2 times
- Joined: Apr 10, 2012 5:50 pm
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Yes it has a battery backed up write cache and it is enabled.
I'm on vacation right now but I'll try setting up the perfmon ahead of time with your specified counters to record the timeframe that the transforms takes place and we'll see what it comes up with.
I'm on vacation right now but I'll try setting up the perfmon ahead of time with your specified counters to record the timeframe that the transforms takes place and we'll see what it comes up with.
-
- Veteran
- Posts: 264
- Liked: 30 times
- Joined: Mar 22, 2011 7:43 pm
- Full Name: Ted
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
I'll support you on this.trafsta wrote:Not if you ask me...
I used perfmon to monitor I/O.
Going to just stop using transforms I think. It is the only thing that is slow.
I gave up on transforms a long time ago.
-
- Chief Product Officer
- Posts: 31816
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Try it once Patch 1 is out - we are seeing performance improvement of up to a few times depending on the storage!averylarry wrote:I gave up on transforms a long time ago.
-
- Novice
- Posts: 9
- Liked: never
- Joined: May 15, 2012 7:16 am
- Full Name: Albert
- Contact:
-
- VP, Product Management
- Posts: 27377
- Liked: 2802 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Hi Albert, what version of Veeam B&R are you running? What is the target repository type?
-
- Novice
- Posts: 9
- Liked: never
- Joined: May 15, 2012 7:16 am
- Full Name: Albert
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Veeam BR - Ver 6.5.0.128
Backup to local disk D:
Now set up a full backup. Let's see what happens.
Backup to local disk D:
Now set up a full backup. Let's see what happens.
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Incremental to rollback transform: possible bottlenecks?
Is it a local disk of a Veeam B&R machine? Is it physical or virtual? Tell us a bit more on your setup, please.
Who is online
Users browsing this forum: Bing [Bot], Semrush [Bot] and 55 guests