Incremental to rollback transform: possible bottlenecks?

[trafsta]

Hmmm, 14 hours for latest transform on Friday Still seems way too long for a 400GB to 600GB file.

[J1mbo]

What's your storage subsystem consist of?

[trafsta]

512mb DDR3 cache RAID6 12 x 3.5" 7krpm 2TB drives (local storage on a Dell PE box). 24GB ram on server.

Very little I/O occurs during the 14 to 30+ hours... it's not saturated at all. As a test during the 14+ hours I can copy hundreds of gigabytes of data on this server manually and the copy goes through speedily. Looking at performance monitors shows very little I/O during the vast majority of the transform. We're talking about a 400GB VBK file and about 250-275GB of VRB files from the previous 6 days that gets transformed, it shouldn't take 14-30+ hours. The latest patch did decrease it down to < 20 hours but its still 14-20 hours, whereas it was 22-35ish hours before the latest custom patch if memory serves...

[J1mbo]

Bear in mind that the transform process involves masses of small, random writes. With RAID-6 and SATA drives, it's not going to be quick. Transferring big files is completely different since the controllers caching almost eliminates the write penalty (mostly, by committing full stripes).

[trafsta]

I've been told that many times, but honestly it's not what is happening here.

Also, yesterday support indicated that they have found a way to speed it up more with a different algorithm. I have installed their new custom 6.1 agent on the server and have a transform scheduled for tonight. Hopefully it works like they think it will

[trafsta]

Darn, 18.5hrs was the latest transform duration

[J1mbo]

Very little I/O occurs during the 14 to 30+ hours...

I just wondered how that IO rate was being measured?

[Gostev]

Darn, 18.5hrs was the latest transform duration

So, down up to 2 times from your original timings of 30 hours? Pretty nice impovement, if you ask me

[trafsta]

Not if you ask me...

I used perfmon to monitor I/O.

Going to just stop using transforms I think. It is the only thing that is slow.

[Cokovic]

512mb DDR3 cache RAID6 12 x 3.5" 7krpm 2TB drives (local storage on a Dell PE box). 24GB ram on server.

Very little I/O occurs during the 14 to 30+ hours... it's not saturated at all. As a test during the 14+ hours I can copy hundreds of gigabytes of data on this server manually and the copy goes through speedily. Looking at performance monitors shows very little I/O during the vast majority of the transform. We're talking about a 400GB VBK file and about 250-275GB of VRB files from the previous 6 days that gets transformed, it shouldn't take 14-30+ hours. The latest patch did decrease it down to < 20 hours but its still 14-20 hours, whereas it was 22-35ish hours before the latest custom patch if memory serves...

Just a little question regarding your controller. Does it have a battery backed up write cache? Cause makes a huge difference in performance. If not you can usually override it in the controller settings (but just for testing purposes as it is not recommended to activate it without at least an UPS to prevent data loss in case of a power outage).

[J1mbo]

To second that re write cache, but also the counters being used on perfmon are important; helpful will be physical disk % Disk Time, Avg. Disk Bytes/Transfer, Current Disk Queue Length and Disk Transfers/Sec as it's a random workload.

Since RAID-6 has a 6 IO write penalty, with 12 drives the array should be able to sustain something like 150 to 300 IOPS in writes and maybe 750 to 1500 in reads (with a queue depth of at least 10), so I'd expect transfers/sec sustained to be something like 300-400 during the transform.

Hope that helps.

[trafsta]

Yes it has a battery backed up write cache and it is enabled.

I'm on vacation right now but I'll try setting up the perfmon ahead of time with your specified counters to record the timeframe that the transforms takes place and we'll see what it comes up with.

[averylarry]

Not if you ask me...
I used perfmon to monitor I/O.

Going to just stop using transforms I think. It is the only thing that is slow.

I'll support you on this.

I gave up on transforms a long time ago.

[Gostev]

I gave up on transforms a long time ago.

Try it once Patch 1 is out - we are seeing performance improvement of up to a few times depending on the storage!

[baa]

Good day!
I had a similar situation. Incremental backup.

http://www.freeimagehosting.net/p28q5

[Vitaliy S.]

Hi Albert, what version of Veeam B&R are you running? What is the target repository type?

[baa]

Veeam BR - Ver 6.5.0.128
Backup to local disk D:
Now set up a full backup. Let's see what happens.

[foggy]

Is it a local disk of a Veeam B&R machine? Is it physical or virtual? Tell us a bit more on your setup, please.