Optimal storage config for transforms

[namiko78]

Is there a best practise guide for repository storage? I've recently moved my production job to an equallogic ps 6000 configured in a raid 10, hoping for faster end of week transform, but have been disappointed thus far. It looks like the bottleneck is the array still, with windows reporting the disk active time at 99% most of the time. The array itself reports low usage, 20MB/S and avgerage latency of 6.5ms

I've formateed the NTFS partition with 64k cluster size, are there any other recommendations you can thing of? It's an iSCSI connection to the array over x2 1gb nics, and they're only about 30% used on average. When i copied the raw files to the array it was over 200mb/s although i know not to expect such speed with the randomness of the transform.

Thank you

[tsightler]

Is it just 16 SATA disks in RAID 10? Do you have more than 1 transform going at a time? Are there backups running at the same time? How long is it actually taking? How big are the full/incremental files?

[dellock6]

Steven 20 MB and 200 mb are almost the same (200 mb = 25 MB), unless there is a typo in one of the two numbers.

If it's 200 MBs, at least Transform requires 4 IO per byte, so this number can lower down to 50 MBs. I know your 20 is even lower, but it's just to set a max you can expect...

If I get Tom's first question correctly, the doubt is that with SATA disks spinning at 7200 rpm, seek times are slow, and the randomness of Trasforms is making this problem even worse.

[namiko78]

It is 14 7200RPM SATA disks. (2 are hotspare)
No other jobs to that volume, although the repository has other jobs going to another array. No other activity on that volume or the ISCSI adapters. The VBK is 1.6 TB and the VIB's are on average 100gig. Actually took 28 hrs, which is much faster then it was when it was doing the same operation on the non equallogic array.

28 hrs is usable, but i'm wondering if i can shorten this somehow.

Oh and it's MB

Thanks

[Gostev]

The only three way to shorten transform times are:

1. User faster RPM hard drives.

2. Increase the amount of hard drives.

3. Change RAID level to one with less I/O impact.
By impact: (most) RAID6 > RAID5 > RAID10 (least)

[namiko78]

Ok, thanks.

[jbarrow.viracoribt]

So, my transformations on my backup jobs which are configured for incremental, synthetic fulls (every day per week), 2 copy retention, are taking 12-14 hours to finish. How can I speed this up? Is it all on the storage device where the transformation is taking place? If so, would something like SSD storage or Nimble storage speed things up over traditional spinning disks?

[dellock6]

Hi Jonathan, yes the transform happens in the repository, so any increase in the repository performances is likely to reduce the time needed for fransformations.

[b.vanhaastrecht]

Hi,

We have a setup with multiple copy jobs running between two repository's. The connection is 1GbE. One of the jobs has 1.1TB of change per day (on 2.3TB to backup data). This is because of SQL dumps beeing made in each of these VM's, so we have a lot of changed data. This copy takes just a few hours, but the transform takes a very long time, about 22hours. (As for a 150GB change on a 1.3TB job, its 5 hours of transforming.)

The target storage is a HP DL180 with 24GB RAM, 10x 2TB SAS MDL DP 7.2K drives configured in a RAID 50 connected to a P212 with 2GB Cache. The veeamagent.exe shows a avarage of 15MB/s read and the same in write IO, continuesly. I think this is relatively slow for the configured strorage. (CPU/RAM is not congested).

I'm looking for ways to improve the speed of this transform.
- Backup/Copy job properties?
- Persioduc full backup, could this help speed up the transform?
- Other RAID setup? Whats best for transforms?
- Faster disks (logical, but not really an option).
- Other suggestions?

Thanks in advance!
Regards,
Bastiaan

[foggy]

Bastiaan, target repository should provide enough IOPS to be able to cope with a very I/O intensive transform operation, so the basic recommendation is to have faster disks in RAID10.

Just to check a few things, do you run just a single backup copy job at the same time against your repository? What type of Veeam B&R repository it is and what are the bottleneck stats for the job?

[b.vanhaastrecht]

Hi Foggy. I understand I need to design the storage to match the transforms needs, my question, what kind of IOPS; sequential, random, burst, in what percentage? The current performance of 15MB/s R/W on the veeamagent.exe seems a bit slow for the storage it's runnign on. Or perhaps the data is defragmentated. The copy job I'm referring about is a the only one running at that timeframe.
Source repository: Windows Server 2012 R2 with local storage repository. 16GB RAM. 10x 2TB SAS MDL DP 7.2K RAID 6.
Destination repository: Windows Server 2008 R2 with local storage repository. 24GB RAM. 10x 2TB SAS MDL DP 7.2K RAID 50.

Both latest B&R version. 1 GbE network connection between them. All copy jobs give Network as bothleneck. (Transfer speeds in the copy jobs are almost reaching the full 1GbE bandwidth)

[foggy]

Hi Foggy. I understand I need to design the storage to match the transforms needs, my question, what kind of IOPS; sequential, random, burst, in what percentage?

Transform is heavy random I/O. In addition to the referred post, backup copy job transformation does 33% less I/O operations than reversed incremental.

That said, in your particular case storage performance does not seem to be the limiting factor, since:

All copy jobs give Network as bothleneck. (Transfer speeds in the copy jobs are almost reaching the full 1GbE bandwidth)

You did not mention how the repositories were added to the Veeam B&R console, hope you're not using CIFS-type repository, are you?

[b.vanhaastrecht]

You did not mention how the repositories were added to the Veeam B&R console, hope you're not using CIFS-type repository, are you?

Sorry, I'ts added as type Windows. Not CIFS.

I doubt if the transform is taken in account with the posible bothleneck check. When I consult this job log and count all VM process times, it takes 6 hours and 8 minutes to transfer 1.1TB of data. And the transform takes 26 hours and 44 minutes. (1.1TB change on 2.3TB of data)

[foggy]

Correct, bottleneck stats are generated on completion of the data transfer period.

[b.vanhaastrecht]

Correct, bottleneck stats are generated on completion of the data transfer period.

so why do you ask if my concern regards slow transforming in a copy jobs

So there are no other recommendations? 15MB R/W is concidered "normal" on my setup?

[foggy]

I just realized that later, when you asked. Unlike backup copy jobs, in reversed incremental mode transform is a part of the Target component and is reflected in bottleneck stats.

The only possible ways to improve transform times are listed above. I wonder if switching to RAID10 will help.