Comprehensive data protection for all workloads
Post Reply
namiko78
Expert
Posts: 117
Liked: 4 times
Joined: Mar 03, 2011 1:49 pm
Full Name: Steven Stirling
Contact:

Optimal storage config for transforms

Post by namiko78 »

Is there a best practise guide for repository storage? I've recently moved my production job to an equallogic ps 6000 configured in a raid 10, hoping for faster end of week transform, but have been disappointed thus far. It looks like the bottleneck is the array still, with windows reporting the disk active time at 99% most of the time. The array itself reports low usage, 20MB/S and avgerage latency of 6.5ms

I've formateed the NTFS partition with 64k cluster size, are there any other recommendations you can thing of? It's an iSCSI connection to the array over x2 1gb nics, and they're only about 30% used on average. When i copied the raw files to the array it was over 200mb/s although i know not to expect such speed with the randomness of the transform.

Thank you
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Optimal storage config for transforms

Post by tsightler »

Is it just 16 SATA disks in RAID 10? Do you have more than 1 transform going at a time? Are there backups running at the same time? How long is it actually taking? How big are the full/incremental files?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Optimal storage config for transforms

Post by dellock6 »

Steven 20 MB and 200 mb are almost the same (200 mb = 25 MB), unless there is a typo in one of the two numbers.

If it's 200 MBs, at least Transform requires 4 IO per byte, so this number can lower down to 50 MBs. I know your 20 is even lower, but it's just to set a max you can expect...

If I get Tom's first question correctly, the doubt is that with SATA disks spinning at 7200 rpm, seek times are slow, and the randomness of Trasforms is making this problem even worse.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
namiko78
Expert
Posts: 117
Liked: 4 times
Joined: Mar 03, 2011 1:49 pm
Full Name: Steven Stirling
Contact:

Re: Optimal storage config for transforms

Post by namiko78 »

It is 14 7200RPM SATA disks. (2 are hotspare)
No other jobs to that volume, although the repository has other jobs going to another array. No other activity on that volume or the ISCSI adapters. The VBK is 1.6 TB and the VIB's are on average 100gig. Actually took 28 hrs, which is much faster then it was when it was doing the same operation on the non equallogic array.

28 hrs is usable, but i'm wondering if i can shorten this somehow.

Oh and it's MB

Thanks
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Optimal storage config for transforms

Post by Gostev »

The only three way to shorten transform times are:

1. User faster RPM hard drives.

2. Increase the amount of hard drives.

3. Change RAID level to one with less I/O impact.
By impact: (most) RAID6 > RAID5 > RAID10 (least)
namiko78
Expert
Posts: 117
Liked: 4 times
Joined: Mar 03, 2011 1:49 pm
Full Name: Steven Stirling
Contact:

Re: Optimal storage config for transforms

Post by namiko78 »

Ok, thanks.
jbarrow.viracoribt
Expert
Posts: 184
Liked: 18 times
Joined: Feb 15, 2013 9:31 pm
Full Name: Jonathan Barrow
Contact:

[MERGED] I need to speed up transformations...

Post by jbarrow.viracoribt »

So, my transformations on my backup jobs which are configured for incremental, synthetic fulls (every day per week), 2 copy retention, are taking 12-14 hours to finish. How can I speed this up? Is it all on the storage device where the transformation is taking place? If so, would something like SSD storage or Nimble storage speed things up over traditional spinning disks?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Optimal storage config for transforms

Post by dellock6 »

Hi Jonathan, yes the transform happens in the repository, so any increase in the repository performances is likely to reduce the time needed for fransformations.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
b.vanhaastrecht
Service Provider
Posts: 833
Liked: 154 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

[MERGED] Slow transform with large incremental

Post by b.vanhaastrecht »

Hi,

We have a setup with multiple copy jobs running between two repository's. The connection is 1GbE. One of the jobs has 1.1TB of change per day (on 2.3TB to backup data). This is because of SQL dumps beeing made in each of these VM's, so we have a lot of changed data. This copy takes just a few hours, but the transform takes a very long time, about 22hours. (As for a 150GB change on a 1.3TB job, its 5 hours of transforming.)

The target storage is a HP DL180 with 24GB RAM, 10x 2TB SAS MDL DP 7.2K drives configured in a RAID 50 connected to a P212 with 2GB Cache. The veeamagent.exe shows a avarage of 15MB/s read and the same in write IO, continuesly. I think this is relatively slow for the configured strorage. (CPU/RAM is not congested).

I'm looking for ways to improve the speed of this transform.
- Backup/Copy job properties?
- Persioduc full backup, could this help speed up the transform?
- Other RAID setup? Whats best for transforms?
- Faster disks (logical, but not really an option).
- Other suggestions?

Thanks in advance!
Regards,
Bastiaan
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Optimal storage config for transforms

Post by foggy »

Bastiaan, target repository should provide enough IOPS to be able to cope with a very I/O intensive transform operation, so the basic recommendation is to have faster disks in RAID10.

Just to check a few things, do you run just a single backup copy job at the same time against your repository? What type of Veeam B&R repository it is and what are the bottleneck stats for the job?
b.vanhaastrecht
Service Provider
Posts: 833
Liked: 154 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Optimal storage config for transforms

Post by b.vanhaastrecht »

Hi Foggy. I understand I need to design the storage to match the transforms needs, my question, what kind of IOPS; sequential, random, burst, in what percentage? The current performance of 15MB/s R/W on the veeamagent.exe seems a bit slow for the storage it's runnign on. Or perhaps the data is defragmentated. The copy job I'm referring about is a the only one running at that timeframe.
Source repository: Windows Server 2012 R2 with local storage repository. 16GB RAM. 10x 2TB SAS MDL DP 7.2K RAID 6.
Destination repository: Windows Server 2008 R2 with local storage repository. 24GB RAM. 10x 2TB SAS MDL DP 7.2K RAID 50.

Both latest B&R version. 1 GbE network connection between them. All copy jobs give Network as bothleneck. (Transfer speeds in the copy jobs are almost reaching the full 1GbE bandwidth)
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Optimal storage config for transforms

Post by foggy »

b.vanhaastrecht wrote:Hi Foggy. I understand I need to design the storage to match the transforms needs, my question, what kind of IOPS; sequential, random, burst, in what percentage?
Transform is heavy random I/O. In addition to the referred post, backup copy job transformation does 33% less I/O operations than reversed incremental.

That said, in your particular case storage performance does not seem to be the limiting factor, since:
b.vanhaastrecht wrote:All copy jobs give Network as bothleneck. (Transfer speeds in the copy jobs are almost reaching the full 1GbE bandwidth)
You did not mention how the repositories were added to the Veeam B&R console, hope you're not using CIFS-type repository, are you?
b.vanhaastrecht
Service Provider
Posts: 833
Liked: 154 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Optimal storage config for transforms

Post by b.vanhaastrecht »

foggy wrote:You did not mention how the repositories were added to the Veeam B&R console, hope you're not using CIFS-type repository, are you?
Sorry, I'ts added as type Windows. Not CIFS.

I doubt if the transform is taken in account with the posible bothleneck check. When I consult this job log and count all VM process times, it takes 6 hours and 8 minutes to transfer 1.1TB of data. And the transform takes 26 hours and 44 minutes. (1.1TB change on 2.3TB of data)
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Optimal storage config for transforms

Post by foggy »

Correct, bottleneck stats are generated on completion of the data transfer period.
b.vanhaastrecht
Service Provider
Posts: 833
Liked: 154 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Optimal storage config for transforms

Post by b.vanhaastrecht »

Correct, bottleneck stats are generated on completion of the data transfer period.
:D so why do you ask if my concern regards slow transforming in a copy jobs :D

So there are no other recommendations? 15MB R/W is concidered "normal" on my setup?
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Optimal storage config for transforms

Post by foggy »

I just realized that later, when you asked. Unlike backup copy jobs, in reversed incremental mode transform is a part of the Target component and is reflected in bottleneck stats.

The only possible ways to improve transform times are listed above. I wonder if switching to RAID10 will help.
Post Reply

Who is online

Users browsing this forum: Google [Bot], legil.miguel and 220 guests