RAID6 or RAID10 for backup storage on local disk array?

[davidb1234]

Veeam backup server is a 12 core monster with 12GB of RAM connected to an MSA60 with 12 x 2TB disks.

We have 2 ESXi 5 hosts connected 8GB FC SAN multipathed. The Veeam server is directly connected to the SAN by 8GB FC.

We have total of 4TB of VMs to back up and are planning on the Reverse Incremental model where it leaves us with a full intact VM backup every night.

If I go RAID10 I'll have about 11TB of Usable space for backups.

If I go RAID6 I'll have about 19TB of Usable space for backups.

Which way did you go and why? Any recommendations? Any regrets? Is RAID10 overkill?

We will also be replicating to a hotsite later this year but I don't think that plays a part in this decision.

[averylarry]

I'd go RAID6. You can lose any 2 disks and still have all your data. RAID10 -- if you lose the correct 2 disks, everything is toast.

It seems highly unlikely that your backup storage will be the bottleneck compared with production storage/compression/deduplication etc. I doubt that the likely performance decrease with RAID6 would be noticable. Besides -- the number 1 rule of thumb for storage performance is number of spindles. Since you'll have the same number of spindles either way . . .

(And for those purists out there, yes -- I'm massively simplifying.)

[TimeKnight]

I have a similar Veeam Server setup (Dell R510 with a Single CPU Xeon X5650 (6 cores + HT) 2.67GHz and 12.0 GB of RAM. For storage we have 12 2.0TB SATA HDD in a RAID 6 configuration giving us about 18TB of space.)

I seem to be having issues with the Transformation process (I only do it once a week) and it takes 35+ hours.

I am starting to think that the low IOps of 7k disks will require RAID 10 for the amount of random IO that is required for the Transformation Process.

[averylarry]

Despite what Veeam says, I think the tranformation process got messed up in v.6. I gave up on it entirely and switched to reverse incremental backups. I don't think (particularly when non-scientifically comparing to v.5) that the slow transformation in v.6 can be entirely explained by system hardware/storage etc. (I'm thinking it's a memory/pagefile problem.)

[Gostev]

Most customers recommend RAID10 for performance considerations (especially when using transform or reversed increments).

@Ted, transformation uses exact same code as the reversed incremental backup... same function that applies incremental data into VBK, and saves replaced data to VRB. So, I am curious why reversed incremental works better for you.

[averylarry]

I started doing transforms on the weekends. That worked fine in v.5. But, as is typical, my overall backup system/strategy is never changing. Eventually I was so happy with v.5 backups that I started doing a mid-day backup. Incremental backups kept it quick and fast. Snapshots didn't cause any problems. Weekly transforms completed in less than a day.

Fast forward many months from then. I now have v.6 instead of v.5. I now have vSphere 5 instead of vSphere 4. I now use vmfs5 instead of vmfs4. My backup datastore is a completely different/separate SAN with a single LUN and native vmfs5. I have close to double the amount of data to backup.

So really -- too many changes to necessarily blame any single thing. With v.6, tranforms started taking days and would extend into the work week. So I added another backup job in the very early morning and did transforms daily instead of weekly (since tranforms run on the 1st job that runs in any given day). That helped, but still would cause problems because that job has to wait for replication jobs to finish etc. Additionally, snapshots are now causing minor issues during the day. So I basically ended up dumping the mid-day backup and switched to reverse incremental which made the early morning backup job superfluous. Now my backup is a more typical daily job with reverse incremental for everything, and everything also replicates to my WAN.


Also -- (this is simplified) transformation reads from the .vib file and writes to .vrb and .vbk. Reversed incremental reads from the source and writes to .vrb and .vbk. That allows for 2 separate LUNS. Reversed incremental is not 1/3 read, 2/3 write random I/O to a single LUN where transformation is (again, simplified). My backup LUN is a lower tier of storage compared to my production LUNs.

[davidb1234]

Obviously if we back up every night with reverse incremental we need the transform process to complete in a reasonable amount of time since it will be done every single day.

From these responses I am actually worried now. I figured a few hours maybe to transform after the incremental runs but it seems that part of it takes a LONG time. Is that really the case???

We have 4TB of total VMs to back up however the daily change rate is about 500GB. Can anyone give me a rough ballpart on what they are seeing for transform times when using reverse incremental leaving us with a full intact VM each day?

What good is reverse incremental if the VM dies in the middle of the day and the transform isn't even done running yet from the previous nights backup?

I'd love to do RAID6 but now I am kind of freaked out by these responses.

[tsightler]

I've worked with a lot of users that are happily using RAID6 for their backup targets. Be sure to choose a good stripe size that makes sense, i.e. generally larger stripe sizes (64K or greater) work best for storing Veeam backups since the blocks are typically large.

However, having such a high change rate is going to work against you. A 500GB change rate for 4TB of data is quite high compared to the typical environment that I see and means that a transform will need to move at least 2TB of data for each nights backup, all on the target storage. The ability to do this depends completely on your targets storage ability to meet the I/O requirements of moving this much data at random I/O. I'd suggest just testing it, it will only take one night of transform before you know whether it will work or not.

[davidb1234]

I determined that change rate from what backup exec is currently grabbing.

258GB Exchange Databases - Full Backup each night
200GB of SQL databases and changed files on the file server(mostly SQL database .BAK files)

I am guessing that this data change rate wouldn't apply the same way to veeam. I plan to use VEEAM to do application aware processing of SQL full backups and exchange. Our backups are extremely ineffecient right now.

Does Veeam truely need to grab then entirely of the Exchange OR SQL databases or will it do block level backups of just the incremental changes each night?

[Gostev]

Only changed blocks, of course.

[averylarry]

reversed incremental is NOT an incremental followed by a transform. It's just a single process that modifies the .vib file while also creating the .vrb file. In my opinion/observation -- reverse incremental doesn't take as long as an incremental followed by a transform.

My Exchange server is about 800Gb of databases with 200 users hitting it pretty hard. My reversed incremental file sizes are about 11Gb (the weekend is larger when the weekly maintenance stuff runs on the Exchange databases).

[Gostev]

I never said this is the same process, I said it is the same code. Backup files interaction is isolated to a few functions which every backup mode uses. So, I would not expect different transform modes to provide drastically different performance...

[averylarry]

I was referring to David's previous post --

"What good is reverse incremental if the VM dies in the middle of the day and the transform isn't even done running yet from the previous nights backup?"

which implies a misunderstanding of reverse incremental as an incremental followed by a transform.

[Gostev]

Oh, makes sense now. You are right.

[albertwt]

Hi All,

So in this case with the latest edition of Veeam 9.0, does the below suggestion is correct:

If most or all of the backup job is still using Reverse Incremental, then use RAID 10
If converting to or most of the backup job is using Forward Incremental, then use RAID 6

As for the Strip / Full stripe size, choose the largest possible value that is supported by the Storage controller eg.
Strip Size: 256 KB
Full Stripe Size: 1024 KB

Can anyone confirm if that's the correct case ?

[dellock6]

Hi Albert,
these rules of thumb are correct, but remember that each storage device has its own features and characteristics, like the cache size of the controller, the way blocks are spread around the disks, and so on. Worth a check with the specific vendor sometimes.

[albertwt]

Cool,

Many thanks for the clarification Lucca.
Your website is always useful and has been a great help to us all.

[albertwt]

Hi Luca,

Does splitting the Backup Repo into multiple smaller LUNs can improve the backup performance ?
rather than one massive big LUNs for multiple backup job ?

[dellock6]

Depends on the underlying array, if all the LUNs are coming from the same raid group for example, the overall performance are always the same, just divided by the number of created volumes.
Be careful on massive luns, rebuild times of degraded raid can be painful.

Luca (with one C )

[albertwt]

Hi All,

I'm currently implementing Disk to Disk backup system within the Data Center infrastructure.

The backend of the backup server is attached using HP D2600 (12x 3 TB 7200 rpm SATA) and D2700 (25x 500 GB 7200rpm SAS) disks.

So which LUN setting is better given the below settings:



Thanks.

[nielsengelen]

Looking at the info RAID50 should be better performance wise.

[albertwt]

Great,

Thanks for the assistance Niels.

The reason that LUN was using RAID 6 or RAID50 was due to the number of disks is more than 12 in each RAID group.

Hence I will destroy that LUN on RAID6 and change it into RAID 50 with the largest strip size possible.