Recommendations for backup storage, backup target

[ACC]

RAID6 write penalty is too high IMHO for a storage dedicated to backups

In 15+ years of IT I've never lost a RAID5... All my servers and storage always have firmware updated (both raid card and disk firmware). I use only Dell PowerEdge servers. In the last years I've prepared many Veeam Backup servers with 2 - 3 - 4 and now 6 TB NLSAS disks, so far so good. I always use an hot spare. On a recent Dell PowerEdge T630 where failed a 6TB disk, the automatic hot spare coverage took about 8 hours to complete the rebuild (it was a RAID5 made of nine 6 TB disks)

I can't see why a disk that get intensive writes during backups should have an higher failure rate due to intensives reads in case of array rebuild...

Marco

Consider your self very lucky, i have seen over the years pretty much all, faulty hardware, buggy firmware, multiple disk failures causing raid arrays to brake down, and thats not because our customers buy shitty hardware even A-brand hardware vendors like Dell and HP have their share in storage issues.

Raid as we know and use is in my opinion outdated and we desperately need something better, which can handle those multi TB disks and their issues. One of them is the infamous unrecoverable read error, if a disk has unrecoverable read error rate of 10^14 it means that statistically every 12 TB read from this disk it will encounter a unrecoverable read error. Your 42 TB effective storage would statistically hit 3,5 of those errors in a single read pass. If such error so happens during a rebuild, the controller will halt the rebuild and offline the array, because there is no more parity available to recover from this error. The very lucky admin, can with assistance from the hardware vendor get the array glued back together and get a successful rebuild done, the less lucky ones can get the array online to evacuate as much data as possible and the unfortunate ones get the message that the data is considered lost. Because of this pretty much all vendors doesnt recommend using raid5 especially for business critical data or systems.

[dellock6]

I can't see why a disk that get intensive writes during backups should have an higher failure rate due to intensives reads in case of array rebuild...
Marco

Marco, this is not the problem. The problem is, during a rebuild you are running in degraded state, and if anything happens to another disk during a rebuild you loose data, simple as that. And with large disks the amount of data going into a single disk is higher than before: if in the past I needed 10TB storage with 2TB disk, your raid5 would have been 6 disks, so 1000 IOPS would have been split in 6. Now, with 5TB disk for example, you have 3 disks to build the same 10TB raid. So each disk has to write double the amount of data each day, and because of this the chances to break are higher.

[guitarfish]

In 15+ years of IT I've never lost a RAID5... All my servers and storage always have firmware updated (both raid card and disk firmware). I use only Dell PowerEdge servers...I always use an hot spare.
Marco

Same here, on all counts. Over the years I've a had a few drives fail, which was easily rectified with a hot swap and online array rebuild. When I first started working with servers in the early 90s, mirrored drives were "the thing". A pair of mirrored 1GB (you read that right) drives was the bees knees, lol. But when RAID-5 caught on, that's what I used for all my servers.

This was until a few years ago when I became aware of the issues regarding RAID-5 on large capacity drives, and the near mathematical certainty of failure. It took me a little while to wrap my head around this, but I have a new appreciation for RAID-1. It's simple, it's fast, and coupled with a good backup regimen, I sleep like a baby. All my servers are now using RAID-1 exclusively.

[m.novelli]

Guys, after reading a lot about large disks and URE, i'll give a try to RAID6 in my next backup server, to check if the write performance remain acceptable.

Good thread!

Marco

[McKITGuys]

Hi,

Is it possible to get some sort of informed summary of this thread? I find that by the end of a long thread, between varying opinions etc, I cannot put together a "generic" enough opinion of what should be used that I can apply it to our circumstances (we have a couple used HP G7's). Or maybe a more current white paper is out there. We are not using a Veeam partner as the reseller we purchased from ended up not knowing as much about Veeam as we would have thought (and it costs $$$ to bring another on just to advise on hardware).

Our situation is what I think is a smaller shop - 15 vm's (9 onsite, 6 at DR site), about 2TB data being backed up translating to about 4TB with 14 retained deltas consolidated on weekends. The onsite is a NetApp SAN (FAS 2220A). Want to backup onsite and move offsite. Right now only backing up to offsite but want to approach management for more hardware.

My outstanding questions from the thread (assuming we re-use the HP chassis):

- type of drive, speed, quality (SATA or should it be SAS? Raid 10? 7200 rpm; Enterprise like Western Digital Red or not needed?)
- controller? lots here mentioned controllers for Dell: how do I translate into one for HP? are there specs I can compare to or ??
- these servers have 4 x 1 Gb ethernet; connection to SAN is also ethernet; hopefully this is good enough?
- I assume the setup should be the VB&R on the HP chassis (repository) and a Veeam proxy on the virtual host connected to the SAN?
- anything else I forgot?

Thanks for any help. Oh, if there are any newer white paper/guides that go out to the partners, is it possible to have access to these and where? Maybe they answer all these questions.
Albert

[bilderg]

Hi All,

Sorry, I'm still new to Veeam, so this may be redundant.

Looking to setup/configure backup repositories. Should we 1, create repositories for each backup job? 2, Create repositories based on OS being backed up? 3, Create one big repository that includes all backups?

What's best practice, and why? Thank you! -Geoff.

[Shestakov]

Hi Geoff,

Should we 1, create repositories for each backup job?

No need to do that.

2, Create repositories based on OS being backed up?

Depends on what kind of deduplication you are using. For in-line deduplication it makes more sense to combine VMs running same OS in the one job.

3,Create one big repository that includes all backups?

It recommended to use one local repository + one or several offsite repositories for the better reliability.
In the thread above lots of relevant information. Please review. Thanks!

[Vitaliy S.]

Hi Geoff,

I would recommend creating at least one repository on-site and the secondary target in the offsite location for DR protection. Single repository hosting all backup files should work fine. The questions you ask would be more applicable to the backup jobs configuration, for which we do recommend to group VMs based on the guest OS, SLAs etc.

Thanks!

[danswartz]

"I can't see why a disk that get intensive writes during backups should have an higher failure rate due to intensives reads in case of array rebuild..."

No-one said failure due to write, it could be a drive failing due to intensive reads during rebuild. Either way, you're dead in the water, no? As far as raid6 write penalty, if you have write-back cache with BBU/flash, write penalty should not be significant?

[bilderg]

Thanks for the responses all! A few more quick questions...

Is there a recommended LUN size for the repository? I have created a 2TB GPT LUN, is there a recommended, or best practices, for block size?

[Shestakov]

Usually for LAN backup repositories it`s recommended to use 512 KB block size.
It provides a better deduplication ratio and reduces the size of a backup file because of reduced data block sizes. Thanks!

[dellock6]

Wait, are we talking LUN block size or file system block size???

[Mirija]

Hi,

Is there a recommended storage type or vendor to use with veeam?
What is best practices?

thanks.

[foggy]

Aina, please review the thread above for some initial information. Thanks.

[ryanp]

Can someone clear up some confusion I'm having with this setup... I've tried to search, but was unable to find what I'm looking for.


Would the physical server with the storage just be the repository, and Veeam installed on a VM?

Or Would Veeam be installed on the physical server with the storage?

I'm just wondering about the transport mode, which would be faster.



thanks!

[dellock6]

Both setup will work fine, which one to choose depends on your environment. The best speed can be usually achieved via DirectSAN, so the important part is that the proxy role is installed in the physical machine, but this machine also needs to be connected to the storage fabric (iscsi or FC, NFS will arrive in v9).
The central console can be deployed in a VM and act as a control machine, and also as a hotadd proxy at least for restores (hotadd restores are quicker than network mode restores, direct san restores are only possible in few situations).

[ryanp]

Would there be any negatives to making the repository a virtual machine running on the free version of ESXi?

[Gostev]

Big one. Impossible to easily get to your backups if that ESXi host dies.

[ITesus]

I am hoping the community here can share their advice and insight regarding potential storage solutions for archiving copy jobs. Our current array only has enough storage for our daily fulls with 14-21 restore points. We also replicate with Veeam over a 100 Mbps dedicated ethernet connection to a DR site, but that array only has enough space to store the replicas. I need somewhere archive my file copies for additional restore points.

I ran a copy job against our daily fulls and the size was 6.6 TB data, 5.1 TB after compression. Ideally I’d like to have weeklies, monthlies, and a yearly saved to disk. Correct me if I’m wrong, but with the file copy job set for that scenario I’d have 3 weeklies, 11 monthlies, and 1 yearly. That’s 15 total fulls. At 5.1 TBs that’s 76.5 TBs plus change and overhead.

Unfortunately, my budget for this portion of the project is hovering around $4,000, so it’s looking like I’ll have to trim my ideal plan down to something more realistic - perhaps monthlies only at best or even more likely, something like quarterlies or possibly something in between like a copy job every 6 weeks.

Performance is not an issue whatsoever as this will ONLY be for storing copy jobs. It could be large disks with RAID 6 parity penalties, etc. I don’t really care if it’s a standalone NAS like Synology, etc or if it’s a rack mount 4U chassis like what I’ve seen on the Backblaze site. I really just need the most bang for my buck. I doubt that there is an online solution that would work, but I certainly am open to one though I only have a 30 Mbps internet connection to upload and download with. Whatever the solution, I definitely would require it to have some sort of option for expansion down the road as I’ll eventually have more money in the budget to add on to it with.

So, any suggestions on how to maximize the storage I can buy with that $4,000? I’d love to hear what others are doing. Any ideas would be greatly appreciated.

[Shestakov]

Hello ITesus,

Correct me if I’m wrong, but with the file copy job set for that scenario I’d have 3 weeklies, 11 monthlies, and 1 yearly. That’s 15 total fulls. At 5.1 TBs that’s 76.5 TBs plus change and overhead.

I suppose we are talking about Backup copy job, not the File copy, assuming mentioned GFS settings?
You also haven`t provided the restore points numbers you set, so it`s impossible to say how many full backups you will have. Cloud you clarify?

As for storage recommendations, please review the topic and ask additional questions if you have any. Thanks!

[james411]

Big one. Impossible to easily get to your backups if that ESXi host dies.

I might be missing something, but why would this be specific to running virtualized. If its just a physical host with Windows installed bare metal and the host dies, isn't he going to have just as hard a time retrieving his backups?

[foggy]

Virtualization adds one more layer to the restore process, since you would need to fire up ESXi host first and then restore the VM to get access to backups inside it.

[ITesus]

I suppose we are talking about Backup copy job, not the File copy, assuming mentioned GFS settings?
You also haven`t provided the restore points numbers you set, so it`s impossible to say how many full backups you will have. Cloud you clarify?

I guess I didn't make that clear because I may not fully understand the difference - is a backup copy job simply a file copy with restore points? I currently have it set to 2 restore points which is the lowest it will go. I'm not quite sure why I would need restore points when the weekly/monthly/yearly are the restore points. If I set it to 7 would I have incrementals for each day in the weekly? At any rate, is my math correct if I have it set to 2? Should I just be using file copy jobs?

[veremin]

Nope, it is not. Kindly, see explanation provided here. And ask for additional clarification after that, if required. Thanks.

[Shestakov]

Simple, short-term backup copy retention works in accordance with the synchronization period. GFS restore points are designed for historical backups.
Detailed information is provided here. Thanks

[ITesus]

I have read both links and I still do not completely understand what the restore points setting is for on a backup copy job if the restore points for weekly, monthly, quarterly, and yearly are already set in the section below. How is it not redundant? In my scenario I have 3 weekly, 11 monthly, and 1 yearly. That's 15 copies or restore points. So, if I set restore points to keep to 2, does it only keep 2 of the 15 points I have set? Am I supposed to set it to 15? Or is it that if I set it to 30 it will keep 2 years of my GFS rotation before deleting old points? If that were the case I would think it wouldn't allow you to set fewer restore points than you have specified in the GFS rotation settings, but it does.

[alanbolte]

You cannot switch between forever-incremental retention and GFS fulls, you can only add GFS fulls to the job. So in addition to your 15 full backups for GFS retention, you will have 1 'recent' full backup which is updated as often as the sync interval you set. If you set restore points to 2, you will have one incremental file at all times, and a second incremental file while data is being transferred. This forever-incremental retention setting has no effect on your total number of GFS full backups over the long run, though setting it to a high number will delay the initial creation of the full backups.

[Shestakov]

Alan is correct.

In my scenario I have 3 weekly, 11 monthly, and 1 yearly. That's 15 copies or restore points.

Another great feature of GFS retention is that restore points are not "overlapped". Means if you schedule weekly GFS to be made on Sunday and monthly GFS on (1st or 2nd ect) Sunday of the month. For one of the weeks, you will have 1 restore point marked as Weekly and Monthly.
So even if you set to keep 4 weekly, 12 monthly, 1 yearly accordingly, you will still have 15 backup files. Thanks!

[ITesus]

Thank you. I understand perfectly now. And that raises another important point for sizing considerations. So aside from my 15 fulls I will have an additional full and at least 1 incremental along with it. So, sizing for this would be 16 fulls and an incremental. Correct?

Now to decide between the hundreds of different ways to implement cheap storage. I'm tempted to grab a backblaze and start filling it with 3, 4, 5, or maybe even 6 TB disks. No sure if I'd run a Linux distro on it or Storage Spaces for the software raid.

[Shestakov]

So aside from my 15 fulls I will have an additional full and at least 1 incremental along with it. So, sizing for this would be 16 fulls and an incremental. Correct?

Yes, your understanding is correct.