Backup appliance (Data Domain vs Exagrid vs Quantum DXi)

[kjstech]

Hello all,

We originally received a quote for 2 EMC Data Domain DD2200's 24TB capacity with all of the licences for DDBoost, replicator and encryption. Plan is to locate one at another facility so the primary can replicate to it, in case of primary hardware failure, environment or physical disaster in the production facility. EMC really did a good presentation and everything sounds very turn-key and straightforward. I think we are going to be approved for around $57k give or take a few for this project to start Q1 2016.

Then I read on various posts on this forum that people with Data Domain (we will call them DD from here on) have very poor performance with restores. In the event of a disaster, restoring your data is very crucial. Sure its great to know it is all there and intact, but if your spending days restoring it that's just not going to cut it for $56-60k.

One post turned me on to Exagrid. Apparently the issue with DD is that anything with inline dedupe is great for writing data to, but horrible at reading data from due to some processing on deduping the data. I guess Exagrid has some kind of temporary work space where data is stored at least for the shorter term (maybe a week or so) in a state where that dedupe process doesn't apply. The longer term storage can be deduped into archived storage. They also promote their scale out ability which sounds great. In googling this I was then turned onto Quantum DXi.

Quantum DXi claims they can cut back on the dedupe restore processing times without the complicated "landing zones" like Exagrid uses. Wow so this sounds like its the best of both DD + Exagrid. Ok there has to be a catch right? If it were that easy than EMC would already be doing it. Then lastly I've read that Veeam V9 which is likely out by our Q1 2016 target for this project will help with the speed on DD units.

What do other veeam users and employees recommend for simple rack and stack turnkey backup appliances? Keep in mind we would like two of them, one for our production facility and one for a remote facility just for redundancy.

[bg.ranken]

I know this post is about deduplication appliances but I still think the best approach cost wise is to invest in local storage solutions even if they don't give you the same deduplication power of a dedicated dedup appliance. You should get better performance all around and have a much more flexible environment and may actually end up with much more space over the long run if set up properly.

There's two servers that come to mind that are very good for this: the HP DL380 or Cisco C3160.

HP DL380:
-12 LFF drives in the front (4-6 TB each).
-2 SSDs in the back for the OS and optional WAN accelerators (or another 3 LFF drives if you wish).
-For the same price as some of these dedup appliances you can get 2 fully loaded dual processor HP 380 servers.
-This means you can have multiple hardware proxies with much more than enough CPU.

Cisco C3160:
-62 SFF.
-Pricier disks than the HP but can be customized with more variety due to how many disks you get.

Wish I could give more information on the Cisco but I haven't used them yet but I've been hearing good things about them. So lets talk about the HPs (since I'm more familiar with the setup) and how I would say a good setup could be done with the same budget you'd spend on a dedup appliance (assuming a 24TB dedupe appliance). This is going to vary depending on if you have a secondary site or not. Note this also means that the hardware for the backup proxy is also included in the cost instead of having to have a separate physical or virtual server in the case of a dedupe appliance.

Primary site
-2 HP DL380 servers with 12 drives (6TB) with two procs and 12-16GB of RAM each and 2 256-940GB flash drives for OS and WAN accelerator (get the smaller drives if you don't have enterprise plus).
-RAID10 on each server gives you 36TB per server which will give you 72TB of total space.
-Since you have so many physical CPUs you can set the backup job compression to high and storage optimization to LAN or WAN target (lowers block size for more deduplication though Veeam)
-Set retention short; 30-60 days

Secondary site
-2 HP DL380 servers with 12 drives (6TB) with two procs and 8-12GB of RAM each and 2 256-940GB flash drives for OS and WAN accelerator (again get the smaller drives if you don't have enterprise plus).
-RAID6 on each server gives you 60TB per server which will give you 120TB of total space.
-Use these servers to place Veeam backup copy jobs.
-Set retention long with Veeams GFS;30 restore points + monthly or quarterly archival restore points
-With Veeam 9 you can use the new per VM backup file option which will split each VM into it's individual backup file, and while this will decrease the dedup you get with Veeam you can then...
-Enabled Windows 2012 R2 file deduplication. This is post process and does have some recommendations like not using it for files over 1TB (removed for Windows 2016), but for now the per VM backup file should keep things under 1TB. Make sure to also split the volumes in half (30TBs per volume) since Microsoft recommends this and you get more deduplication streams (again something addressed in Windows 2016) Set the data deduplication to dedupe files older than the amount of restore points you plan to keep. This way you will always have a full non-deduped backup on the drive and the data deduplication can go to town on the restore points kept for archival purposes set in the GFS method. If you're doing a monthly or quarterly checkpoint this should eventually end up giving you the 70-80% deduplication similar to what some of these dedupe appliances are promoting. And when 2016 hits it'll offer more improvements to file deduplication which should make this scenario even better: http://blogs.technet.com/b/filecab/arch ... iew-2.aspx

I can't give you exact pricing since there are so many dedup appliance configurations but from what I have seen this option should be cheaper. The real difference here is performance which you will have to compare to whatever dedup appliance you are looking at. With that many disks at RAID10 and multiple proxies on the primary landing site performance should be decent, especially considering the data now goes directly from the proxy to the local disks rather than back over the network. Since the second site would be used for Veeam backup copy dumps having RAID6 shouldn't be an issue in terms of performance, especially since it's not hitting production at that point. And expanding is as easy as adding more servers either in the secondary site for more retention or in the primary site for more space for initial backups. In terms of management Veeam 9 is adding the Scaled-out Backup Repository http://www.veeam.com/blog/introducing-s ... te-v9.html which should make all of this setup even easier since you'd only need one scale-out repository for each site and everything else would be automatic.

[aman4God]

Regarding the Data Domain for restore. I too was at VeeamON and had several conversations with people that were either struggling with the restore times for Data Domain or had recently purchased an entire DR architecture based on the premise that Data Domain would perform perfectly for restores. I currently use a Data Domain in our environment as a repository, but solely for my BC jobs for long term. What EMC does not share is Data Domain is best suited for LTV. While there may be some argument, in the 14 years I have worked on Data Domain, I have come to the conclusion that it is best suited to long term archive or post 60 day restores. If you check out the EP story, "Tales from a Backup Nerd", https://youtu.be/rhQuSqprR8k done by a friend of mine at VeeamON this year, he speaks very succinctly to the idea of architecting for the most common restore periods. Within our enterprise, I run a short term backup window to a Nimble CS-500 that is also part of my primary storage. Then with Backup Copy stage the backups to Data Domain/Panzura for long term archive of backups, as we in our health care space have a long retention requirements. This gives me phenomenal restore times from the high performing storage, drastically reducing my RPO and allowing my to even run my instant restores on the same storage that the live servers are running on. While this specific design may not be for everyone, I would highly suggest the idea of placing a higher performing layer in between the Veeam and the Data Domain Archive appliance. Designing a BCDR plan based on restores is somewhat of a paradigm shift, but that is what really matters.

[bg.ranken]

I agree with aman4God. It's important to know where a deduplication appliance is useful. I'd say you could keep the same primary site setup I recommended (with the HP servers) and utilize the dedupe appliance in the secondary site for Veeam backup copy jobs and the longer retention. Some people do like to carve out their primary storage for the initial backup like aman4God presented, and while this is a viable option, I've always believed that backups should always be to a secondary target on different hardware to follow the 3-2-1 rule.

[kjstech]

Yeah I toyed around with creating another filesystem on our VNX5200 for daily's / weekly, but that VNX5200 is also hosting production data.

Right now our Veeam target is a FreeNAS box with 7.1 TB usable capacity. I can only store 5 day incremental and 1 weekly before it runs out of space. Backup copy jobs copy this to a second, identical FreeNAS box. The issue with the capacity is they are old 1U Dell PowerEdge servers and can only fit 2 hard drives, which are currently 4TB drives in ZFS Raid-0 (which is why we backup copy to a second box). They are just hand me down units and we really need something better.

bg.ranken, in your HP example are you installing Windows Server 2012 R2 right on the box and installing Veeam to it? Right now Veeam is on a 2012 R2 virtual machine (12GB RAM, 8 vCPU). Thats ok we can move it, or are we keeping it where is and just mapping a share to the HP server in your post? Sounds like your proposing putting Veeam directly on the HP server so the storage is local to Veeam itself. Then maybe policies to shoot longer term out to a Data Domain or something of that sort?

I also just filled out a survey to size and price out an Exagrid system, so it will be some time until I hear back and get a conference call scheduled with one of their Engineers.

[bg.ranken]

bg.ranken, in your HP example are you installing Windows Server 2012 R2 right on the box and installing Veeam to it? Right now Veeam is on a 2012 R2 virtual machine (12GB RAM, 8 vCPU). Thats ok we can move it, or are we keeping it where is and just mapping a share to the HP server in your post? Sounds like your proposing putting Veeam directly on the HP server so the storage is local to Veeam itself. Then maybe policies to shoot longer term out to a Data Domain or something of that sort?

Yep that's exactly what I'm suggesting. I've heard it said many times by Veeam reps that it's the recommended configuration instead of using NAS or SAN targets due to simplicity and performance. You get more performance since you get full usage of whatever CPU you have without taxing any hosts and all the data is local.

With the amount of data you're backing up you could probably just do a single HP server in both sites, or if you only have one site follow the same build-out but just separate your "second site" to a different rack or part of the office if it's viable.

[kjstech]

If I use one server at two sites, I would have to upgrade my Veeam from Enterprise to Enterprise Plus to get the WAN acceleration right? My furthest site is a 50mbps wan link, also competing for this traffic is VMWare replication, which maybe Veeam can take care of it all.

6 ESXi 5.0 servers at production site connected to EMC VNX5200 (veeam resides here as a VM)
2 ESXi 5.0 servers at dr site connected to EMC NX4
VMWare Site Recovery Manager and vSphere replication
Since that SRM and vSphere replication is real time, we also want the weekly / nightly veeam backups stored offsite as well.

So you think I could place one of those HP servers at each site, add two Windows 2012 R2 standard licences for each side, and upgrade Veeam to Enterprise Plus, I can install Veeam on both sides right? Currently licensed for 12 CPU sockets because I only use Veeam to backup those 6 (dual core) ESXi servers at the production site. I just want to make sure I have my laundry list of items in a row to send to my rep I do business with. This way I can compare all three solutions. My existing two DD2500 quotes, a new Exagrid quote (hopefully coming soon), and then this home grown solution, and any combination in between.

[bg.ranken]

Yes you would need to upgrade to Enterprise Plus for the WAN accelerator, but that's only if you need it. With a 50mbps pipe you may not need it but it really depends on your daily change data. You can always try it without to see if it can copy during the day, and ask for temporary Enterprise Plus licenses to demo the WAN accelerator to see if it helps if the 50mbps is not enough.

Regarding where to install Veeam, you only have to install it once and then you can put Veeam backup proxies (under Backup Infrastructure) on the other server. Some people prefer to have Veeam installed in the primary site which pushes data to DR, however you will have to rebuild Veeam if your primary site fails and you're trying to recovery from Veeam (normally just an import of the config file is all that's needed). Otherwise I think it's more common to install Veeam in your DR site and pull the data. This way in case of a DR everything just works, and if you DR site goes down it only affects your backups momentarily.

For licensing, it's my understanding that you only need licenses for each socket of any host you are backing up FROM. You may want to check with your Veeam rep but I believe Veeam replication TO another host and backup copy TO a second site do not require licenses. I believe you may be able to spin up servers in your DR with instant recovery as well without using a license, but again check with your Veeam rep.

[kjstech]

You my friend have been most helpful. I put together an email reply with a lot of this information, requesting pricing on the HP DL380 Gen9 with 12x4 TB SATA and also the 6TB drives (to compare cost) (dual CPU and 16GB RAM). Our Veeam support renewal quote for 2016 is up so I am asking to see that renewed with Enterprise Plus just to gauge the cost difference.

It may turn out for at least our size business that two of these HP servers using Veeam to do the WAN acceleration to the remote site would still place us further ahead than what we are doing today. Using 4TB drives in RAID10, I think thats about 24TB space. Remember today we only have 7.1TB (after ZFS formatting) on 2005 era Dell PowerEdge 1U systems. Perhaps then the golden ticket here is use a smaller DD appliance and Veeam backup copy jobs for long term backup storage. That way the near term is always ready to go. I don't see the reason to go for SAS drives vs SATA though because I think that would negate the cost benefits and like you said were talking RAID-10 with 12 spindals, that should assist with 7200 RPM SATA, but I am always open to suggestions. In an emergency it would be nice to be able to run a few VM's from this. In fact last week we had a 10gbe switch cluster go which severed NFS storage from our VNX5200 array from our ESXi servers in production, effectively putting a debilitating blow to our organization. Three hours later this was resolved, but had we another storage appliance on a 1gig or 4gig (4x1gig lacp) made available and presented to the ESXi systems, we may have been able to have some services up and running temporally until the main storage connectivity was resolved.

[bg.ranken]

No problem, glad to be of assistance. For you I'd probably recommend the 4TB drives for your primary site and the 6TB drives for the secondary if you wanted lots of retention. Or like you mentioned use the smaller dedupe appliance for the secondary site for long term storage.

Just remember that the dedupe appliances really only start to shine when you have lots of full backups by using the GFS within Veeam. But this means that minimum you should expect to need about 2-3 times your current backup data to even be able to use these devices, which would put you close to a 20TB dedupe device. So make sure to compare the properly sized model with the 60TB HP server with Microsoft's deduplication.

Also, I'd recommend adding a 10gbe NIC to the HP server in your primary site (secondary site shouldn't need one if you're only doing Veeam backup copy). You could probably get one pretty cheap used if you don't want to pay full price for a new one but it may come in handy for backing up or doing instant recovery of multiple servers in the event you had to.

[mkaec]

On the topic of dedup appliances, ExaGrid actually does something unusual for a company in the storage industry. They give more space than advertised. If you purchase a 13 TB unit, they actually give you a 26 TB unit. By default, 13 TB would be used as a repository area where the deduped data is stored and the other 13 TB as a landing zone. Since the data goes into the landing zone, dedup is post-process, which allows for backups to run without slowing down. This is also the reason why most restores will be quicker than the other appliances. The data on the landing zone is not removed after it is processed. If you need to restore from the most recent backup, the data is sourced from the uncompressed version sitting in the landing zone. Quantum DXi is quoted as saying this is "complex". But, that doesn't matter. What matters is cost parity. I don't know if ExaGrid has to price higher than the competing appliances in order deliver the extra drives for the landing zone. If they don't, then I think the landing zone becomes a very nice advantage.

[mkaec]

...This is post process and does have some recommendations like not using it for files over 1TB (removed for Windows 2016), but for now the per VM backup file should keep things under 1TB...

Everything I've seen indicates that the 1 TB limit is still alive in Windows Server 2016. For 2012 R2, the official line was that files "approaching" 1 TB were "not good candidates". With 2016, they are saying that up to 1 TB is now fully supported. That's actually not that much of a change.

http://blogs.technet.com/b/filecab/arch ... iew-3.aspx

And B&R 9's option to split backup files per VM is nice. But, the news of both is ultimately a let down for me. We have some volumes on our file server that are over 2 TB. Splitting per VM will not help and the 1 TB limit will still be a problem for using Windows as a repository. I was really hoping B&R 9 would implement backup file splitting based on a configurable size. This is one feature of Backup Exec that I miss.

If you are aware of a newer Microsoft article that talks about the 1 TB limit actually being gone, please share it. That would be some good news.

[ckbrou]

Keith, I would strongly caution against using a Dedupe Appliance as a backup copy target. I tried for a long time to make my Data Domain work as a copy target and the transforms just took too long. There is a lot of read activity as a part of the backup copy which means that a lot of data has to be "rehydrated". See this post: http://forums.veeam.com/veeam-backup-re ... 17785.html

I probably wouldn't mess with Windows Server dedupe either. Direct attached storage is cheap. My primary target is a Dell R620 with 2 MD1200 shelves full of 3TB drives in Raid 60. I can easily add additional MD1200s in the future if I need more space. My offsite backup copy target is another Dell R620 with 1 MD1200 with 2 TB drives in Raid 10 (raid 10 for faster transform operations of the backup copy jobs). This would be easy to scale to your needs with larger disks in the DAS and more shelves.

If you haven't yet, check out this thread outlining the benefits of a setup like this: http://forums.veeam.com/veeam-backup-re ... 21304.html

[lightsout]

I've found this script to be helpful when using v8:

https://poulpreben.com/active-full-back ... -copy-job/

This should not be necessary when v9 comes, but it is helping me right now.

[ndolson]

We are a Data Domain customer (DD2500 with twelve 7.2K RPM drives) and have an appliance in each site. As your research suggests, the ingest rates are very good, and we are getting great deduplication rates, even on things normally considered poor dedupe candidates, and it's very efficient on bandwidth for WAN replication. I can also confirm that restores are rather slow. What complicates the rehydration of data is the number and speed of the spindles in these appliances - without expansion shelves giving you more spindles (and in our case unnecessary capacity = $$$). EMC stated that we could expect roughly 45 MB/s raw out of the twelve 7.2K drives, not factoring in any "logical" efficiencies from rehydrating the data, giving an "effective" restore rate higher than 45 MB/s. That being said, restoring a large VM (say, a several TB file server) would take a painfully long time. We replaced our servers and storage recently and are relying on storage level snapshots and replication for our "tier 0" restores of entire VM's and have relegated Data Domain to a more file based/archive type role. The poor performance of random reads makes things like synthetic fulls using Veeam difficult to do, and in fact we still use daily incrementals + a weekly full (since DD dedupes so well, there's really no capacity penalty for a full, just a time penalty). It probably goes without saying, but I will anyway, that the issues we had with Data Domain's performance have absolutely nothing to do with Veeam, and in fact Veeam's integration with DD Boost is actually quite nice on the ingestion side of the equation.

I have no personal hands on experience with either of the other two products you mention, though the Exagrid "landing zone" should resolve the issue of slow restores from deduplicated data, assuming that the data you need to restore from (the last full backup I believe) is actually in that landing zone and not from an older backup.

[yasuda]

If you're using Server 2012 (R2), obviously you need hardware RAID for RAID 6, but would you use hardware RAID 10 or the Storage Spaces equivalent?

I am still trying to wrap my head around Storage Spaces terminology, but it appears if I had a single virtual disk in hardware RAID 10 with 6 drives, I could later expand the RAID 10 virtual disk by adding 2 drives, whereas with Storage Spaces I would need to add 6 drives. Are there advantages to Storage Spaces? Quicker rebuild time when a failed drive is replaced? There's data tiering, but I don't think that would have much benefit for writing backup data.

[bg.ranken]

Everything I've seen indicates that the 1 TB limit is still alive in Windows Server 2016. For 2012 R2, the official line was that files "approaching" 1 TB were "not good candidates". With 2016, they are saying that up to 1 TB is now fully supported. That's actually not that much of a change.

I think you have to realize why they say 1 TB files were "not good candidates". The reason why is that you can experience "significant performance degradation while reading/appending data from/to very large files" http://www.veeam.com/kb2023. This is still true with Windows 2016 but you have to realize that deduplication in this model is primarily used for the grandfather-father-son full backup files which are for archival purposes. Performance reading/appending to these files doesn't matter since they are written to once and then only accessed again during a restore process of old data. It's why I recommended setting "...data deduplication to dedupe files older than the amount of restore points you plan to keep". This ensures that the only thing deduped are the old GFF archive points and the regular forever forward incremental chain is left untouched and unaffected by the limitations of the Microsoft dedupe.

Even if they are over 2TB they can still be good candidates for two reasons:
-They are supported by Microsoft.
-Amount of savings you can get between GFF archival backup files. If you're doing monthly GFF checkpoints these GFF backup files should have 90-95% in common and give you more than enough reason to dedupe.

And remember, if for any reason you still do not want to dedupe the 2TB files you can simply exclude the folders for your file servers from the deduplication, you'll still be gaining space from the remaining files on the disk that are being deduped.

[bg.ranken]

If you're using Server 2012 (R2), obviously you need hardware RAID for RAID 6, but would you use hardware RAID 10 or the Storage Spaces equivalent?

I am still trying to wrap my head around Storage Spaces terminology, but it appears if I had a single virtual disk in hardware RAID 10 with 6 drives, I could later expand the RAID 10 virtual disk by adding 2 drives, whereas with Storage Spaces I would need to add 6 drives. Are there advantages to Storage Spaces? Quicker rebuild time when a failed drive is replaced? There's data tiering, but I don't think that would have much benefit for writing backup data.

I really haven't used storage spaces to it's limit to give you a fair answer, but my thinking is that most servers you purchase will come with a RAID controller. There can be some cases where going with some whiteboxes or lower level servers without RAID controllers will be cheaper but that varies depending on how expensive of a RAID controller you're getting. You also have to remember that in order to properly use storage spaces then you'll want to bypass any RAID controller settings (essentially do JBOD) effectively making them useless if you did get one.

But in the end, if you have a server with a decent RAID controller with cache I don't see any reason to not use it. Storage spaces may give you better and/or safer rebuilds but you aren't going to get the same level of performance you get out of a dedicated RAID controller. When going with storage spaces you have to remember there's extra overhead involved (I believe handled by the CPU) that the RAID controller is no longer able to handle. And again, for as much data as Veeam uses I'd still recommend using RAID10 or RAID6 (or RAID60 if your controller supports it) which does give you more protection than typical RAID5.

[kjstech]

I met with the Exagrid guys and got a quote back. For two Exagrid units replicating to each other plus support it actually costs MORE than two DD2500 appliances. However they (and many posts in this forum) have justified reasons why the restore stinks from DD appliance. So if I place a server with a bunch of disk in front of a DD unit, then your getting (or surpassing) the Exagrid quote.

Still very impressed with the Exagrid guys and their presentation, much more so than EMC DD. I have completely ruled EMC DD out of the picture at this point. It comes down to if we can get the money approved for Exagrid or not. If not then yes I am going to build an HP server for both locations, upgrade Veeam from Enterprise to Enterprise plus to take advantage of WAN deduplication for replication, and that should bring us close to or under our EMC DD budget item that did make it into the 2016 budget sheet for consideration at the end of the year budget finalization meeting.

I have a call this afternoon with my re-seller and HP specialist to go over this HP server config.

Whats important to me? If I can actually not only restore a VM quickly if needed, but if I could run a VM (or a few mission critical VM's) directly off the backup in case we have an issue with our primary storage array. SureBackup too is a feature we want to start using.

[nmdange]

For cheap local storage servers, this is my favorite option. It's been absolutely awesome for our Veeam backup repositories, fast backups, fast restores and running VMs directly off the backup repository works well too.

http://www.aberdeeninc.com/abcatg/Stirling-X46.htm
and then for more storage add JBODs
http://www.aberdeeninc.com/abcatg/jbod-s41.htm

Unlike Dell or HP servers, you use un-branded and un-marked up drives. It ended up being half the price of a Dell server with the same amount of storage, and since the system is double-sided, it took up less rack space as well.

[emachabert]

If you are looking for uber high density, take a look at the HP Apollo 4510, 68*LFF.

[ckbrou]

If you are really looking to save money I recommend sticking with HP or Dell and going to the secondary market to buy something a few years old. For example, you can buy a Dell MD1200 bare bones used and then purchase new drives separately (or used drives if necessary). You do not need Dell Branded drives to make this work. The nice thing about sticking with Dell or HP is that parts are very easy to get and there is so much great support knowledge out there.

[kjstech]

For cheap local storage servers, this is my favorite option. It's been absolutely awesome for our Veeam backup repositories, fast backups, fast restores and running VMs directly off the backup repository works well too.

http://www.aberdeeninc.com/abcatg/Stirling-X46.htm
and then for more storage add JBODs
http://www.aberdeeninc.com/abcatg/jbod-s41.htm

Unlike Dell or HP servers, you use un-branded and un-marked up drives. It ended up being half the price of a Dell server with the same amount of storage, and since the system is double-sided, it took up less rack space as well.

This is an interesting avenue. I like the instant price updates when configuring the box. Is this hardware or software raid? Says raid-6 is supported - yet not an option when configuring. The pricing is great, too good to be true.

One thing you get with Exagrid is that great support where the device phone homes when there's an issue. You could walk in next day to a new hard drive or power supply to resolve something you didn't even know went bad. Sure you can alert on this with other solutions like HP iLO, Dell DRAC, but what about Aberdeen? I'd be interested to know how it really performs and if you used any Server 2012 R2 dedupe on it or not. Also interested in which raid level you used (raid-10 I suppose) and what you would use to replicate to a remote location. I assume I would have to upgrade Veeam to Enterprise Plus to get the WAN acceleration, however pricing all of this out is still less than Exagrid or two DD2500's.

If something happens to our 10gig network or primary array, I'd like to be able to run some critical VMs from this using Veeam Instant Restore over a bonded 4gbps lacp connection to a separate 1 Gbps network (each esxi has 2gbps connection to this currently). At least use it as a "spare tire" until production 10gig or array is diagnosed and resolved.

[bg.ranken]

I just wanted to point out that the model I stated with the HP servers really works with any regular server model; it doesn't matter which vendor you pick as long as you meet the few basic guidelines. Using Dell, Cisco, or even a whitebox will work just fine as long as they meet the space and performance requirements for you. The RAID controller will make some difference in speed, but more often you're going to be limited by the number of spindles and speed of the disks. You could even do SSDs for the primary backup target if you want extreme performance as long as you're willing to pay the price.

The main concept of the build is to simply use faster local DAS on the Veeam proxy server (HP, Dell, Aberdeen) to skip any network bottlenecks and reduce complexity and then use Veeam backup copy to go to a secondary site, preferably offsite, with Veeam backup copy. As others had mentioned you can do a dedupe appliance at that level (since performance isn't a priority for the secondary backup target) or more of the same servers built for more data and less performance (hence the RAID6 over RAID10)

To get why I recommend HP I would say I agree with ckbrou in that HP is pretty much the sweet spot between performance, price, reliability, and support; and buying refurbished even further lowers the price. But then again that's just my opinion, regardless of which vendor you choose it's important to make sure everything is designed to perform the tasks you put before it. If you want to be able to spin up critical VMs you need to design your primary backup proxy and storage target (both in the same box if using the model I recommended) to have the performance to meet those demands. If that means going with faster 10k or 15k drives, adding 10Gbe, or a second proc then by all means do it. If you plan on getting the Exagrid for the primary backup target make sure that it'll provide enough IOPs for whatever servers you plan to spin up with instant recovery so that they are actually usable.

[nmdange]

The server has an LSI MegaRAID controller with a battery-backed cache, so it is hardware raid. I use RAID 50 with 4TB drives, along with a 4-disk 10k RPM RAID 10 set which contains the Veeam and SQL Server installs. I also included a 10gb NIC with the server which is connected to the storage network. Throughput varies, but I have a throttling rule set up to throttle at 8Gbps and sometimes the throttling is the bottleneck, so I know the system can push 8Gbps throughput. Most of the time the source system is the bottleneck during backups. I have not tested 2012 R2 dedup, I am honestly waiting for the improvements to dedup coming in WS2016 before using it. But the storage was so cheap, even without dedup I have more than enough space for everything.

As far as alerting, it has an out-of-band management like iLO and DRAC. (It's a Supermicro motherboard so it's Supermicro's interface). The trade-off on the price is that part replacement is shipped with standard shipping so it takes a few days, rather than 4 hours or next-business-day.

[bg.ranken]

Also remember it might also be worth looking into SSDs for the vPower NFS Service since it handles the instant VM recovery http://helpcenter.veeam.com/backup/80/v ... rvice.html. It may help you get a few more servers running at the same time with instant recovery like you mentioned.

You're still going to be limited on IOPs from the backup target you're running the VMs off of but it should provide some boost. And 256-512GB SSDs are cheap enough I'd still recommend two in RAID 1 for the OS of the Veeam proxy, vPower NFS service, and the WAN accelerator. Just be aware there's been some debate on if SSDs help the WAN accelerator any more since v8, see here: http://forums.veeam.com/veeam-backup-re ... 25151.html But if you have it I don't see how it could hurt to put the WAN accelerator on it rather than on the same drives the repository is going on.

[Delo123]

I can also recommend commodity hardware for veeam proxy & repository.

We currently use multiple 4U 36 Bay Supermicro servers with 4TB HGST SATA NAS Disks and 256GB of RAM each.
Total costs for each node with one set (36) of disks are around 12K US$ with dual E5-2620 CPU's to give it just enough processing power to handle proxy & repository.

Each Node has 2 16 Port Adaptec 7x series Controllers so we do not ned expanders (for performance and redundancy reasons).
Per 6 Disks we create a Raid 5 resulting in 6 disk groups of 18TB each. Each Raid volume is checked weekly at adaptec level.

These 6 now "physical disks" are put into a non redundant storage pool (since we also have some redundancy in each disk group) resulting in an 108TB Storage Space.
Now we create multiple virtual thin volumes (4 in our case) of 62TB since 64TB is the maximum size microsoft vss can handle at this time.
Another Reason why we create multiple volumes is that we can run microsoft dedupe on all of the volumes in parallel so total time needed for post dedupe is decreased.
Additionaly we tweak the dedupe jobs so they run in highest priority and use 100% memory which still isn't much more then 10GB per job. Dedupe is set to general file server (vdi doesn't work)We also setup to dedupe data older than 1 day so the most recent backup can be restored without being hit too hard from random io ever more in case of an instant recovery which is already an issue with disks in general.

Important: The Volumes need to be formatted with the /L setting (very important!) so there will be no issues when there are a lot of files on the volume later on!
Garbage collection and scrubbing is also set to daily instead of weekly so deleted data is cleaned up immediately when it occurs.

Now we offer these 4 volumes to veeam as a local repository and set the job settings to dedupe friendly and LAN target. This gave is best performance and good dedupe rates after a lot of testing. In the repository settings we choose align backup blocks & decompress backup blocks before storing to have a high speed during backup but also good results on windows dedupe.

These repostitories gather data for about 6-9 Months before they are full (fulls & daily incrementals for 6-9 Months), at this time we take out all the drives to store them (we never delete a backup) and put 36 fresh new drivers in the "appliance" and start over. We have multiple appliances and swap disks on a different schedule so we always have at least some weeks history in one or the other box.

To the results:

Maximum ingest rate running a single job is around 1Gigabyte per second, the bottleneck usually is network, we are still investigating the reason for this as we are using shared memory (datamover setting 2).
Last disk swap was 01.09.2015 so there currently around 2,5 months of backup data on the device (we use monthy fulls and forward incrementals).
Of the 108TB currently 49,1TB are free.

Dedupe savings:
PS C:\Windows\system32> get-dedupstatus

FreeSpace SavedSpace OptimizedFiles InPolicyFiles Volume
--------- ---------- -------------- ------------- ------
53.6 TB 25.97 TB 99 97 J:
50.99 TB 43.3 TB 88 88 G:
56.04 TB 27,51 TB 140 140 I:
42.06 TB 111.67 TB 328 328 H:

So totally the node currently holds about 250TB of Data which is deduped to 49,1TB physically used. Dedupe rates of course will increase further as more backups are stored on the node. Usually we get around 700TB of data on the nodes before the storage pool is completely full.

One of the issues we have with the supermicros is that that the hdd cariers need 4 screws to secure each harddisk resulting in 144 screws which need to be attached each times we buy new disks and cages. For this reason and also to further expand capacity per node we are looking at "cheap" 84 & 96 bay 4U Jbods which are top loaders and feature screwless hdd cages which should allow for quicker installation of the hdd's.

Ps. our biggest VBK's (Veeam Fulls) are around 6TB, until now we never had a problem deduping them with 2012R2 dedupe (even if MS says bigger than 1TB is not recommended), the only issue we have seen is that the dedupe window can get too high so the next backup will start before dedupe is done, but in this case windows dedupe will continue on the same file at the next run. (2x daily).

We also do regular restores / instant recovery tests to ensure there is no hidden data corruption. anyway. Until now we had hoped for veeam to offer the options of splitting vbk's in chunks so they would be easiser for windows to dedupe but as server 2016 is coming this should be an issue of the past however we would still like to be able to split for multiple reasons (copying 10 100GB files is quicker than 1 1TB file due to multi threading and in case of a copy error of a certain file recopying is easier).

Regards,
Guido

[bg.ranken]

Guido this is a really interesting setup. So essentially you're doing RAID 50 by mixing the hardware RAID5 plus "RAID0" from storage spaces. Did you do testing that showed this to be superior to just using RAID 50 at the hardware level? Also regarding your deduplication on your regular backup chains, you said you set them to dedupe after 1 day. Are you using forward or reverse incremental? It sounds like reverse incremental since you said you don't dedupe the latest file. In this case, how much are you getting in deduplication on just the incremental files? Do you find it to be worth it?

Also, at least Veeam is getting an option to split VMs into their own separate VBK files so perhaps that will help with your larger ones.

[Delo123]

Hi BG,

Yes, essentially this is a Raid 50, we however opted for multiple Raid5s because of multiple reasons:
- In case of a rebuild not all disks are affected (of course one could argue if this is an actual benefit, but it feels better to use)
- Right now we have 6 Groups of 6 disks divided over 2 Controllers. When using Raid 50 everything would be on a single Controller. Remember we are using SATA disks only.
- We could do Raid 6 / 60 but since we have multiple nodes doing Independent Jobs (2 backup runs each night & an additional backup copy to offsite) we could lose one of the nodes without losing any backup data.
- On top of storage spaces we create simple vdisks (so no redundancy from storage pools), but we did do some testing with parity Setting. We could always create an additional vdisk with parity Setting for "very top secret ultra important backup data) in this case we could lose an entire raid set (since parity pool protects from a single disk /disk Group failure) and in parallel lose 1 disk in each of the 5 left over groups, however parity on top of parity Comes with some obvious penalties. Streamed backups were ok, but restores and especially instant recoveries hits the disks real hard, also of course more space is needed for the additional protection so we stopped using parity on top of raid 5 quite early.
Another alternative would be not using Hardware Raid but only to use storage spaces parity / mirroring but we felt better doing is like this, on of the reasons was that we feel a bit better since the raid Controllers do regular parity checks, not sure how good Windows is at this....

We are using Forward incremental. It's actually very easy not the dedupe the latest full with it's increments, you just have to Setup the time for Windows dedupe to process files (older than x days), you should ofcourse have a full which is not to old (we do have some different dedupe Settings per repository), that is why we distribute our Jobs accross the nodes and try to take care the latest full the the Job in question isn't too old, this takes a bit of practice... Also not all backups Need to be non dedudplicated, for fileservers we don't care because usually all we Need is a couple of files etc.... In case of a real issue we have veeam replicas in place since we do not like to rely on on instant recovery in an emergency. (even the best backup isn't nearly as good as a replica

[backup.master]

Deduplication backup target appliances are playing a more active role in the enterprise backup architecture, beyond tape replacement. The Gartner 2015 Magic Quadrant for Deduplication Backup Target Appliances (September 2015) research, evaluates the strengths and weaknesses of 10 vendors that offer such appliances for I&O leaders:
http://www.gartner.com/technology/repri ... 0622&st=sb