Veeam Backup Server HP Apollo 4200

[bg.ranken]

Hi All,

We are currently looking into replacing our current Veeam server as well as all of our proxies (used offsite with Veeam backup copy) with servers loaded with direct attached storage. Being an HP shop, we were originally looking at DL380s with 12 4TB LFF drives in each. However it looks like HP has released this Apollo series this year that can hold up to 24 LFF drives in a 2U box.

Has anyone had a chance to get their hands on one of these yet and test them out with Veeam? If so I'd like to know your thoughts on the platform. I've always been more of a fan of using DAS instead of iSCSI or other NAS devices when it comes to pure archival data, but I've never had a server loaded with more than 12 drives and am not sure what other "gotchas" I should look out for.

Also for this many drives we're planning on doing two sets of RAID 6, so we'll get about 80TB of space. Most of these will be used for archival use through GFS so performance isn't a major factor. We will be layering on Windows 2012 deduplication on top of the archives though so I'm hoping we'll be able to fit a lot of data on these.

Does this sound like a good idea or are there better alternatives? Our current full backup size sits around 25TB so I'm hoping 2 of these boxes with the Microsoft deduplication will be enough for us to keep a years worth of monthly checkpoints through GFS with the backup copy.

[poulpreben]

Just before the holidays, we received the HP Apollo 4200 in the Veeam labs for some testing together with HP, so expect some official performance results in the near future.

As it is still too soon for us to conclude anything, the only gotcha is the missing FC HBA option. We just found a standard HP HBA on our shelves and installed it, but it's not officially supported as far as I understand. If you're not a FC shop, or if you don't want to use the server as backup proxy, I guess it is no problem. For us it was just quite critical to get the best throughput from our FibreChannel storage array.

This particular box has 22x 6TB and 2x SSD drives connected to the P84x controller and the SSD drives are used with the SmartCache feature. It means additional write-back caching on SSDs which seems to give a significant performance boost.

Windows deduplication... well, I know some of my colleagues have good experiences with it, but I never saw it working at scale. The biggest limitation is the 2 TB file limit, which even with v9 prohibits you from backing up any VMs larger than 2TB to those repositories. I am probably a bit biased having had to deal with numerous customers who never bothered reading the release notes or system requires for that feature.

Merry Christmas!

- Preben

[bg.ranken]

Awesome, thanks for the information, I'll be looking forward to the performance results.

That's very interesting regarding the FC HBA. I didn't even think about it not being an option this day and age and definitely something I will look out for. I guess it won't matter too much for our archival storage with Veeam backup copy, but we were also planning on replacing our actual Veeam server as well which means we'd definitely need FC.

Regarding Windows deduplication, the reason we're planning on using it is because it's going to be a near perfect solution with the release of Windows 2016 and Veeam 9. Combining the per VM backup file option in Veeam 9 with the improvements in the deduplication engine in 2016 I'm hoping it's going to do very well. I also agree with you in terms of scale, but take a look at this article for 2016 since it seems like much of the scaling problems are being fixed in the new version: http://blogs.technet.com/b/filecab/arch ... iew-3.aspx

[bg.ranken]

So paulpreben, happen to get anywhere on that testing yet? No rush or anything I'm more just curious.

[poulpreben]

Good timing of you to ask this question. The draft whitepaper was in my inbox this morning. Stay tuned

[Andreas Neufert]

This is such a good device... Why you want to slow down it and make Instant VM recovery nearly unusable with Windows Dedup.
If you do so I suggest to create 2 Repositories on it (different folder) 1 with actual restore points... no dedup... the one for Long Term Backup Archive (GFS) with enabled dedup.

There are several limitations Anton Gostev (VP Product Managent) discuss at the moment with the MS win 2016 team. He shared this information in the Forum Digest.

[dellock6]

And to add it to Andreas notes, create two volumes using two different raid groups and even different raid controllers, like basically having two arrays in the same box. Otherwise dedupe activities will still impact the performance of the non deduped volume.

[bg.ranken]

Oh, I had no intention of deduplicating the actual backup files. As dellock6 said my plan would have been to create different volumes with different RAID groups and only dedup the backup copy files (instant recovery would be unaffected). Though according to Microsoft there's only a 3% performance loss for opening files that are deduplicated so technically we could do instant recovery from the backup copy files, but ideally we would just have them for archival purposes and only do file level restores from them. And I've been reading the Gostev digests and he's been mentioning all the improvements for Windows 2016. Just last week he mentioned that it's possible the 1TB recommended limit (the only real problem with using it with Veeam) may not even apply to read-only backup files and I've been hoping he'll hear from the team at Microsoft and confirm that.

Though plans have changed and I am now thinking about getting 3 of these with 80TB each. One would be completely flat for the actual backup jobs. Then the other two would be put in our DR site and have Veeam backup copy jobs going to it, and those would have deduplication turned on. They would be strictly for archiving at that point, with monthly checkpoints going back two or three years.

But based on the pricing I've gotten that 240TB with the Apollo is still cheaper than many of these deduplication appliances even assuming you get a 10-20x deduplication ratio and they can sell you two 10-20TB appliances (one for onsite and one for DR). Layer on top the post-deduplication with Microsoft to get more than 240TB for archival backup copy files and the better performance (from having so many drives) I just see this as a win-win. But still it's all speculation on my part which is why I've been waiting for poulpreben's testing.

[pirx]

Anyone using a Apollo 4510? This device looks interesting with its 68 drives. Not sure how a configuration should look like regarding disk type/size number/type of RAID controllers.

[Andreas Neufert]

Anyone using a Apollo 4510? This device looks interesting with its 68 drives. Not sure how a configuration should look like regarding disk type/size number/type of RAID controllers.

Hi Pirx,

thanks for the request.
Apollo Systems are strait forward standard systems with typical standard raid controller.
So if you use big disks, Raid rebuild times are very high. To protect your backup data you should use Raid6 or Raid60. As Veeam do a lot of Random Write in many cases Raid 6/60 isn´t that optimal.
My recommendation for huge disks (above 1TB) or Nearline (7200rpm) is:
- Create a Raid60 with 4+2 or 5+2 (to address random write penalty at Raid 6)
- Use Apollo recommendations for Spare disks. 1 for each 20 at min.
- Latest Raid Controller Firmware usage
- Latest Raid Controller Firmware default block size settings are OK for Veeam. Maybe set it one value higher.
- Add at least 2 SSDs in Raid 1 and create a Smart Cache (use latest Raid controller configuration boot media to enable this).
- User forward incremental Veeam backup modes to process more data sequentially.
- Per VM backup chains can help to increase speed with that system
- Don´t use any Raid controller based dedup or compression (don´t know if any raid controllers are available for that system that support this... just in case).

A general recommendation for Raid Controller selection:
- use battery buffered controllers
- choose the best one that is available for the systems.
- More raid controllers help in case of performance.

[poulpreben]

Hi pirx,

Maybe it is just the appliance lent to us by HP, but we realized that all backplanes are SATA based - not SAS. Personally, I would be slightly concerned to trust the ATA with my corporate data simply due to the lack of additional verification mechanisms in the data channel. There is a great write up on the differences between SAS and SATA here: http://www.enterprisestorageforum.com/s ... ata-1.html

Again, it is possible that you can get a SAS variant, but the one we had in for testing was not.

- Preben

[Andreas Neufert]

Correct there are several options including SAS Backplanes.
So best practice is to use SAS backplanes and Nearline SAS disks or faster SAS disks.

[pirx]

- User forward incremental Veeam backup modes to process more data sequentially.

So you would not recommend to use this system for reverse inc backups? I'm not fully up to data with Veeam but in the past reverse inc was the recommendation in most cases (with higher IO requirements on the target side).

[Andreas Neufert]

Reverse Incremental is in the product since v1 and is a good recommendation.
Depending on the amount of VMs you want to backup to that apollo system and the nature of the Apollo Raid Controllers forward incremental chains work a bit more performant on this system as more sequential processing happens on storage side.
In most cases Reverse Incremental is as well a good choice for Apollo systems. Just test and compare it.

[NightBird]

Hi pirx,

Maybe it is just the appliance lent to us by HP, but we realized that all backplanes are SATA based - not SAS. Personally, I would be slightly concerned to trust the ATA with my corporate data simply due to the lack of additional verification mechanisms in the data channel. There is a great write up on the differences between SAS and SATA here: http://www.enterprisestorageforum.com/s ... ata-1.html

Again, it is possible that you can get a SAS variant, but the one we had in for testing was not.

- Preben

SATA backplane with P840ar SAS controller ? are you sure ?
The datasheet do not mention it...

[poulpreben]

Pretty sure, yes. It is using SATA 6 TB drives and Intel S3500 SSDs (the SATA version of S3700).

[NightBird]

Don't forget that SAS backplane is backward compatible with SATA drive.
I think that you can plug SAS ou SAS MDL drives without problem with the embedded P840ar controller, so the backplane should be SAS

For the read hdd cage kit you can plug it to the embedded B140i SATA controller (that is SATA only) or with the optional option HPE
SAS Controller Mode for Rear Storage
The main 24 LFF or 48 SFF drives cage backplane should be SAS

[poulpreben]

In that case, just ensure you get it configured with NL-SAS drives instead of SATA

Thank you for the clarification!

[NightBird]

With scale-out backup repo it should rocks

[pirx]

Reverse Incremental is in the product since v1 and is a good recommendation.
Depending on the amount of VMs you want to backup to that apollo system and the nature of the Apollo Raid Controllers forward incremental chains work a bit more performant on this system as more sequential processing happens on storage side.
In most cases Reverse Incremental is as well a good choice for Apollo systems. Just test and compare it.

Testing for an environment with 1000 VMs is always a bit difficult. We would need at least 4 physical repository/proxy servers for our environment. So each of the Apollo's would need ~100-150 TB (VM usage is ~180 TB and we want to keep 15 weeklies and 14 dailies). I can test this with a couple of VMs but nothing near 1000. So a good recommendation of what is possible and what not would be nice. There is always the option to use an SAN device for this. I'm evaluating the price difference at the moment. This is easier than the performance to be honest.

[Andreas Neufert]

Both will work for sure. (Reverse and Forward Incremental).
Forward Incrmental would work better with Apollo Storage and Controllers. So my suggestion is to use it.
If you backup a good number of VMs and Reverse Incremental finishs in nearly the same time (VM snapshots are a bit longer open) there is no blocker to use RI.

Just one thing needs to be considered at Apollo. MS Filesystems support Checkdisk processes only up to 64TB. So you should create multiple volumes at the array to be on the save side at Microsoft.

[pirx]

The filesystem gives me some headaches. Even on our larger fileservers we don't use NTFS filesystems larger than 10 TB. And we are having enough issues with them.

[Andreas Neufert]

You can install Linux on the apollos as well so that you can use them as repository servers => then you have to create additional Proxy and Veeam Servers.
Maybe as well an idea to use it as host and ceate backup server + linux repo + windows proxy on it... maybe with Hyper-V to use the Host as DirectSAN Backup with FC if needed.

[bg.ranken]

So poulpreben, do you have any write ups or anything else on the 4200 yet or was what you posted in this thread the only information you had so far?

I'd like to know what you actually recommend in terms of configuration you would change from the model you got. Were you happy with the performance of the SATA controller or would you say it's best to stick with SAS? Any performance benchmarks done? Any other configuration issues other than the lack of FC HBA?

[pirx]

You can install Linux on the apollos as well so that you can use them as repository servers => then you have to create additional Proxy and Veeam Servers.
Maybe as well an idea to use it as host and ceate backup server + linux repo + windows proxy on it... maybe with Hyper-V to use the Host as DirectSAN Backup with FC if needed.

If we use the Apollo as physical repro server with linux we would also need additional physical servers as proxies with FC connection? With "use it as host and ceate backup server + linux repo + windows proxy" you mean a host as hypervisor with a linux VM on it and proxy and backup server installed on it not as VM?

[Andreas Neufert]

@Prix, correct. That was just an idea. You can add the capacity to the VM with path through disks.

@Randall,
In our Lab we got a Apollo 4200 Gen9 with SATA configuration and smallest CPUs. Primary Storage System was a 3PAR with 18 SSDs and round about 30 SAS and NearLine Disks. Apollo was by far not the bootleneck here when we backed up some hundred VMs. As it was not the bottleneck any pefromance publishing would be not got. At Active Full it was at least able to handle 1GB/s Backup Stream (this is read at Source).

The Apollo isn´t that complicated pice of hardware. It is a Standard Server DSL380 gen9, Raid Controller and direct attached storage in one chassy. So everything what you would size on such a default server system apply to the appollo as well.
- Buy the sweetspot of CPU (price/performance)
- Add enough RAM 64GB (RAM is cheap)
- Add the best RAID controller that you can afford (at least battery buffered with the option of Raid10, Raid50 and Raid60). Don´t save money here!
- Add FC HBAs with max speed of your environment if you want to do DirectSAN withFC. If not available in the official option list, you can add standard FC HBAs as it is a normal server with PCIe Slots.
- Add 10GbE cards or use default ones if available
- Use the Best disks you can afford.
- Use SAS disks and Backplanes (you would not buy a SATA Server for your hosts... so you should not buy it as well for backup)
- Redundant Power
- Colling + USV is important here as well.
- Protect the Server physically

By design of that System you can backup some hundred VMs for sure to that system. It will perform well in most of the scenarios. If you think that you would use 100% of the Disk Subsystem in case of Capacity and Space together with thousands of VMs, you should think about to buy a more redundant storage system at one point. For example some DL380 server with a small 3PAR and redundant controllers and Storage pathes.

[poulpreben]

So poulpreben, do you have any write ups or anything else on the 4200 yet or was what you posted in this thread the only information you had so far?

To be fair, Andreas has conducted all the testing for the whitepaper, so I totally agree with the checklist he has already posted above Since we were not able to make the Apollo server sweat even at 1 GB/s backup throughput from our 3PAR, I am planning on doing some simple I/O tests with 'fio' this Friday. We are not including these tests in the whitepaper, as the configuration we are testing only has SATA drives, and thus the performance results will be worse than what most other customers will be experiencing. If you are interested, I will be more than happy to post them here.

[bg.ranken]

To be fair, Andreas has conducted all the testing for the whitepaper, so I totally agree with the checklist he has already posted above Since we were not able to make the Apollo server sweat even at 1 GB/s backup throughput from our 3PAR, I am planning on doing some simple I/O tests with 'fio' this Friday. We are not including these tests in the whitepaper, as the configuration we are testing only has SATA drives, and thus the performance results will be worse than what most other customers will be experiencing. If you are interested, I will be more than happy to post them here.

Oh I'd be more than happy to get more information, this is honestly the most intriguing backup device I've seen in a while. While I know Veeam is storage agnostic I remember when Gostev last spoke about something that was pretty different it turned out pretty good (Cisco C3160 I think?)

Honestly storage density + price + performance seems to be at just the right sweet spot. Are you planning on releasing the whitepaper as well?

And Andreas, thanks for all the work you've done on this. I actually have a question regarding that 3PAR option. I'm not a big enough shop that we're into the 1000s of VMs so I don't have insight into a bigger network like that, but wouldn't it still be a better option to go with local DAS based on the new features in Veeam 9? At this point it seems like you could throw multiple servers with DAS (Apollo, DL380s, or other variant from other vendors) and combine it with the scale-out repository and backup copy jobs to get completely redundancy in your backup data. I'm not sure if it would be cheaper once you get into the bigger scales but it still seems with the same disks that you would get better performance over DAS to the same disks than you would get with a 3PAR over FC or 10g E. I'd be interesting to hear your thoughts on that.

[pirx]

I received a quote for a 4510 with 2 x E5-2630 and 256 GB RAM, 1 x 10GB NIC (no FC HBA yet), P840 RAID controller and 47 x 4 TB SATA 7.200 disks. The price looks promising but I think I've to readjust some parts. 256 GB maybe is a bit too much for repository + proxy server what is more reasonable number? 64 GB? I guess we will need 4 server for our 1000 VMs. A backup copy will be stored on StoreOnce. The choice of the disks and RAID configuration is also not easy, the HP quick spec document lists 2 pages with drives (http://h20195.www2.hp.com/v2/GetDocumen ... c=de&lc=de).

Currently configured:
HP 4TB 6G SATA 7.2K rpm LFF (3.5 inch) Low Profile Midline 1yr Warranty Hard Drive 797265-B21

I think SATA is not a good idea, maybe SAS with 7.200 rpm?

HP 4TB 6G SAS 7.2K rpm LFF (3.5inch) Low Profile Midline 1yr Warranty Hard Drive 797267-B21

The price of this drive will be ~double compared to the SATA one.

[poulpreben]

64 GB is a bit on the small side. I think 128 GB RAM should be sufficient, as you only have 12 cores in total. But RAM is not going to ruin your budget anyway, so I suspect you will not save big bucks by cutting back on that.

SAS is more expensive than SATA for a reason. It is simply a much more reliable data path (I can only recommend reading the article I linked earlier). If you can afford it, definitely go for SAS.

For estimating the storage requirements for you environment, you can always have a look at the Restore Point Simulator here > http://rps.dewin.me.