Budget SAN ideas for DR storage

[bbricker]

Sorry in advance for the long post. I am looking to put together a budget SAN for our DR site. This will be used to store Veeam backups (40TB), vm replication (20TB), and a large amount of radiology imaging (250TB and growing rapidly). So I need about ~300TB to start with (minimum, would probably start closer to 350) and would like to have something that will grow to over a PB in the coming 5-6 years. I prefer SAN versus server+DAS for better redundancy and growth, not to mention for replication of all vm's, but I am open to ideas, or a combination of things. I would prefer not to silo data and make it harder on myself to grow later.

The current setup at our DR site is an old EMC CX4 SAN and a Dell PowerEdge server with internal storage and DAS storage via MD1300. The CX4 is old and needs to be retired, and as I mentioned, not a fan of silo'd storage on the Dell server. We are also limited on rack space and cooling at our DR space, so I need something more dense than the CX4 which is limited to 30TB per 3U based on the 2TB drive limit in 15-drive enclosures. Yeah, its old.

I have a VNX5300 that was just retired from our Production site, and it has ~100 x 4TB drives. We initially considered replacing the head end (since VNX5300 maxes out at ~115 drives), but even if I swapped over to a head end that supported say 500 or even 1000 drives, I would still run out of physical rack space if I continued at 4TB x 15 in 3U. That is just not dense enough for future growth with the limited rack space we have.

So one route I am currently looking at is taking the ~100 x 4TB drives from the recently retired VNX and putting them into a refurb VNX2 (maybe a 7600) and using the newer 60-bay deep disk enclosure option, which can hold 240TB raw in 4U of space. I would combine that with a couple 25-bay 2.5" disk enclosures that could hold SSD's and/or 15K SAS drives. That way I'd have a high speed pool of storage for replicated vm's, and then a "slower" pool for the Veeam backups and radiology data. The obvious advantage of this is having nearly 400TB RAW of disks that I already own and I know the history on (which is extremely low IO load). The only issue with this plan is it's possible the 4TB drives from a standard VNX 15-bay enclosure will not be compatible with the 60-bay enclosure as they use different caddy's for the drives. I am still trying to get a firm answer on that. Any one here have experience with that?

The other thought, and the reason why I made the post- was to see if there are other "budget" SAN ideas from the Veeam community, and branching out away from EMC which I have used almost exclusively the last 15 years (in fact we just bought a new UnityXT for Production). The goal would be having the capability of both "cheap and deep" dense drives (8TB+) as well as high-speed SAS or SSD, and having SAN capability (prefer Fibre Channel but I could do iSCSI) so I could have multiple hosts connected. Growth to 1-2PB would be necessary, preferably without downtime to add storage. Cost is also a concern.

Thoughts / suggestions? thanks!

[alex1992]

try TrueNAS

[NightBird]

Dell ME4 with ME484 (84x3,5) enclosure ?
With 16TB drive

[bbricker]

try TrueNAS

Honestly hadn't really considered NAS. How well does Vmware run on it? I see their website says they are Vmware compatible. I always had the impression NAS as a back end was not "best practice" for good performance.

[bbricker]

Dell ME4 with ME484 (84x3,5) enclosure ?
With 16TB drive

That looks promising. I didn't realize Powervault could be a SAN now. Thought it was all DAS. I'll read more about this.

Keep the tips coming!

[Gostev]

Cisco S3260 and HPE Apollo 4150 are most common used within the Veeam customer base, so they are most proven. You can look for cheaper alternatives for example from SuperMicro, but they will likely not have as good of RAID controllers, connectivity etc.

[NightBird]

Anton, I think he wants san for redundancy

He said «  I prefer SAN versus server+DAS for better redundancy and growth »

[Gostev]

Thing is, for example S3260 too has redundant power supplies (4, redundant as 3+1) and enterprise-grade RAID controller that is actually better than in most low-end SAN. If you really need more redundancy than this, then unfortunately the following requirement will need to be removed "Cost is also a concern", as you're looking at an Enterprise-grade SAN next.

Growth is super easy with Veeam via scale-out backup repository.

I would not be so confident recommending this, if we did not have Veeam Cloud Connect service providers hosting petabytes of tenants' backups in those boxes successfully for many years now (originally it started from Cisco C3260).

[NightBird]

I'm agree with you, I love HPE Apollo or DELL R740xd2 (or DELL Server with ME484 Jbod) for hosting Veeam Backup.
Agree too with "Cost is also a concern".

But he wants to store VM replicas too, he wants to connect multiple hosts too.
I don't think mixing backup and replicas on the same storage is a good idea too.
DELL ME4 is an entry level Seagate OEM SAN (as HPE MSA1050/2050, or Lenovo, may be others).

A lots of needs, with "cost concern" there're no miracles !!!

He should buy a second Unity XT SAN

[Gostev]

Sorry, I missed the "replicas and multiple hosts" part.

[bbricker]

Yeah I definitely am in the "want my cake and eat it to" category here. Ha. Thanks all for the comments, I'm loving it.

I have thought about splitting the 2 needs out:

1) "Small" but decent performing SAN with ~30TB of fast disk/SSD for my replication and running vm's needs

2) Cheap/deep server/DAS/JBOD with loads of big disks for Veeam backups and my radiology imaging backup. Honestly this would be fine, just as long as I could continue to scale that platform from ~300TB up to 1-2PB. Maybe this is the way to go.

I have been looking at the ME4 platform. Looks like it can be purchased as either SAN or DAS/JBOD.

Nightbird- you mentioned, "Seagate OEM SAN"... what do you mean by that? Is Dell-EMC leveraging another product?

From the product details website, it sounded fairly capable:
- ADAPT (Distributed RAID): Improved data protection functionality that delivers faster drive rebuild times
- Thin Provisioning: Allocate and consume physical storage capacity as needed in disk pools
- Snapshots: Easily recover data with point-in-time copies
- Replication: Replicate data to any global location
- Volume Copy: Enables seamless volume relocation and disk-based backup and recovery
- SSD Read Cache: Increase execution speed of applications by caching previously read data
- 3-Level Auto-tiering: Optimizes data performance with less expense
- Virtualization integration: Integrate ME4 arrays with VMware vSphere and vCenter SRM and Microsoft Hyper-V
- Up to 320K IOPs performance / Up to 5500 MB/sec bandwidth
- Up to 4PB capacity (Use HDDs to 4PB in three 5U 84 drive dense enclosures)
- Performance at scale with 12G SAS backend

In fact, I am kind of scratching my head wondering why Dell-EMC and my partner/reseller didn't bring ME4 to my attention for our production side instead of Unity XT which was definitely a few factors more expensive. And it's not like I told them I needed high-end performance, they knew we had a small Pure AFA for those needs.

[DaveWatkins]

That looks promising. I didn't realize Powervault could be a SAN now. Thought it was all DAS. I'll read more about this.

Keep the tips coming!

We have an MD3860f for our backup repo and haven't had any issues with it (Ours is FC). Same concept as the ME4084. You can add the ME484's to the ME4084 if you need more shelves too

[nitramd]

@bbricker,

I agree with @alex1992, take a look at TrueNAS/iXSystems. The scalability of iXSystems M series storage is eye-popping: https://www.truenas.com/m-series/ . Their M60 device is capable of storing 20PB (max capacity) of data.

iXSystems products are not as expensive as one would think. I'd suggest that you request a quote - nothing to lose and a lot to gain.

Good luck.

[derekmerritt]

I think a Dell Compellent SCv3000 SAN might be perfect for what you need on price, density, and performance. We recently purchased a SCv3000 and an SCv300 expansion tray to serve as the primary datastore for a new vSAN cluster. We carved out ~30TB for VMs, and the rest is for storing video archives. Right at 300TB usable in 5U of rack space, for a little less than $60k. 28 x 12TB 3.5" SAS drives in 2 enclosures (5U total), all 10GB SFP+ iSCSI networking. And Compellent makes scalability pretty easy, so when you need more storage, buy another expansion tray (12 drive, 2U) and add it to the cluster.

I think for the SCv3000/300 line the max drive capacity is 12TB, so to go beyond 300TB usable, you'd have to get another expansion. I don't remember what the disk size options were as far as 2.5" vs 3.5". I seem to remember 3.5" was the only choice though.

[Mikeymas]

Hello

May you want to look at

https://stonefly.com/backup/dr365-for-veeam
VVeeam Ready SAN-NAS-S3 Object Air Gapped Purpose build Veeam backup and DR appliance. Very affordable and compete turnkey. Talk to Jim West, he is very helpful.

[SimonS]

... This will be used to store Veeam backups (40TB), vm replication (20TB), and a large amount of radiology imaging (250TB and growing rapidly). So I need about ~300TB to start with ....

You can look to FAST-LTA Silent Brick - https://www.fast-lta.de/en/products/sil ... -overview/. Storage is primarily designed for long term storing of radiology imaging, and is also works very fine with veeam.

[markharford]

A couple of good sources for working out solutions and finding out usable vs raw capacity, IOPS, etc can be found here:

https://midrangesizer.emc.com/

HPe 'One Config Simple'
https://h22174.www2.hpe.com/ngc/Welcome#

Obviously vendor specific but even so they give a nice way to calculate sizing which you can extrapolate to others to some extent.

[pesos]

How many hosts are you looking to serve? If you are ok with 4 redundantly-pathed hosts (or up to 8 single pathed hosts) then we have always been very happy with shared sas (dell powervault MD or hp msa2050 based) instead of the tuning and extra infra required with iscsi or fc. Super simple, plug and go, and 12gb/s with no tuning required. I now prefer the HP route because they don’t require hp certified drives the way dell does, so you could easily reuse your storage.

[rbance67]

Since you are looking at block storage, I think the Lenovo DE series might be what you want here. There are a number of different configurations.
The DE6000H comes as a 4RU 60LFF version and you can put 16TB drives into it. This is essentially a NetApp array too..
You can get expansion trays (up to 4) with 60 or 40 LFF disk configurations as you storage requirements grow.

Just did one like this...
1x DE6000H (60x 16TB HDD)
3x DE600S (60x 16TB HDD)

btw...I do not work for them, but this is a good solution

[Seve CH]

Hello,

I have 20 years experience on Hospitals, so I understand you .

What I won't do: Internal storage. It does not scale. Do not use any VNX2. It will reach EOL soon and the maintenance costs more than just buying a dedicate new system and the performance will be similar or lower.

What I would do: Look for ADAPT (Dell) or DDP (Netapp) arrays. AFAIK, Lenovo discontinued the IBM line which used that kind of distributed RAID implementation. The more disks, the more IOPS and shorter rebuild times. You can mix disk types on the same array (different Pools) if you wish and they can be extended in most cases one disk at a time.

I was really happy with Dell MD3400 (SAS, Veeam backups) and MD3860f (FC, we were running the PACS and documentation archive VMs direct on it using 8 and 10TB disks only, no flash). I am talking about 200 (MD3400) and 400 TB (MD3860f) of data. They were Netapp/(LSI?) systems with Dell software and labels.

They were extensible with a granularity of 1 hard disk. : "So... you got a new modality? yes? buy me 3 more 8TB disks for the backups and 2 more for main storage". Install, extend, happiness. No need to prepare huge budgets to buy a complete shelf with disks, licensing, etc. Each disk at that time costed around 800$ each and provided additional 70 IOPS

Now I am testing the Dell ME4 series (Seagate based) and they look very good. We got 240TB net for main backup, 350TB for backup archiving (Veeam repos). Once you see the price and performance of these systems (~85$/TB, double SAS controller, 2100 IOPS for our archive), you stop considering cheap NAS solutions for anything professional. If you buy the ME4 SAS, do not forget to get a Dell SAS HBA controller (they certify one SAS-HBA model only).


Regards

[rweis]

I had great success with Imaging & Video backups on Nexsan storage when I was the consulting storage architect for a large Maine health organization a few years back. Nexsan has a very fast rebuild time for even the largest drives (16TB). E Series and Beast are both highly scalable, dense and provide very reliable imaging and video storage.
I found the technical assistance, set up and performance to be excellent at the time.
Disclaimer: Investigate Nexsan's current product offerings and corporate status (I think it was recently acquired) - my first hand knowledge is 3-4 years out of date.

[WorkingHardInIt]

We have used the PowerVault for this or the DELL EMC SC series (iSCSI/FC)for these purposes. Today I would choose SC for all needs, but is still use the older PowerVault solutions for secondary copies.

[ddayton]

try TrueNAS

I have used FreeNAS, now TrueNAS, using iSCSI with VMware ESXi 6.x with 10TB and 20TB with good performance, but no idea about scaling out to this huge size. Their X series is rated for 1PB...

The TrueNAS X-Series is our compact 2U enterprise storage system built on the powerful OpenZFS file system for unbeatable value and performance. Available in hybrid or all-flash configurations, the X-Series easily integrates into any environment with support for all major block, file, and object protocols.
A single X-Series system can support up to 1 petabyte of raw capacity and is over 70% more cost-effective over five years compared to AWS and other cloud-based solutions. Powerful enterprise features, like ZFS data protection and high-availability, ensure up to 99.999% uptime to keep your business running even when components fail. Choose from gold, silver, or bronze Enterprise support packages with flexible options from 24×365 support to on-site repair from our US-based team.

The TrueNAS M-Series is the flagship model of TrueNAS Enterprise systems and is ideally suited for heavy IT storage workloads and intensive data center applications.

The TrueNAS M-Series provides High Availability (HA), hybrid capacity, and all-flash performance configurations. The M-Series grows to support multiple 100GbE ports and over 20PB of storage. Read the ESG Technical Report on the M50 performance.

[FECV]

Funny thing is I have both the Dell SCv3000 as well as the Dell ME4084 (i know it was the 484 mentioned above) for my onsite Veeam backups. I have them using direct 12Gbps SAS versus ISCSI or fiber as I only need them connected to one or two servers and did not have the budget for 100GB networking setup. While my experience with both units has been positive and have had no issues with Veeam VBR i will give you my thoughts on both units as it applies to the Veeam backups. I do not have experience with these units and VM replication. My SCv3000 i have had for a few years. Its has 136 10TB drives. I am currently using about 900TB out of the estimated 1.2TB total. While i like this SAN over all i think this unit is overly complicated for just backups and actually prevents certain configurations i may want to use. However the things i don't like may be good for your VM replication environment. My first growing pain coming from MD Dell units is that it forced all my new backups to a RAID 10 Dual redundant tier. Under typical use case for this unit as the data gets older it would go to what it calls a RAID 6-10 tier. You can imagine buying a 1PB storage solution and only getting a quarter of the space. I finally found there was a way to create a storage tier and force it to the lower tier from the start. Something Dell support could not figure out. The ME4084 with the 84 12TB drives is a bit newer as i got it this year. This i think is a better solution overall. You have more control over the layout of the disks and can create say a high performance volume and then a large volume for backups. It will offer you the ability to expand as you grow, but the more disks you can spread-out your backups onto the better performance you will have so you may need to find a sweet spot on cost to disks to performance. If you can not afford the FC connections and cards, i would recommend looking into the SAS controllers. I find them higher performing and easier to deal with than the traditional 10Gbps iSCSI.

[rbance67]

AFAIK, Lenovo discontinued the IBM line which used that kind of distributed RAID implementation. The more disks, the more IOPS and shorter rebuild times. You can mix disk types on the same array (different Pools) if you wish and they can be extended in most cases one disk at a time.

The Lenovo range is essentially NetApp. The features you mention are part of this range now..

[bbricker]

All of these responses are awesome, thanks so much. I knew I could count on the Veeam Community for some good tips. I'm currently looking at Nexsan and Dell ME4 right now. Will dig into the others, too. Thanks guys!