StoneFly Air Gapped Backup Repository

[kmeyers@iupat.org]

First post and I was not sure where to put this question, so I tried to put it in best category.

We are starting to looking into a new Backup Repository for our 5-6TB (raw unencrypted) daily backups. What is everyone's impression of StoneFly? It seems to be a well-built suite of hardware and tools.

I am most intrigued by their Air-Gapped Backup Repository and wondering if it's full proof and if anyone has any experience with it?

[HannesK]

Hello,
and welcome to the forums. I just searched for StoneFly and it seems you are asking for another backup product in the Veeam forum (https://stonefly.com/backup).

Can you please explain what exactly you mean and how you want to use that box with Veeam?

Best regards,
Hannes

[kmeyers@iupat.org]

Hello,

Sorry for the misconception I a referring to their Veeam-Ready backup appliance. (https://stonefly.com/backup/dr365-for-veeam) I have heard of other people in these forums using their appliances as backup repositories and was curious about individuals' thoughts on their hardware.

They have a new piece of tech that allows for an Air-Gapped Backup Repository and was also curious if anyone had experience with it.

[JHuston]

I've been evaluating their solution as well and although the price sure seems right, i'm passing on it. It seemed to me that their "solutions" are just old tech all put together in a new way and manual processes. The way the air gapped solution was explained in my demo is basically a storage lun that you can offline manually in the system after your backups are complete. Then bring it back online when you want to back up to it, rinse and repeat. Meh....

Read their system admin guides, the impression I got was old feeling tech and heavy admin overhead to get things done.

Also, I could not find them on the Veeam partner list even though they say they are...? Just didn't get the "warm fuzzies" from their solution if that makes sense.

[Andreas Neufert]

Hello everyone.
Stonefly is not part of our Veeam Ready program. https://www.veeam.com/ready.html

Update 2020-07-23: StoneFly went through the official testing and achieved Veeam Ready Repository status. https://www.veeam.com/kb3241

[Mikeymas]

Stonefly seems to be a technology partner with Veeam and Microsoft.
https://www.veeam.com/wp-how-to-deploy- ... azure.html
https://stonefly.com/backup/dr365-for-veeam
https://azuremarketplace.microsoft.com/ ... b=Overview
veeam reseller and cloud service providers. they also participated in creating iSCSI protocol for the industry and have in business
for 20 years, iSCSI.com. they are also listed and veeam reseller and cloud service provider.
They sell their appliance mostly to Veeam and other backup manage service providers. They also have a lot of government and large commercials listed as customers.
I guess they should fill out the Veeam ready form to make veeam and us happy.

[Andreas Neufert]

Sounds good. Stonefly uses "Veeam Ready" status on their pages without receiving the corresponding self-verification approval from Veeam (see comment above by Kurt).

Update 2020-07-23: StoneFly went through the official testing and achieved Veeam Ready Repository status. https://www.veeam.com/kb3241

[dpeach01]

@JHuston, That is exactly my impression as well. I reviewed the product a couple of months ago. It looks good until you start asking questions. Deduplication looks like a nightmare to manage. The 'Air-Gapped backup" is simply a separate NAS device that is kept offline until needed. It uses physical nodes running VMWare or Hyper-V as controllers. It apparently has some sort of HCI implementation, but who knows. I did a customer referral call and wasn't impressed. At the end of the day, I will likely fall back to my priors - Exagrid.

[PastequePro]

I did an initial call with them in ~November, was convinced to put in a PO, and have had it for a couple months. I've been really happy with their appliance and from what I gathered the references I talked to had the same experience. This is what I can tell you from what I have seen so far. The gist of the product is that a purpose built Backup and DR appliance with integrated Veeam (version 9.5 or 10 - im running 10). For me they offered up to 20 Scale Data Movers, which have really helped with concurrent backup jobs and shortened my backup windows. The DR365v appliance I have has 3 Storage Engines (1) SAN-block (iSCSI, Fibre) for high IOPS (2) NAS-file for Data movers and user direct restore, (3) S3 Object (AWS, Azure Blob) which I use as a on-prem long term archive.

I have been really impressed with the data services that were included in my package. I implemented their air gaps and I feel pretty confident that they will protect me in case of an attack. Specifically, I use their immutable snapshot, WORM, object lockdown, hardware-based encryption, and anti virus. I've partially implemented their auto transportable lifeline and sync/async replication to cloud. I'm thinking of either using a physical or virtual appliance in my remote datacenter as well - not sure yet as depends on my budget.

The feature that has really taken the cake though has been their public cloud backup (azure, aws) that has enabled me to do a direct restore from my backup in Azure to Azure without download to an on-premise appliance. I have a really short RTO requirement and so its been really practical for me to have the option of spinning up in Azure. Plus, its been cost effective as they bundle Azure and their cloud connector together for a reasonable flat rate so I dont have to worry about the variable and sort of unpredictable Microsoft cloud cost. Their air gap features are also available for their cloud offerings which is pretty neat too.

Specs of my appliance if its helpful - My appliance had a built-in NVMe and was built to order. I have Flash SSD and Dual power supply. I got a 12 bay but they offered a range from 4 to 90 bay in a single and Dual-HA controller node. They told me their appliance can scale up and out to 128 nodes. However, to be honest, the best part of their offer to me was the price. It was 1/3 of what other vendors quoted me with a lot more features so that what really got me.

[Andreas Neufert]

Not really sure why you need NVMe in a backup target but OK it can be fast.

Maybe as StoneFly Resellpartner, you can answer the following?

1)
Maybe you can explain why you think it is more cost effective to use the StoneFly to work with Object Storage compared with our native integration?
Veeam transports only deltas forever into the cloud (fast block cloning approach) and is optimized for egress charges reduction by actually restoring most of the blocks from on premisses.

2)
As well can you please explain how you achieve a "cloud internal" restore as highlighted above? You can always do this the same way with Veeam.

3)
Offloading on NAS or block written data to object storage within the storage system is not really a good option and not supported for specific technical reasons: https://helpcenter.veeam.com/docs/backu ... ml?ver=100
As well storage internal replication are not treated as additional copies in the Veeam world as if one chain is affected for whatever reason the second will be affected as well.

4)
Do you have knowledge how the system protect from Admin access delete of the Immutable storage snapshots? The most critical access pattern that we see in support is when an hacker got access to the systems iteself and delete things.

[PastequePro]

no problem Andreas - you could use it for backup but it would be too expensive. StoneFly appliances use NVMe for multiple use cases like Running the Base OS, Dedup Indexes, Flashcache, Tiering, spin up, Fast Storage, Logs etc .. especially for Heavy I/O activity. I have found that it has optimized and tuned all our core components.

Answers to Questions 1 and 2:

Its been my experience that Veeam uses Object Storage by adding an Object Storage Repository which can't be used directly, it needs to be tiered with another Backup Repository/Extent which can be based off of NAS or ISCSI. This way you end up with a SOBR that can be used as the target for Backups/Archives. The big BUT here is that you need to size the local extent large enough so that it can fit the backups, as they are copied/moved to the Object Storage immediately after job completion or later on depending on how you configure the settings. In other words you can't really save the backups directly on the Object Storage. but with Stonefly's you can do it directly.

Dont get me wrong - I use Veeam integration for cloud archive and its great. I use it for certain use cases where I don't have a short RTO requirement. What I had said in my comment was that I use StoneFly for on-prem s3 local target archive storage (on-prem, in the Stonefly appliance not the cloud). Another feature that I mentioned using was Stonefly Cloud Connect, which doesn’t have any egress charges as it's a direct restore from the cloud to the cloud. Additionally, it provides similar Air-Gapped features and data optimization in the cloud like the ones in my on-prem appliance. So to answer your question, Stonefly’s cloud connect is more cost effective for me because of the lack of egress charges and data optimization features I can use. On top of this I don't need to have the staging storage/local extent on my on-prem appliance to get all this done.

Answer to Question 3:
Thanks for that article. I read it and definitely agree, but StoneFly does offload NAS to Block. It uses an object as an object. I believe it’s part of their cloud package technology. From my discovery process, I have found that they have been providing these services to a lot of Veeam Cloud Connect Service Providers and End-Users for years now.

To your second point - yes, I agree. You need to have both – async replication with no retention which is near real-time and replication with the user-defined retention policy, which stonefly's features can provide.

Answer to Question 4:
Again, I agree 100%. Thats why I feel you need a lot more Air Gapped features in the product (and kinda the main point of this thread). StoneFly offers different tiers of air gaps to prevent various threats - two of such tiers I think specifically address admin access deletion.

feel free to throw me some more qs if u have any more - too much time on my hands with this whole lockdown thing haha I'm a data migration specialist but seems like im on my way to becoming a Veeam specialist too

[Andreas Neufert]

Update: StoneFly went through the official testing and achieved Veeam Ready Repository status. https://www.veeam.com/kb3241

[Mccarlin]

Have been impressed with StoneFly. Look to be purchasing soon! It was 1/4 the price is Exagrid.

[richenroth]

I am using the Stonefly product with veeam B&R.
When a backup job completes the file copy job starts which fires off a windows/API script mounting the air-gapped volume on the linux proxy:
/mnt/veeambackup02 or whatever you have named it.
Next the file copy job block syncs the changes from /mnt/veeambackup01/job to /mnt/veeambackup02/job
upon completion the veeam script umounts the volume, hence the air-gapped ransomware-proof repository.

The supermicro computer inside the Stonefly appliance has it's own exsi host, Windows server 2019 and Linux proxy server with 128gb system memory and a 512GB NVMe SSD for the OS.

You can use it as a nas, san, nfs file server and ISCSI with AD and LDAP
In my use-case the backup network is 10g on a closed subnet between my 3 vmware hosts and vsphere.
Our Stonefly model XD-Series DR365V has 128TB formatted raid 10 configuration with 18 16TB drives in the 24 bay 4u appliance
Cost was under 40k about 1 year ago and includes Server 2019 16 core license and a 4 core add-on for the Backup Management VM.
Also included is the 5 year 24x7 Tech support with advanced parts replacement, professional services/Remote installation, configuration, testing and 10 hours of training.
The Stonefly support is excellent and friendly, they help move your existing veeam backup plan to the stonefly and anything else you need help with. I can't say enough about Stonefly support, they really are a world class organization.