Discussions related to using object storage as a backup target.
Post Reply
dougsoltesz
Novice
Posts: 6
Liked: 2 times
Joined: Mar 25, 2013 9:00 pm
Full Name: Douglas Soltesz
Contact:

Object Backup Repository supporting OpenStack Swift & S3

Post by dougsoltesz » 1 person likes this post

All Object Storage Users,

I am very happy to see that in v8 Veeam is continuing to add new Backup Repository types (Deduplicating Storage Appliance). I would like to propose that Veeam look into supporting OpenStack Swift and S3 as a Backup Repository type for a future version of Veeam.

I am aware that Veeam v7 had the Cloud Edition, which had a way to send Veeam images to the Cloud, and that Veeam v8 has Cloud Connect enabling cloud backup to select partners. However, my company and I'm sure many others are implementing private cloud (onsite) object storage systems as data is increasing expodentialy. This storage is highly economical and perfect for a backup target.

I also understand that Veeam today writes a large file, and object storage is more tuned to many smaller objects. However, seeing as Veeam reads 256KB to 2MB chunks of data from VMware CBT, I'm sure that the brilliant developers at Veeam can figure out a way to store these objects and use a database or object manafest to tie all the pieces back to together.

If others would like to see this implemented in Veeam future, please reply to this thread, and hopefully Gostev will work his magic.

Doug Soltesz
Budd Van Lines
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by Gostev » 3 people like this post

dougsoltesz wrote:[Object] storage is highly economical and perfect for a backup target.
Actually, neither of these statement are true, at least from my experience.

Regarding the "highly economical" part, the math is really easy: data ends up on the same hard drives having the same price tag. However, object storage requires more expertise to manage (comparing to classic file systems), and is less reliable (new technology, comparing to polished general-purpose file system). Finally, object storage does not provide self-contained files, thus significantly complicating data migration, synchronization, recovery, transportation, export, tape archival – which are all very important backup data management tasks. All of the above facts result in significantly increased TCO of object storage based solution, comparing to general purpose storage on classic file systems. So, while object storage may be economical comparing to some alternatives, it's still not as cheap as storage that Veeam supports and recommends our customers to use.

As far as "perfect for a backup target", we've actually found the opposite in our testing: performance of object storage is poor comparing to general purpose file system, so it's less than optimal as a backup target. Real-world feedback from hundreds of our larger Cloud Edition customers has only proved our findings. This is exactly why Cloud Edition was scrapped, and I just don't see us stepping on the same rakes again...
dougsoltesz
Novice
Posts: 6
Liked: 2 times
Joined: Mar 25, 2013 9:00 pm
Full Name: Douglas Soltesz
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by dougsoltesz »

Gostev,

While I agree that object storage does require more expertise to manage as it is a new technology, I could have made the same arguement years ago for VMware (virtualization) and many other disruptive technologies (Veeam).

I believe that Object storage is great for backups as it takes care of scaling up storage and is highly tollerant to failed drives, nodes, etc. Object storage is not a good use case for running VMs, but archival systems (Veeam), video survellance, log files, photo repositories all make a ton of sense. We are purchasing production object systems for 10x less than traditional systems offering up NFS / CIFS. Every big player is currently deploying object storage at a fraction of SAN/NAS pricing - EMC (Viper), NetApp (StorageGrid Webscale), HP (StoreAll), Quantum (Lattus). Lesser know brands are far cheaper.

Object Storage plays into the Veeam 3-2-1 model as it is a "different" kind of media, and it replicates offsite automatically.

I certainly don't want Veeam to step on any rakes. The Cloud Edition was not done nearly as eligantly as other Veeam features. It was simply a reskinning of the Cloudberry product. Cloud Edition did have poor performance, and did complicate restores, as it required multiple steps to get at the data. I for one would be glad to give Veeam input into doing Object Storage the right way.

I know you are fair, and if others don't want this feature, than there is no reason to invest any resources in it. I'm just asking the Veeam community if they see value, and to help me push Veeam to create a future repository type. Otherwise for now there are a number of gateways including Twinstrata which is a Veeam partner that can temporarily handle this impediment. ( http://www.veeam.com/success-stories/ve ... ckups.html )

Thank you again for responding to this thread,

Doug Soltesz
Budd Van Lines
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by Gostev »

dougsoltesz wrote:We are purchasing production object systems for 10x less than traditional systems offering up NFS / CIFS.
Sure, but how does it compare in price with a cheap physical Windows or Linux server stuffed with hard drives (or with a few JBOD DAS arrays connected, depending on your capacity requirements)? Can you provide some factual numbers as far as costs and capacity of what you are buying?
dougsoltesz wrote:Cloud Edition did have poor performance
Sure, but performance testing we have done was not using that. We were prototyping from ground up, trying to find out if object storage would work at least in "ideal" scenario...
dougsoltesz wrote:I know you are fair, and if others don't want this feature, than there is no reason to invest any resources in it. I'm just asking the Veeam community if they see value, and to help me push Veeam to create a future repository type.
It does not work this way, unfortunately. Ask about any feature at all that you can imagine, and you will always find people who "want" it.

My real goal is always to turn similar discussion into "why" they want it, and specifically what does not work in the capabilities we provide today. I would much rather enhance those capabilities, instead of pulling additional technology (such as object storage) into the picture.

Occam's razor principle teaches us to always go with the fewest possible entities, and this is the perfect principle for IT (because in IT, each additional entity needs to be managed, monitored and troubleshoot separately, which adds up to costs and complexities while at the same time reducing reliability of the overall solution due to more "moving pieces" involved). You know how every equation becomes so much simpler every time you remove one unknown from it? Then why add unknowns instead?
dougsoltesz
Novice
Posts: 6
Liked: 2 times
Joined: Mar 25, 2013 9:00 pm
Full Name: Douglas Soltesz
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by dougsoltesz »

Gustev,

Again thank you for the reply. I am glad to address each of these points as best I can.

Point #1 - Cost:

We purchased production Object Storage for around $100/TB this year, with support. However, I do understand the use case you present of a cheap Windows or Linux server stuffed with hard drives. I would like to run with this use case you presented.

A refurbished SuperMicro X8DTN+, with dual power supplies, dual Xeon 5520 CPUs, 24GB and an OS drive of memory can easily be found for $400. NewEgg is selling 6TB hard drives for $280. An enterprising Veeam backup engineer at a company can easily build a cheap Linux server with 60TB raw disks for $3,200 ($400 X8DTN+ & 10 x $280 6TB HD). This translates to $53.33/TB raw if you load the system with Ubuntu or Centos and use CIFS or Veeam Linux repository.

Of course the same system can be loaded with Ubuntu or Centos and OpenStack Swift for the same $53.33/TB. So the question is why pick Object Storage over a RAID of disks? Well, if the Linux system above suffers an outage, the FSCK can take days on most journaled file systems. Likewise a failed drive will take days to rebuild on the system. Object Storage is able to rebuild with an overhead of 1/Total drives. So if the system has 10 drives and you lose one, the remaining data is rebuilt onto the remaining 9 drives in 9 times less time.

Now lets assume that our admin likes what they built and wants to expand. They can add a JBOD, but this unit really has just a single controller and that is a single point of failure. With object storage, they can add more nodes, and the data is rebalanced between all of the nodes. Now if a node fails the data is still available on the other 2 nodes (the controllers “Scale Out”).

As we said before, this solution is more complex, but more resistant to failure. (A good reason to buy an appliance all ready to go) It gets even better if the admin adds nodes to another site (region). Now with 2 regions the data is replicated and balanced between the regions.

Point #2 - Performance:

My developers had the same issue with performance when first using object storage. It is a different animal than block or file. Instead of sending a single serial stream of data to it, it is better at concurrent streams of data. So depending on the architecture used in Veeam, it may be inferior for a single VM backup, but superior when multiple proxies are concurrently streaming data. Again, I would be glad to give input.

Lastly Point - Why do this:

I think the biggest why Veeam should add this feature is for your Cloud Connect partners. Surely they plan to handle Petabytes of data from this new and exciting v8 feature. I’m sure many of them are using Object Storage today, as it is cheap and replicates across multiple regions. You should check with them, I’m guessing they would want this type of repository more than SMBs.

Ultimately, while I believe in Occam’s razor, lets face it, Veeam is looking to be #1 in the market. Veeam has added Tape and Deduplication Archives as backup targets. This has increased complexity of the Veeam system and introduced a large number of moving pieces. While tape is dead (just kidding, long live tape!!!), object storage looks to be the future between cheap tape, and expensive Deduplication appliances. Just because the Cloud Edition in v7 was not successful does not mean that Object Storage is not a good avenue for Veeam to pursue. Ultimately, there will be a Veeam v9, which will prove to be better than v8 & v7.

Thank you again for taking the time to read my request. I really appreciate that you responded to my post.

Douglas Soltesz
Budd Van Lines
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by dellock6 » 2 people like this post

(note: heavy object storage fan and Ceph user here...)

Hi Douglas,
for what I understand, your main interest for object storage is not about the concept itself of an object, but its common way to scale using a scale-out approach. The two characteristics usually go hand by hand, but one does not implies the other. There are scale-out block storage solutions that can be "today" great solutions for Veeam repositories, and in fact many customers are already using it. Scale-IO, Ceph, but also HP StoreVirtual VSA, EMC Isilon and others. They all have in common the scale-out design, but even more important the "single namespace", that is the same final volume exposed to Veeam can be scaled in size by simply adding nodes, without the heavy forklift operation when you finish the available space on a jbod/nas/san and you have to move Veeam backup files to another location.

But, if we talk instead about Object Storage APIs, the whole scenario changes. Assuming we are talking about AWS S3 APIs (and products exposing their solution with the same libraries), here is where the problem with Veeam lies, and what Anton is talking about. Simply said, those APIs are designed to deal with a huge moltitude of small files like office docs, pics, and such. Veeam needs scsi-based storage because the backup files are, as seen by the storage, large files. Try to deal with a 10TB file (not only a Veeam one...) on any S3 storage and you will see by yourself. We did also some tests with some service providers and some commercial solutions, and the way the object storage deals with such files was simply too slow for any Veeam operation.

So, what's the solution? I usually suggest to use a NAS gateway as you said, or to split front-end and back-end of the backup repository: while the backend can be an object storage if it can deal with block volumes (I said before I use Ceph a lot, and the RBD block device is perfect in this scenario), the frontend will be a regular windows or linux mchine (thus be able to run the Veeam datamover) and is going to mount the object storage as a local disk device. But this design is not something that regular users can achieve based on their skills.

Also, your maths are missing a point: without local raid, which will have no sense in a scale-out design, redundancy is created by replication, so even with the smaller replication factor of 2 (two copies of every block/object saved into two different nodes), the price will be at least double of the cost of a single node. A possible workaround is to use erasure code, but it's not yet mature enough to guarantee performances like a non-erased volume, so some Veeam operations could be heavily impacted by it.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
zfs
Novice
Posts: 6
Liked: never
Joined: Nov 13, 2014 9:03 pm
Full Name: Data Integrity
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by zfs »

Hi dougsoltesz
I fully agree and also think that Veeam will need to be compatible with swift or s3. And yes Object storage is far superior for this purpose if the application is designed for it. So I think the question is when will it be supported ;-)
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by Gostev »

The answer to that is simple: currently, there are no plans to support object storage. However, I am more than open to consider your feedback on how can we enhance functionality of our existing backup repositories to avoid you having to pull a separate technology into your environment to sit in between Veeam B&R and those hard disks. Thanks!
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by dellock6 »

to both @dougsoltesz and @zfs (nice nickname btw!). You didn't answered however to my questions, what is the main point that you see as an advantage of an object storage compared to a block storage? If it's scale-out capabilities, I've already replied.
It's the object access and the possibility to split our backups into chunks to be then saved into objects?

Also, giving the fact that a NAS Gateway as a temporary workaround is indeed a workaround for software that do not talk natively the language of the object storage, would a block-device based object storage be a viable solution, as I explained? Have you considered it? What solutions are you looking at right now?

Saying it again, I'm a big fan of object storage. I have my own ideas on how we could use object storage, but I'm one and inside Veeam, as Anton said I'd prefer to listen about how do you plan to use it. But I'd like to get more info then "I like swift/S3 because it's the hype right now and everybody is talking about it". Gimme more reasons (I already have mines, trust me).
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
dougsoltesz
Novice
Posts: 6
Liked: 2 times
Joined: Mar 25, 2013 9:00 pm
Full Name: Douglas Soltesz
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by dougsoltesz » 1 person likes this post

Luca,

I’m sorry for the delayed reply to your response. Your totally correct in my desires with this post. I’m looking for a low cost, highly reliable, scale-out storage solution for my backups.

I also agree with your list of vendors that support such a solution today for scale-out block or file storage systems. However, the vendors on the list with the exception of Ceph aren’t as low cost as I’m looking for.

Ceph is very promising, and I’ve been testing it in the lab since Dumpling. However, the Ceph RDB is not to the maturity that I’m going to put it into production today. I have not upgraded to Giant, but a number of bugs in Firefly have kept it in testing status.

You are also correct that if I throw a 10TB file at object storage the performance is going to be less than stellar. The way that Veeam backs up will need to be slightly changed. Today Veeam uses CBT to pull a block of changed data (object?). That block of data could be from 256KB to 2MB depending on my storage optimization selected under Advanced settings. This block of data is hashed (if deduplication is enabled) and compressed. Then it is injected into a vbk or vib file. Somewhere in Veeam (and in the vbk file) a list of the hashes/blocks is kept so that the backup can be reassembled. If Veeam was to use object storage, I think most of the operations would be the same, except for the last part of injecting the pieces into a large VBK file. The pieces would be stored in an object store instead, and that would result in millions of objects for a 10TB VBK file which is more along the lines of how object storage should be used.

Again, I am very happy that Veeam team (you and Gostev) have given the time to respond to my requests. I am glad that @zfs also chimed in, and understand that maybe today is not the right time for Veeam to take on object storage.

I am happy to honor Gostev’s request and start other posts with enhancements to the current Veeam product. I caught him in the hall at VeeamOn to let him know that I thought the new forward-incrementals were the best new feature of v8. It would be nice to see a future enhancement where incrementals could be rolled up into a delta. But I will create a new forum post for that.

Lastly, to answer your question as to what I use today. Budd backs up our primary VM storage system (Nexenta) to another ZFS based system (OmniOS) which is a big cheap bunch of disks just like Gostev recommends. Then we use the use Veeam Backup Copy to take these backups to our SwiftStack Object Storage. SwiftStack today has a CIFS/NFS gateway with their product which makes it easy to hook into Veeam. Because we have object storage at multiple Budd locations (Regions) the backup copies are replicated to them as addition copies. This works very well for us today as I feel it meets the needs of the Veeam 3-2-1 strategy, runs on commodity hardware, and is cost efficient. While this works great, I do still look forward to possible future where the gateway is not required.
Thank you again for you attention to this request,

Douglas Soltesz
Budd Van Lines
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by dellock6 »

Thanks for your additional explanation, we are on the same page here as in regards to the "best way" to use an object storage by "mapping" blocks to objects. But as you already imagined, it requires a different approach. On the other way, a file (our vbk, vib, vrb) can be written in any supported file system, being it NTFS on windows repositories or any available fs on linux repositories. This means a single design in Veeam today can support a multitude of targets giving freedom of choice to customers, while object storage will require a dedicated solution. Indeed S3 is almost a "de facto" standard, but for example you already said you are using swift. I'm not saying it will never come, but you can understand timeline and commitment cannot be defined here.

As per Ceph, probably it's me running edge solutions, but I was quite happy when we used it in production with Dumpling and Emperor. I didn't had any possibility when I joined Veeam to load lots of I/O against newer builds, but I can assure you we have some service provider running their storage services on top of it with great results.

Thanks again,
Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
atakacs
Enthusiast
Posts: 45
Liked: 4 times
Joined: Nov 10, 2012 8:26 pm
Full Name: Alexandre Takacs
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by atakacs »

Hi

I know it is an old thread (and that the world has evolved since) but what is the status of OpenStrack Swift support ?

I have a (top) cloud provider that sells me 5To of storage for 25$/month with no ingress/egress fees. It would be a perfect secondary backup target... if I could make use of it :)
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by Gostev »

Hello, actually it is listed in the sticky topic on object storage compatibility. Thanks!
atakacs
Enthusiast
Posts: 45
Liked: 4 times
Joined: Nov 10, 2012 8:26 pm
Full Name: Alexandre Takacs
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by atakacs »

Thanks - I actually reviewed said post but did not see an answer.

Would it fit into "SwiftStack (6.15 or later) " ?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by veremin »

OpenStack Swift S3 (Rocky 2018 or later)
Thanks!
atakacs
Enthusiast
Posts: 45
Liked: 4 times
Joined: Nov 10, 2012 8:26 pm
Full Name: Alexandre Takacs
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by atakacs »

Ooops - wonder why FF search would not find it on the page...
rowin@atea.se
Lurker
Posts: 1
Liked: never
Joined: May 11, 2023 1:12 pm
Full Name: Robert Winbladh
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by rowin@atea.se »

I am also interested in fully working setup for openstack swift.
It is mentioned in the "supported", but I can verify that it does not fully work.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Object Backup Repository supporting OpenStack Swift & S3

Post by Andreas Neufert »

Hi Robert,

can you please share configuration details and what kind of issue you ran into? This thread is very old and there were no complaints for a longer time about this topic.
Post Reply

Who is online

Users browsing this forum: No registered users and 19 guests