Discussions related to using object storage as a backup target.
mkretzer
Veeam Legend
Posts: 1159
Liked: 394 times
Joined: Dec 17, 2015 7:17 am
Contact:

V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by mkretzer »

Hello,

with V12 coming with the option to backup directly to object storage i am trying to understand what the main benefits for local repos (for example using a Dell ECS) there are compared to XFS repos.

From what i understand the benefits are:
- No more synthetic operations (but still synthetic GFS generation from my limited tests with V12)
- Generally object storage might scale better with data in the PB range (but our linux repo works extremly well with ~700 TB, it might also work well in the PB range)

No benefit compared to XFS:
- Immutability
- Flexibility (from my point of view, a well-designed LVM under the XFS is more flexible than a single object storage system as i can move the data online at any time to any system as long as its bigger WITHOUT stopping backups)

What i also wonder is how we could handle unbalanced SOBR when having multiple object storage systems in a SOBR - with XFS we can just copy the files from one storage to another (while loosing block clone in V11), with object its not an option, correct?

Markus
chris.childerhose
Veeam Vanguard
Posts: 573
Liked: 132 times
Joined: Aug 13, 2014 6:03 pm
Full Name: Chris Childerhose
Location: Toronto, ON
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by chris.childerhose »

You have to wonder in v12 with the SOBR Rebalance if that will also apply when you use Object for your extent and if so that would help to solve the unbalance within any Object SOBR.
-----------------------
Chris Childerhose
Veeam Vanguard / Veeam Legend / Veeam Ceritified Architect / VMCE
vExpert / VCAP-DCA / VCP8 / MCITP
Personal blog: https://just-virtualization.tech
Twitter: @cchilderhose
chris.childerhose
Veeam Vanguard
Posts: 573
Liked: 132 times
Joined: Aug 13, 2014 6:03 pm
Full Name: Chris Childerhose
Location: Toronto, ON
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by chris.childerhose »

I know the VeeaMover will fix the XFS/ReFS issues keeping the block cloning so that is a game changer.
-----------------------
Chris Childerhose
Veeam Vanguard / Veeam Legend / Veeam Ceritified Architect / VMCE
vExpert / VCAP-DCA / VCP8 / MCITP
Personal blog: https://just-virtualization.tech
Twitter: @cchilderhose
jorgedlcruz
Veeam Software
Posts: 1403
Liked: 623 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by jorgedlcruz » 10 people like this post

My personal opinion would be more basic than functionality because, in the end, it will be very similar.

You can argue a bit that erasure coding is more efficient, and secure than regular RAID 6, which is what you will tend to see on a Linux XFS most likely. Despite v12 having Backup Copies to Object, or SOBR, some like MinIO will offer you bucket replica as well. < Not tested officially btw.

But as well, on big environments, you might find it easier to write full/maximum disk speed to block, rather than S3-API calls, where if you want more API calls, you need to scale the cluster and add more nodes.

But beyond those functionalities, and many others, or perhaps blocks, space savings, etc. There is the more basic question:
  • What is your business expecting in terms of Data Protection?
    • Do they feel more comfortable with an appliance-based S3, as you said, Dell EMC ECS, Cloudian, MinIO, HPE Scality.
    • Or perhaps they feel better with a Linux appliance based on HPE Apollo for example if there is enough know-how internally to harden properly, etc.
  • Who is going to maintain a DYI Appliance, and who owns the responsibility for it at the end of the day?
    • What I am saying here is, okay you Markus are working on a great organization. You recommend the Linux Hardened Repo path, then you build it using HPE, Dell, etc. Now, the rest of the team might not be very good in Linux, or if the Raid controller, or some PSU breaks, or some Linux vulnerability happens. Most likely Markus will own that, and they will ring you every time. Plus, if the person with Linux skills leaves the organization it can generate somehow some problems to keep and maintain the OS, the xfs partitioning, and overall health. Things are easy to fix with proper puppets or other orchestration tools.
    • On the other hand, leveraging an on-prem Object Storage solution, usually comes with vendor training, or it is very Web wizard-driven in most the cases, and if something happens at OS, Hardware, or anything, Markus opens a ticket with the vendor and they fix the whole stack of the Object Solution.
From what I have seen in the UK over these last four years as a Systems Engineer, seeing Customers of all sizes:
  • Linux is great, and by far the cheapest solution, you can run your numbers comparing 1PB of data between the block and on-prem object. But, you need people with enough skills to do this properly. That, or outsource this deployment with a Veeam VASP for example. Deploying and configuring our Veeam Linux Hardware Repo is not difficult anyways, we even have a Community package to do this with a CLI interface, but still a bit out of reach for some.
  • Windows with ReFS is still the goat (especially running on proper hardware like HPE Apollo, or HPE Nimble it just flies, but eSeries for example, is really versatile) for the initial tier and then Copy to Onprem S3, or Public S3. But no immutability of course for Performance Tier.
    It is still the most used Backup Repository by far, from what I have seen in the field, after Windows Repos, then I have seen a lot of dedupe appliances, mostly ExaGrid on new deployments, but a lot of DD and HPE StoreOnce as well. I do not have Big Data on this, just field experience.
  • The Customers I have seen using Object Storage on-prem, it has been because they were using Object Storage as well for other applications. Like image processing, or other S3-based web applications.
    Saw Dell EMC ECS, Scality, and a lot of Cloudian, usually on Banking, and Investment Companies, but as you see, even the Tottenham Hotspur are running Veeam+HPE Scality nicely. This does not mean they will be great targets for v12, and as you saw by my previous points, I think it is the easiest "one button" Immutable Repository. You rack it, connect to the GUI, create buckets, and go. But, that simplicity, and great capabilities, can be clearly seen in pricing vs DYI Linux Appliances
Bottom line: If it was me working on an organization, and my Company had enough budget for Data Protection. I would totally go for Object Storage for the reasons mentioned (easy to set up, not a possible manual error, or manual maintenance of OS, easy to train new hiring as the knowledge can be reused from regular AWS S3 standards, etc. And as said, it could even be used for other purposes, perhaps just some buckets, etc.)
That said, depending on your data retention as well, imagine 5+ years' worth of monthlies sometimes, etc. Perhaps your best option would be a Dedupe Appliance, ExaGrid has Immutability, and HPE StoreOnce will bring Immutability managed from Veeam in v12 as well.

Moreover, I am expecting with v12 when it is out, that Customers and Partners will leverage our Solutions Engineers/Architects to help them with the best solution for each case based on bandwidth, data sovereignity, data retention, skillset in house, etc.

Not a small answer, but as this is a forum, took the liberty to put all my thoughts and experiences.
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
mkretzer
Veeam Legend
Posts: 1159
Liked: 394 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by mkretzer »

chris.childerhose wrote: May 31, 2022 8:21 pm You have to wonder in v12 with the SOBR Rebalance if that will also apply when you use Object for your extent and if so that would help to solve the unbalance within any Object SOBR.
What SOBR Rebalance? If i understand correctly you can move backups between repo without loosing fast cloning but not between extends of a SOBR - which i think would be an extremly useful feature!
Mildur
Product Manager
Posts: 9009
Liked: 2380 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by Mildur » 1 person likes this post

@mkretzer
Rebalancing is a new feature in V12 announced at VeeamOn which will leverage moving backups between different performance tier extends without loosing FastClone savings.

1) Admin starts the process
2) all performance tier extends goes in maintenance mode
3) Veeam rebalance the backup files over all extends
4) performance tiers goes out of maintenance mode

Give it a try in the Beta to see how it works :)
Product Management Analyst @ Veeam Software
mkretzer
Veeam Legend
Posts: 1159
Liked: 394 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by mkretzer »

@Mildur
I am currently using the beta, where do i find the "rebalance" feature for SOBR?
Mildur
Product Manager
Posts: 9009
Liked: 2380 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by Mildur »

CTRL+Right click on the SOBR Name. Then you have the rebalance option.
I tested it earlier today on my object storage performance extends, it's currently not supported for object storage.
Product Management Analyst @ Veeam Software
mkretzer
Veeam Legend
Posts: 1159
Liked: 394 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by mkretzer »

Oh wow - i will CRTL+Right click on EVERYTHING in the Beta now :-)
What happens if we cancel the balance in the middle of the balancing?

In a system with 200-400 TB extends "real" rebalancing might take a few weeks...
Mildur
Product Manager
Posts: 9009
Liked: 2380 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by Mildur » 1 person likes this post

As far as I know, the rebalancing process cannot be canceled.
CTRL+Right is not beta specific, it‘s available in the product since years :)
Product Management Analyst @ Veeam Software
Gostev
Chief Product Officer
Posts: 31709
Liked: 6898 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by Gostev » 5 people like this post

Answering the initial question of this topic: those who are not afraid of DYI, are able to "well-design" LVM under the XFS and then secure the Linux server itself will always get a bigger bang for the buck by using a Hardened Repository. While for everyone with no Linux experience, object storage will likely be a better choice of these two options.
mkretzer
Veeam Legend
Posts: 1159
Liked: 394 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by mkretzer »

Understood, thank you all!
Its nice that the choice exists!
gummett
Veteran
Posts: 404
Liked: 106 times
Joined: Jan 30, 2017 9:23 am
Full Name: Ed Gummett
Location: Manchester, United Kingdom
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by gummett » 3 people like this post

To add to Jorge's excellent post, and Gostev's clear summary, it's worth considering what performance you can expect from Object Storage with Veeam.

Each data block (1 MB by default, before compression) is written as a separate object, and each of these writes takes time to complete (especially if erasure coding is used). To work around this and provide great performance Veeam runs a huge number of parallel tasks, but this doesn't help if the target storage can't process them fast enough (or limits parallel connections).

Erasure coding multiplies back-end IO significantly, especially with relatively small object sizes. For example, with 12 data and 4 parity each incoming 1 MB Veeam object requires 16x 85 KB writes (all sizes before Veeam compression). And because of all the parallel write threads, turning this workload into sequential disk writes is so much harder. It's worth noting that because of this overhead many object storage systems simply replicate small objects rather than bothering to erasure code them. And some object storage doesn't use erasure coding at all. I had a good look at the forthcoming Object First appliance at VeeamON, and was intrigued to learn it uses RAID 6 in order to provide high throughput for Veeam workloads!

That's why I was so impressed I could drive over 12 GB/s (bytes not bits!) write throughput into Amazon S3 (from 1 job) when testing with the V12 beta... More details in my VeeamON session #shamelessplug :P
Ed Gummett (VMCA)
Senior Specialist Solutions Architect, Storage Technologies, AWS
(Senior Systems Engineer, Veeam Software, 2018-2021)
mkretzer
Veeam Legend
Posts: 1159
Liked: 394 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by mkretzer » 1 person likes this post

Yea, i don't know what it is but object with V12 is... Somthing else speed-wise.
To a single minio container on a 4 core VM i saw write rates in the range of 3-5 GB/s.
chris.childerhose
Veeam Vanguard
Posts: 573
Liked: 132 times
Joined: Aug 13, 2014 6:03 pm
Full Name: Chris Childerhose
Location: Toronto, ON
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by chris.childerhose » 2 people like this post

gummett wrote: Jun 06, 2022 10:11 am To add to Jorge's excellent post, and Gostev's clear summary, it's worth considering what performance you can expect from Object Storage with Veeam.

Each data block (1 MB by default, before compression) is written as a separate object, and each of these writes takes time to complete (especially if erasure coding is used). To work around this and provide great performance Veeam runs a huge number of parallel tasks, but this doesn't help if the target storage can't process them fast enough (or limits parallel connections).

Erasure coding multiplies back-end IO significantly, especially with relatively small object sizes. For example, with 12 data and 4 parity each incoming 1 MB Veeam object requires 16x 85 KB writes (all sizes before Veeam compression). And because of all the parallel write threads, turning this workload into sequential disk writes is so much harder. It's worth noting that because of this overhead many object storage systems simply replicate small objects rather than bothering to erasure code them. And some object storage doesn't use erasure coding at all. I had a good look at the forthcoming Object First appliance at VeeamON, and was intrigued to learn it uses RAID 6 in order to provide high throughput for Veeam workloads!

That's why I was so impressed I could drive over 12 GB/s (bytes not bits!) write throughput into Amazon S3 (from 1 job) when testing with the V12 beta... More details in my VeeamON session #shamelessplug :P
It is amazing the speed in v12. Your session was great at VeeamON too. 8)
-----------------------
Chris Childerhose
Veeam Vanguard / Veeam Legend / Veeam Ceritified Architect / VMCE
vExpert / VCAP-DCA / VCP8 / MCITP
Personal blog: https://just-virtualization.tech
Twitter: @cchilderhose
mkaec
Veteran
Posts: 465
Liked: 136 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by mkaec »

One pain point for many (and a long-standing feature request) is that VBK files are never split and can cause problems when the VM is multi-TB. I'm expecting that won't be an issue any longer with object storage.
mkretzer
Veeam Legend
Posts: 1159
Liked: 394 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by mkretzer »

For us its basically the same issue from what i understand. At some point you will have multiple object store systems (older, newer generation) in a SOBR.
Since one backup can only be contained in one system as a whole the same balancing issues exist.

For example we currently have 5x 200 - 300 TB extends - our backups are getting so big that every 4-5 weeks our backups do "slow clones" all at the same time to the "most free" volume which pings back and forth between the two biggest extends. We have to manually stop some of the processes, delete some old points so that some backups stay on the full extend, because two weeks later that extend is quite empty again because everything moved. We already scheduled synthetics more distributed over the week, the issue still happens...
mcz
Veeam Legend
Posts: 882
Liked: 191 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by mcz »

mkretzer wrote: Jun 06, 2022 10:52 am To a single minio container on a 4 core VM i saw write rates in the range of 3-5 GB/s.
Can you please tell me which container you were using? Was that minio? Thanks!
mkretzer
Veeam Legend
Posts: 1159
Liked: 394 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by mkretzer »

https://min.io/
Basically a pretty cool and simple open source object store
JaySt
Service Provider
Posts: 418
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by JaySt »

jorgedlcruz wrote: May 31, 2022 9:37 pm
[*]The Customers I have seen using Object Storage on-prem, it has been because they were using Object Storage as well for other applications. Like image processing, or other S3-based web applications.
Saw Dell EMC ECS, Scality, and a lot of Cloudian, usually on Banking, and Investment Companies, but as you see, even the Tottenham Hotspur are running Veeam+HPE Scality nicely. This does not mean they will be great targets for v12, and as you saw by my previous points, I think it is the easiest "one button" Immutable Repository. You rack it, connect to the GUI, create buckets, and go. But, that simplicity, and great capabilities, can be clearly seen in pricing vs DYI Linux Appliances[/list]

can you tell me something about the way you've seen those onprem object storage solutions being deployed in regards to amount of buckets and/or size of those buckets (e.g. some limits may exist for some vendors)?
I've seen some VeeamOn 2022 sessions stating 50TB limits as a general guideline for buckets. I think its rather. in a big environment 10+ buckets would not be rare in that case.
Veeam Certified Engineer
HannesK
Product Manager
Posts: 14460
Liked: 2924 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by HannesK » 1 person likes this post

I've seen some VeeamOn 2022 sessions stating 50TB limits as a general guideline for buckets.
yes, that was on one of my slides... it's a relatively "safe" value, because many object storage vendors struggle with many objects. I currently see hardly any vendor today (ObjectFirst would be a V12 thing), that can handle hundreds of TBs or PBs of Veeam backup data in one bucket. In general, you it's recommended to stay between 8-16 extents per SOBR (not 50 or 100 to scale in 50TB steps)

while backup might be fast, it usually becomes interesting, when retention is applied and tons of objects need to be deleted.
JaySt
Service Provider
Posts: 418
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by JaySt »

thanks Hannes. Ok good to know. I guess we'll see alot more best practices getting communicated and explained and hopefully even some limits raised once v12 is out ;)
Veeam Certified Engineer
olafurh
Service Provider
Posts: 25
Liked: 16 times
Joined: Oct 29, 2014 9:41 am
Full Name: Olafur Helgi Haraldsson
Location: Iceland
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by olafurh »

Can't Veeam autogenerate buckets perjob/chain to overcome the "bucket of death" issue?
mcz
Veeam Legend
Posts: 882
Liked: 191 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by mcz »

Today I've started a thought experiment. I've asked myself if object storage would also simplify merges (necessary due to the forever forward incremental approach). My answer was that it does also help there and reduce the load/time. The traditional file-based-backups will have to write the oldest VIB-blocks into the VBK before it remove the VIB from the chain. On the object storage, you have every block in a single object and this object is decoupled from the other blocks - that means that you only have to delete the no longer needed blocks/objects (in the full and in the incremental) and just update metadata in the full to make sure that the blocks from the incremental are part of the full/point to the full.

Please correct me if I'm wrong, but if I'm not mistaken, then there are no traditional merges in combination with the object storage integration. Of course, same applies to the filesystems with block cloning APIs like ReFS and XFS, but there it's managed by the filesystem and here it's managed by veeam directly, so less dependencies. Thanks!
Gostev
Chief Product Officer
Posts: 31709
Liked: 6898 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by Gostev »

There's no file system with object storage, so there are no backup files... in other words, there's nothing to merge to start with :D

This is unlike backup repositories with ReFS/XFS file systems because these do have backup files and so require merges, however those merges are much accelerated (comparing to NTFS/ext4 for example) through leveraging advanced file system functionality. But it is definitely not correct to say that object storage is "like" ReFS/XFS repository in any sense at all.

But otherwise, you are correct with your thinking.
mkretzer
Veeam Legend
Posts: 1159
Liked: 394 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by mkretzer »

For me, the merges are not as big of an issue as the rebalancing - and thats also what i still don't understand: One specific VM is always in one bucket, correct? Or can a VM be distributed over multiple buckets / S3 systems in one SOBR?
Gostev
Chief Product Officer
Posts: 31709
Liked: 6898 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by Gostev »

Yes, one specific VM is always in one bucket.
mkretzer
Veeam Legend
Posts: 1159
Liked: 394 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by mkretzer »

@Gostev ok - So the same rebalancing problems exist here. Even worse with "50TB limits as a general guideline for buckets". Are there objects stores which work well with buckets > 600 TB? Our XFS repo works extremely well at those sizes (and i would have no problem scaling that up > 1 PB).
Gostev
Chief Product Officer
Posts: 31709
Liked: 6898 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by Gostev »

Amazon S3, for example... as far as I remember, they only start having issues when reaching 1PB in a bucket, so 600TB should not be a problem.
For on-prem S3 storage, our support big data shows EMC ECS as having the biggest buckets (up to the size you need).
JaySt
Service Provider
Posts: 418
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: V12: Main Benefits of using Local Object Storage vs. Linux Repo

Post by JaySt » 1 person likes this post

mkretzer wrote: Jul 08, 2022 6:40 am @Gostev ok - So the same rebalancing problems exist here. Even worse with "50TB limits as a general guideline for buckets". Are there objects stores which work well with buckets > 600 TB? Our XFS repo works extremely well at those sizes (and i would have no problem scaling that up > 1 PB).
this is exactly how i was thinking about those limits and comparing to XFS experience. I guess we'll need to wait a bit for more field-numbers (and v12) and optimizations at vendors that will boost their confidence when stating their limits. Some onprem object storage solutions have no real software limit but have a "safe" limit, not really backed by anything hard (yet).
Veeam Certified Engineer
Post Reply

Who is online

Users browsing this forum: masahide.k and 10 guests