V12: Main Benefits of using Local Object Storage vs. Linux Repo

[mkretzer]

Hello,

with V12 coming with the option to backup directly to object storage i am trying to understand what the main benefits for local repos (for example using a Dell ECS) there are compared to XFS repos.

From what i understand the benefits are:
- No more synthetic operations (but still synthetic GFS generation from my limited tests with V12)
- Generally object storage might scale better with data in the PB range (but our linux repo works extremly well with ~700 TB, it might also work well in the PB range)

No benefit compared to XFS:
- Immutability
- Flexibility (from my point of view, a well-designed LVM under the XFS is more flexible than a single object storage system as i can move the data online at any time to any system as long as its bigger WITHOUT stopping backups)

What i also wonder is how we could handle unbalanced SOBR when having multiple object storage systems in a SOBR - with XFS we can just copy the files from one storage to another (while loosing block clone in V11), with object its not an option, correct?

Markus

[chris.childerhose]

You have to wonder in v12 with the SOBR Rebalance if that will also apply when you use Object for your extent and if so that would help to solve the unbalance within any Object SOBR.

[chris.childerhose]

I know the VeeaMover will fix the XFS/ReFS issues keeping the block cloning so that is a game changer.

[jorgedlcruz]

My personal opinion would be more basic than functionality because, in the end, it will be very similar.

You can argue a bit that erasure coding is more efficient, and secure than regular RAID 6, which is what you will tend to see on a Linux XFS most likely. Despite v12 having Backup Copies to Object, or SOBR, some like MinIO will offer you bucket replica as well. < Not tested officially btw.

But as well, on big environments, you might find it easier to write full/maximum disk speed to block, rather than S3-API calls, where if you want more API calls, you need to scale the cluster and add more nodes.

But beyond those functionalities, and many others, or perhaps blocks, space savings, etc. There is the more basic question:

• What is your business expecting in terms of Data Protection?
  • Do they feel more comfortable with an appliance-based S3, as you said, Dell EMC ECS, Cloudian, MinIO, HPE Scality.
  • Or perhaps they feel better with a Linux appliance based on HPE Apollo for example if there is enough know-how internally to harden properly, etc.
• Who is going to maintain a DYI Appliance, and who owns the responsibility for it at the end of the day?
  • What I am saying here is, okay you Markus are working on a great organization. You recommend the Linux Hardened Repo path, then you build it using HPE, Dell, etc. Now, the rest of the team might not be very good in Linux, or if the Raid controller, or some PSU breaks, or some Linux vulnerability happens. Most likely Markus will own that, and they will ring you every time. Plus, if the person with Linux skills leaves the organization it can generate somehow some problems to keep and maintain the OS, the xfs partitioning, and overall health. Things are easy to fix with proper puppets or other orchestration tools.
  • On the other hand, leveraging an on-prem Object Storage solution, usually comes with vendor training, or it is very Web wizard-driven in most the cases, and if something happens at OS, Hardware, or anything, Markus opens a ticket with the vendor and they fix the whole stack of the Object Solution.

From what I have seen in the UK over these last four years as a Systems Engineer, seeing Customers of all sizes:

• Linux is great, and by far the cheapest solution, you can run your numbers comparing 1PB of data between the block and on-prem object. But, you need people with enough skills to do this properly. That, or outsource this deployment with a Veeam VASP for example. Deploying and configuring our Veeam Linux Hardware Repo is not difficult anyways, we even have a Community package to do this with a CLI interface, but still a bit out of reach for some.
• Windows with ReFS is still the goat (especially running on proper hardware like HPE Apollo, or HPE Nimble it just flies, but eSeries for example, is really versatile) for the initial tier and then Copy to Onprem S3, or Public S3. But no immutability of course for Performance Tier.
  It is still the most used Backup Repository by far, from what I have seen in the field, after Windows Repos, then I have seen a lot of dedupe appliances, mostly ExaGrid on new deployments, but a lot of DD and HPE StoreOnce as well. I do not have Big Data on this, just field experience.
• The Customers I have seen using Object Storage on-prem, it has been because they were using Object Storage as well for other applications. Like image processing, or other S3-based web applications.
  Saw Dell EMC ECS, Scality, and a lot of Cloudian, usually on Banking, and Investment Companies, but as you see, even the Tottenham Hotspur are running Veeam+HPE Scality nicely. This does not mean they will be great targets for v12, and as you saw by my previous points, I think it is the easiest "one button" Immutable Repository. You rack it, connect to the GUI, create buckets, and go. But, that simplicity, and great capabilities, can be clearly seen in pricing vs DYI Linux Appliances

Bottom line: If it was me working on an organization, and my Company had enough budget for Data Protection. I would totally go for Object Storage for the reasons mentioned (easy to set up, not a possible manual error, or manual maintenance of OS, easy to train new hiring as the knowledge can be reused from regular AWS S3 standards, etc. And as said, it could even be used for other purposes, perhaps just some buckets, etc.)
That said, depending on your data retention as well, imagine 5+ years' worth of monthlies sometimes, etc. Perhaps your best option would be a Dedupe Appliance, ExaGrid has Immutability, and HPE StoreOnce will bring Immutability managed from Veeam in v12 as well.

Moreover, I am expecting with v12 when it is out, that Customers and Partners will leverage our Solutions Engineers/Architects to help them with the best solution for each case based on bandwidth, data sovereignity, data retention, skillset in house, etc.

Not a small answer, but as this is a forum, took the liberty to put all my thoughts and experiences.

[mkretzer]

You have to wonder in v12 with the SOBR Rebalance if that will also apply when you use Object for your extent and if so that would help to solve the unbalance within any Object SOBR.

What SOBR Rebalance? If i understand correctly you can move backups between repo without loosing fast cloning but not between extends of a SOBR - which i think would be an extremly useful feature!

[Mildur]

@mkretzer
Rebalancing is a new feature in V12 announced at VeeamOn which will leverage moving backups between different performance tier extends without loosing FastClone savings.

1) Admin starts the process
2) all performance tier extends goes in maintenance mode
3) Veeam rebalance the backup files over all extends
4) performance tiers goes out of maintenance mode

Give it a try in the Beta to see how it works

[mkretzer]

@Mildur
I am currently using the beta, where do i find the "rebalance" feature for SOBR?

[Mildur]

CTRL+Right click on the SOBR Name. Then you have the rebalance option.
I tested it earlier today on my object storage performance extends, it's currently not supported for object storage.

[mkretzer]

Oh wow - i will CRTL+Right click on EVERYTHING in the Beta now
What happens if we cancel the balance in the middle of the balancing?

In a system with 200-400 TB extends "real" rebalancing might take a few weeks...

[Mildur]

As far as I know, the rebalancing process cannot be canceled.
CTRL+Right is not beta specific, it‘s available in the product since years

[Gostev]

Answering the initial question of this topic: those who are not afraid of DYI, are able to "well-design" LVM under the XFS and then secure the Linux server itself will always get a bigger bang for the buck by using a Hardened Repository. While for everyone with no Linux experience, object storage will likely be a better choice of these two options.

[mkretzer]

Understood, thank you all!
Its nice that the choice exists!

[gummett]

To add to Jorge's excellent post, and Gostev's clear summary, it's worth considering what performance you can expect from Object Storage with Veeam.

Each data block (1 MB by default, before compression) is written as a separate object, and each of these writes takes time to complete (especially if erasure coding is used). To work around this and provide great performance Veeam runs a huge number of parallel tasks, but this doesn't help if the target storage can't process them fast enough (or limits parallel connections).

Erasure coding multiplies back-end IO significantly, especially with relatively small object sizes. For example, with 12 data and 4 parity each incoming 1 MB Veeam object requires 16x 85 KB writes (all sizes before Veeam compression). And because of all the parallel write threads, turning this workload into sequential disk writes is so much harder. It's worth noting that because of this overhead many object storage systems simply replicate small objects rather than bothering to erasure code them. And some object storage doesn't use erasure coding at all. I had a good look at the forthcoming Object First appliance at VeeamON, and was intrigued to learn it uses RAID 6 in order to provide high throughput for Veeam workloads!

That's why I was so impressed I could drive over 12 GB/s (bytes not bits!) write throughput into Amazon S3 (from 1 job) when testing with the V12 beta... More details in my VeeamON session #shamelessplug

[mkretzer]

Yea, i don't know what it is but object with V12 is... Somthing else speed-wise.
To a single minio container on a 4 core VM i saw write rates in the range of 3-5 GB/s.

[chris.childerhose]

To add to Jorge's excellent post, and Gostev's clear summary, it's worth considering what performance you can expect from Object Storage with Veeam.

Each data block (1 MB by default, before compression) is written as a separate object, and each of these writes takes time to complete (especially if erasure coding is used). To work around this and provide great performance Veeam runs a huge number of parallel tasks, but this doesn't help if the target storage can't process them fast enough (or limits parallel connections).

Erasure coding multiplies back-end IO significantly, especially with relatively small object sizes. For example, with 12 data and 4 parity each incoming 1 MB Veeam object requires 16x 85 KB writes (all sizes before Veeam compression). And because of all the parallel write threads, turning this workload into sequential disk writes is so much harder. It's worth noting that because of this overhead many object storage systems simply replicate small objects rather than bothering to erasure code them. And some object storage doesn't use erasure coding at all. I had a good look at the forthcoming Object First appliance at VeeamON, and was intrigued to learn it uses RAID 6 in order to provide high throughput for Veeam workloads!

That's why I was so impressed I could drive over 12 GB/s (bytes not bits!) write throughput into Amazon S3 (from 1 job) when testing with the V12 beta... More details in my VeeamON session #shamelessplug

It is amazing the speed in v12. Your session was great at VeeamON too.

[mkaec]

One pain point for many (and a long-standing feature request) is that VBK files are never split and can cause problems when the VM is multi-TB. I'm expecting that won't be an issue any longer with object storage.

[mkretzer]

For us its basically the same issue from what i understand. At some point you will have multiple object store systems (older, newer generation) in a SOBR.
Since one backup can only be contained in one system as a whole the same balancing issues exist.

For example we currently have 5x 200 - 300 TB extends - our backups are getting so big that every 4-5 weeks our backups do "slow clones" all at the same time to the "most free" volume which pings back and forth between the two biggest extends. We have to manually stop some of the processes, delete some old points so that some backups stay on the full extend, because two weeks later that extend is quite empty again because everything moved. We already scheduled synthetics more distributed over the week, the issue still happens...

[mcz]

To a single minio container on a 4 core VM i saw write rates in the range of 3-5 GB/s.

Can you please tell me which container you were using? Was that minio? Thanks!

[mkretzer]

https://min.io/
Basically a pretty cool and simple open source object store

[JaySt]

[*]The Customers I have seen using Object Storage on-prem, it has been because they were using Object Storage as well for other applications. Like image processing, or other S3-based web applications.
Saw Dell EMC ECS, Scality, and a lot of Cloudian, usually on Banking, and Investment Companies, but as you see, even the Tottenham Hotspur are running Veeam+HPE Scality nicely. This does not mean they will be great targets for v12, and as you saw by my previous points, I think it is the easiest "one button" Immutable Repository. You rack it, connect to the GUI, create buckets, and go. But, that simplicity, and great capabilities, can be clearly seen in pricing vs DYI Linux Appliances[/list]

can you tell me something about the way you've seen those onprem object storage solutions being deployed in regards to amount of buckets and/or size of those buckets (e.g. some limits may exist for some vendors)?
I've seen some VeeamOn 2022 sessions stating 50TB limits as a general guideline for buckets. I think its rather. in a big environment 10+ buckets would not be rare in that case.

[HannesK]

I've seen some VeeamOn 2022 sessions stating 50TB limits as a general guideline for buckets.

yes, that was on one of my slides... it's a relatively "safe" value, because many object storage vendors struggle with many objects. I currently see hardly any vendor today (ObjectFirst would be a V12 thing), that can handle hundreds of TBs or PBs of Veeam backup data in one bucket. In general, you it's recommended to stay between 8-16 extents per SOBR (not 50 or 100 to scale in 50TB steps)

while backup might be fast, it usually becomes interesting, when retention is applied and tons of objects need to be deleted.

[JaySt]

thanks Hannes. Ok good to know. I guess we'll see alot more best practices getting communicated and explained and hopefully even some limits raised once v12 is out

[olafurh]

Can't Veeam autogenerate buckets perjob/chain to overcome the "bucket of death" issue?

[mcz]

Today I've started a thought experiment. I've asked myself if object storage would also simplify merges (necessary due to the forever forward incremental approach). My answer was that it does also help there and reduce the load/time. The traditional file-based-backups will have to write the oldest VIB-blocks into the VBK before it remove the VIB from the chain. On the object storage, you have every block in a single object and this object is decoupled from the other blocks - that means that you only have to delete the no longer needed blocks/objects (in the full and in the incremental) and just update metadata in the full to make sure that the blocks from the incremental are part of the full/point to the full.

Please correct me if I'm wrong, but if I'm not mistaken, then there are no traditional merges in combination with the object storage integration. Of course, same applies to the filesystems with block cloning APIs like ReFS and XFS, but there it's managed by the filesystem and here it's managed by veeam directly, so less dependencies. Thanks!

[Gostev]

There's no file system with object storage, so there are no backup files... in other words, there's nothing to merge to start with

This is unlike backup repositories with ReFS/XFS file systems because these do have backup files and so require merges, however those merges are much accelerated (comparing to NTFS/ext4 for example) through leveraging advanced file system functionality. But it is definitely not correct to say that object storage is "like" ReFS/XFS repository in any sense at all.

But otherwise, you are correct with your thinking.

[mkretzer]

For me, the merges are not as big of an issue as the rebalancing - and thats also what i still don't understand: One specific VM is always in one bucket, correct? Or can a VM be distributed over multiple buckets / S3 systems in one SOBR?

[Gostev]

Yes, one specific VM is always in one bucket.

[mkretzer]

@Gostev ok - So the same rebalancing problems exist here. Even worse with "50TB limits as a general guideline for buckets". Are there objects stores which work well with buckets > 600 TB? Our XFS repo works extremely well at those sizes (and i would have no problem scaling that up > 1 PB).

[Gostev]

Amazon S3, for example... as far as I remember, they only start having issues when reaching 1PB in a bucket, so 600TB should not be a problem.
For on-prem S3 storage, our support big data shows EMC ECS as having the biggest buckets (up to the size you need).

[JaySt]

@Gostev ok - So the same rebalancing problems exist here. Even worse with "50TB limits as a general guideline for buckets". Are there objects stores which work well with buckets > 600 TB? Our XFS repo works extremely well at those sizes (and i would have no problem scaling that up > 1 PB).

this is exactly how i was thinking about those limits and comparing to XFS experience. I guess we'll need to wait a bit for more field-numbers (and v12) and optimizations at vendors that will boost their confidence when stating their limits. Some onprem object storage solutions have no real software limit but have a "safe" limit, not really backed by anything hard (yet).