Support for backing up Object Storage

[lgravatt]

I know that Veeam supports backing up to AWS S3 Buckets or a S3 compatible as a target.
Is it in the near future road map for Veeam to support backing up Object Storage data that resides on a on-prem server such as StorageGRID or in the cloud?

[Gostev]

Near future is v11, and this feature is not there

I'm curious how you see back up and restore of such data working though, care to share your thoughts? This will be very helpful for us.

1. Incremental backup. Since there's no "changed object tracking" API available in object storage today, an incremental run would require reading all objects every time to determine the changed ones. Which means each incremental run will take days depending on your data size, while also creating a massive continuous load on the object storage. Do you see these issues acceptable (RPO of a few days, constant load on production storage from backup activities).

2. Consistency. Especially in light of 1, without some sort of a snapshot available, you will be backing up a completely inconsistent state spanning some hours or days. In other words, the first object you will backup up over the course of a given run may change a few times after it was backed up before you get to backing up the last object during this incremental pass. Is this acceptable?

3. Restore. What are the restore scenarios here? Let's imagine you sit on the backup of a few hundred millions of objects. Applications will probably name them with GUIDs to ensure uniqueness, making it impossible to say which object is which. Do you envision getting the specific missing object name from the application, and restoring one from backup? Or are you thinking restores from an entire object storage loss?

Thanks!

[edirschedl]

Hi together,

we use our S3 storage also for storing persistent data from our Kubernetes/Container infrastructure. My problem is now - how to backup this data? I know that VEEAM has no implementation for this scenario (e.g. from S3 to S3 backup or similar) currently. Is there any alternative idea to do real backup beside any snapshot or replication approaches? Is VEEAM anything planning in this direction?

Thank you,
Emanuel

[Gostev]

Hi, Emanuel. Could you please also answer the questions above to better understand your use case? Thanks!

[edirschedl]

1. Incremental backup. Since there's no "changed object tracking" API available in object storage today, an incremental run would require reading all objects every time to determine the changed ones. Which means each incremental run will take days depending on your data size, while also creating a massive continuous load on the object storage. Do you see these issues acceptable (RPO of a few days, constant load on production storage from backup activities).

2. Consistency. Especially in light of 1, without some sort of a snapshot available, you will be backing up a completely inconsistent state spanning some hours or days. In other words, the first object you will backup up over the course of a given run may change a few times after it was backed up before you get to backing up the last object during this incremental pass. Is this acceptable?

3. Restore. What are the restore scenarios here? Let's imagine you sit on the backup of a few hundred millions of objects. Applications will probably name them with GUIDs to ensure uniqueness, making it impossible to say which object is which. Do you envision getting the specific missing object name from the application, and restoring one from backup? Or are you thinking restores from an entire object storage loss?

Hi Gostev,

1. Not optimal, but together with additional snapshot / replication technologies I can live with higher RPO's. Regarding load it would be good then to reduce parallelism/IO - similar to the existing NAS backup feature. Could you check also the the command "mc.exe mirror --newer-than" from Minio (https://github.com/minio/mc), if they have any optimization in this mechanism?

2. Understand the issue. But it's better to have some corruptions than have completely nothing.

3. In our case the S3 instances with important data aren't so big. So in the first a step a bucket based restore or something similar is OK, which I can provide on a second instance (cross restore would be then important!), where the responsible team can looking for the right stuff. I don't see also no problem to delegate this topic to my application responsible to go through the data and say me which data is needed. So I think that's a solvable topic.


In general my problem is, that I've now productive data from my business which I've to backup. So I'm happy also with a basic integrations with restrictions in the first step!

[Gostev]

1. Yes, throttling is easy of course... it will just take long backups even longer.

3. Publishing backup as a new bucket, similar to instant NAS backup publish in v11 might be a good idea for restores indeed. Can you give an idea of the total size and the total number of objects you need to protect?

Yes, I understand your need perfectly, and that at this point you'd be happy with any solution at all. Thanks a lot for taking the time to provide a detailed reply, it does help a lot.

[molan]

It is becoming more and more common to use S3 based storage as the primary storage location. However currently there is no solution for backing up data using S3 storage as the source. We all know putting our data in the cloud does not remove the need for a good backup solution in case of data corruption, user error, service disruptions, etc.

I would like to submit this request as a feature request for Veeam Backup & Replication to implement backup jobs with S3 storage as the source data being backed up.

In my case I would specifically like to be able to backup Wasabi S3 Storage. I would like the ability to backup the data to local storage as well as other S3 and cloud based locations.

Thanks
Matt

[HannesK]

Hello,
and welcome to the forums.

As for today, there is only the limited scale approach mounting S3 to a server and backing up with NAS backup.

Additionally to the questions from above post399270.html#p399270, I would be interested whether
1) metadata of an S3 object is important for you?

Code: Select all

    [...]
    "Metadata": {
        "keyname1": "name1",
        "keyname2": "name2"
    }

2) whether your application (I assume that your S3 data is application generated data) has a database that could offer "change object tracking"?

Best regards,
Hannes

[molan]

In my case I don't believe Meta Data would matter. The application generates keys on the fly and they expire quickly. Previous version data does mater however as we use this feature to track file versions and people upload changes.

Yes the data is written to S3 storage by an application it isn't directly access by people. I am looking a 500GB to 1TB of File data in my current example. Its Mainly PDF files containing various information.

To try and answer the first three questions as best I can

1. Incremental backup. Having an RPO of a few days is better than having nothing. Most of the Data in my above example is static. Once written it doesn't change often. If it changes it only changes in the first few days to weeks after being written and then it is static data that doesn't normally change past being a few months old. It would be great if the S3 interface would develop to the point of being Archive aware on files so that it could be more backup friendly but I understand that isn't under your control. I would be happy with any backup at this point.

2. Consistency. Especially in light of 1, again. Any backup is better than no backup. and my data doesn't change frequently. Couldn't the backup software keep an index of what it has backed up and only look for new data? Then once the new data is backed up go and do a comparison on the old data. the comparison could happen at a slower rate than the backup of the new data. ?

3. Restore. What are the restore scenarios here? Well in my case the application doesn't name them with a Guid, can't speak for other apps. Restoring to a new bucket would be a good option that way the existing data isn't affected and is retained just in case. It isn't hard to repoint an app to a new bucket. I only envision doing a restore in the following two scenarios
A) Full restore - all data is restored to a new bucket.
B) Individual restore - restore of individual files, those files would be restored to an alternate location and re-uploaded manually or via the app if they need to go into the live bucket.

[HannesK]

thank you for the answers, that helps.

For 500GB - 1TB, I would really give NAS backup a chance. Just mount the S3 bucket to a machine and back up that path with NAS backup.

[mark49808]

I see various posts about using Ceph as a Repo, but curious if anyone has used Veeam to actually back up data stored on Ceph? I know the product itself has replication, redundancy, etc, but so does NetApp, and Veeam works with that and there are use cases to back up certain datasets.

Best I can think is to set up some linux server mounting a ceph volume and backing that up, but thats likely not ideal.

Curious if i'm in uncharted territory? Or if anyone has done this / has any tuning recommendations?

[veremin]

This feature does not exist as of now, but if you want to see it in the future product versions, you could start with answering three magic questions Anton's posted above. Thanks!

[pybfr]

Hello,

I just got two requests those last few days from two customers

Concerning the questions
1) this is a document archive, meaning that existing documents will not change and most of the work will be to add new documents, so incremental would probably be an important feature, as long as load doesn't prevent the application to do its job.
2) In this context, objects being unrelated to each other, it is probably OK.
3) Recovery of deleted objects and protection from malware. So it could probably span from single object restore to full bucket restore.

thx for your time.

[Gostev]

Thanks for answering questions. We're have already created a working prototype based on the same assumptions, and will consider including this functionality in future releases.

[TheBrain82]

Hello,

I am missing the possibility to backup any Kind of S3 Storage by Veeam. The option is even missing inside of this Forum! Using Object Storage as a backup source would be the correct requirement. Will there be a possibility to protect data residing in S3 storage with Veeam?
Thank you

[Mildur]

Hi Oleg

Thanks for the request. Currently it is not possible to backup S3 Buckets with Veeam.
I moved your post to this topic where it is already discussed.
If you can provide some answers to Antons questions above, it would help.

Thanks
Fabian

[mcz]

Hi everyone,

while one of the most asked questions of the past (probably) was "can I do backup to object storage", my question today is: can we expect a backup of an object storage itself in the near future?
I think writing backups to object storage(s) by using copy jobs, etc. is one thing, but what if there's no veeam data on that object storage?

One of our software partners uses a very expensive block storage for documents and now they've asked me for an idea to reduce costs - I immediately mentioned the object storage with it's S3 API. But if you're not able to backup such a bucket, this approach might fail even 99.99 % of all the criterias would match...

Curious about all the answers. Thanks!

[Gostev]

Yes, you can expect that Michael. Our NAS backup engine was built with support for other unstructured data source in mind - not just file shares. Meanwhile, it will help if you could address the few questions asked in the first response to this topic. In particular, question 2 would be interesting to run by the actual software partner you mentioned. Thanks!

[mcz]

Thanks Anton! Here are my answers:

1) it would be an archive, so longer RPOs are ok. I wonder if there's a way to only read the checksum of an object from a bucket as this would help to build a fast changed object tracking
2) this is for sure a problem if the data changes during the backup run - you would have to modulate the writing to the object storage and the backup. I wonder if maybe versioning/immutability is the way to pull the data from an identical point in time, having kind of a invisible/synthetic snapshot. what do providers like wasabi say to such a request?
3) I think it's important to protect against a fatal data loss of the whole bucket - a kind of instant bucket recovery in the style of the existing IR would be cool - in the application you'd change the API endpoint and the access keys and you would at least have a read access to the whole bucket. Restoring a single object is also important, in case of data corruption or if something has accidentially deleted/modified an object.

[dasfliege]

We have a customer who is running an application on AWS. The application is maintained by another company. They perform daily backups of all the relevant application data to a separate AWS S3 bucket. Our customer now wants to have a copy of this bucket in his on-prem infrastructure. They do have an Object Storage on-prem as well.

Is there a possibility with veeam to just backup or copy an AWS bucket to an on-prem Object storage? And if yes, is the Veeam for AWS appliance needed for that as well?

[Mildur]

Hello

I moved your post to the existing topic about this feature request.
Could you please answer the three questions from Anton?
post399270.html#p399270

Best,
Fabian

[dasfliege]

Near future is v11, and this feature is not there

I'm curious how you see back up and restore of such data working though, care to share your thoughts? This will be very helpful for us.

1. Incremental backup. Since there's no "changed object tracking" API available in object storage today, an incremental run would require reading all objects every time to determine the changed ones. Which means each incremental run will take days depending on your data size, while also creating a massive continuous load on the object storage. Do you see these issues acceptable (RPO of a few days, constant load on production storage from backup activities).

2. Consistency. Especially in light of 1, without some sort of a snapshot available, you will be backing up a completely inconsistent state spanning some hours or days. In other words, the first object you will backup up over the course of a given run may change a few times after it was backed up before you get to backing up the last object during this incremental pass. Is this acceptable?

3. Restore. What are the restore scenarios here? Let's imagine you sit on the backup of a few hundred millions of objects. Applications will probably name them with GUIDs to ensure uniqueness, making it impossible to say which object is which. Do you envision getting the specific missing object name from the application, and restoring one from backup? Or are you thinking restores from an entire object storage loss?

Thanks!

1. In my understanding the "aws s3 sync" cli command is able to copy only changed files. But i actually don't know if this is only working "to" S3 or also "from" s3.
2. That's an issue. But in our usecase, we only would like to copy files which are stored on the bucket once a day. So we actually know the time when the data is consistent and ready to be copied.
3. See point 1. I guess we would just use the "aws s3 sync" on a reverse way.

It seems there are ways to copy data from an S3. In my wonderful little world, i was hoping that veeam is able to just make the process easier as you always do But i must admit, that i am not very deep into the S3 protocol, so i can't judge wheter or not somthing is making sense at all.

[Gostev]

@dasfliege we're building this feature as we speak and plan have a technology preview build around end of April that will cover main S3 backup and restore scenarios. We will really appreciate it if you join the beta and give it a try. Thanks!

[dasfliege]

Sure. How can i join the beta?

[mcz]

oh wow, really cool that this is already in work. we will need a backup solution quite soon as well...

[Mildur]

@dasfliege
@mcz

You will get private message from me.

Best,
Fabian

[gummett]

Happy to test with Amazon S3

[Mildur]

Hi Ed

I will send you a private message.

Best,
Fabian

[Gostev]

@dasfliege we're building this feature as we speak and plan have a technology preview build around end of April that will cover main S3 backup and restore scenarios. We will really appreciate it if you join the beta and give it a try. Thanks!

This is now available. Fabian will personally take care of everyone who asked for it so far, if anyone else is interested please contact your Veeam sales rep as per our usual private beta process.

[mcz]

Thanks Anton!