Rotation scheme for backups to Glacier

[Kahuna]

Hello, I searched around for an answer to this, but I'm still a bit fuzzy on the options. I'm trying to put together a backup scheme that will let me get around 3TB worth of backups out to Amazon Glacier via a 30Mbps connection. Our hope is to get weekly, biweekly (every two weeks) or worst case monthly. Our daily's are reverse incremental with 45 restore points.

Since the backups will take so long I can't use the daily's as they sit for the upload. So for testing I just did separate full backups to get a copy off site. It looks like it will take a week or more to upload everything from a full backup.

So what I'm trying to figure out is if there is a reasonable way to do incremental backups to Glacier so I don't always have to be uploading full backups? And regardless of that answer, what's the recommended B&R backup scheme to that will make the best use of my bandwidth? I'm not so much worried about the cost of Glacier backups as I am about eating up all our bandwidth all the time.

Thanks for any advice.

[foggy]

Will, you can go with forward incrementals and synthetic fulls. This way you will copy only incremental changes to the cloud and have full backup “synthesized” from data that is already there (from the previous full backup file and a chain of increments) instead of retrieving data from the source from scratch.

[dellock6]

Uhm, Alexander are you sure you can run synthetic fulls towards Glacier? It's an object storageso it has only get and put commands, and since GET is limited into Glacier, I'm not sure synthetic full are a viable way... Perhaps a workaround is to use a virtual machine inside AWS, install Veeam Repository on it, and run there the synthetic full creation. Then from here it's easy to move backup files to Glacier since they are already inside Amazon network.

Luca.

[foggy]

Fair note, thanks, Luca.

[Kahuna]

OK, so it does sound like with Glacier, there is no practical way to do incrementals and basically I'm doing full backups for everything. Is that about the size of it?

[veremin]

The situation is a little bit different.

What Luca has mentioned that there is no way to perform synthetic full on Amazon Glacier side and it’s an active full backup that should be performed from time to time.

In other words, you can set forward incremental mode with an active full option being enabled and take everything to Glacier.

However, please don’t consider Amazon Glacier as a normal cloud, it’s nothing but a long-term storage for data which unclaimed for long period of time. If you’re after high-availability of information, kindly see for other provided solutions.

So far many of users appear to misunderstand the basic Glacier concept, and it, in its turn, results in unsatisfaction.

Hope this helps.
Thanks.

[dellock6]

Right, especially the retrieve policy is almost alwyas ignored and could lead to problems if RTO values need to be low. Glacier has a requests form for getting back data, and it can vary from 2 to 6 hours. None of the backups saved to Glacier is immidiately available for download. Is completely designed for long term retention without a real need to restore data so often. I'm pretty sure the super-lower price is so interesting people does not read anything else about it.

If you're looking for fast restores, look at Amazon S3 or some other provider. Basically any Cloud Provider offering Openstack Object Storage (called Swift) is a good fit for Veeam Cloud Edition.

Luca.

[Kahuna]

Right, I do understand all that. We are using Glacier purely as off site storage instead of sending media away to Iron Mountain. We may add some S3 storage for faster RTO on short term off site once we get the backup to Glacier nailed down.

So we are looking to have data stored off site at Glacier for a year or more as pure disaster recovery or backup of last resort.

If we go with forward incrementals, to do a restore I need to find the last full backup and then all the incrementals that came after it, correct? So with cloud edition and Glacier, that would be a restore job for the full and one each for the incrementals, is that correct?

[foggy]

If we go with forward incrementals, to do a restore I need to find the last full backup and then all the incrementals that came after it, correct?

Correct.

So with cloud edition and Glacier, that would be a restore job for the full and one each for the incrementals, is that correct?

You can select all the files that you want to restore at once when creating a restore plan in Veeam B&R Cloud Edition (no need to create separate plans for each incremental).

[Kahuna]

Got it, thanks.

Just to be clear also, there is no way in Cloud Edition to copy backups from S3 to Glacier or to do complex rotations beyond deleting after so long, correct?

[foggy]

I guess only if you install it on a VM inside AWS. What Veeam B&R Cloud Edition basically does, is simply uploading the local files to the specified cloud storage, restoring them back from there, and restoring VM guest OS files from Veeam backups stored in the cloud.

[dellock6]

Will, since Cloud Edition basically is a file mover, until you keep it in your company, any move from S3 to Glacier would involve you bandwidth, since you will have to download it from S3 and upload it again to Glacier. Really think about a Veeam repository running inside an AWS machine.

For example use reverse incremental, store all your recent backups into this AWS (thus not even involving S3), and maybe once per week copy the full vbk file into Glacier.

Luca.

[Kahuna]

OK thanks. That's what I thought, but I know there is a way to transfer data from S3 to Glacier so thought I should check.

Thanks again for all the help.

[andharri]

Hi,

We are running B&R 7.0.0.764 in our DC backing up a number of VMs to general disks stored offsite over high speed links.

This all works fine and has banished memories of waiting hours and hours to load tapes in to recover a single file!

I have decided that for extra resiliency I am going to store a copy of all full and snapshot backups in AWS Glacier just encase we have problems with the general disk. Before I do that I have been exploring the most efficient way of doing this without sending huge amount of data over to AWS each day.

Reverse Incremental - Going with this option means that essentially we are uploading full copies of every VM to AWS each morning, which for me is basically going to lead to it uploading 24/7!

Incremental with synthetic full every Saturday - This options is slightly more efficient as it means we have to do big uploads every Saturday not every day.

Incremental with fulls once a year - This seems like a favourable option, however I am assuming that as we reach the maximum restore point period for the backup the "full" will be moved to the oldest restore point so we will essentially be uploading a lot of data everyday anyway.

I would be interested in people's thoughts on this?

[veremin]

Hi, Andrew,

With the reversed incremental mode the new full backup will be copied each time. Also, it's not possible to run synthetic full backup in Glacier storage. So, the only option you have is to use forward incremental mode with full backup.

This seems like a favourable option, however I am assuming that as we reach the maximum restore point period for the backup the "full" will be moved to the oldest restore point so we will essentially be uploading a lot of data everyday anyway.

Not sure whether I follow you on that. Full backup doesn't move forward whatsoever - it stays in location, until it's removed by the retention policy.

Thanks.

[foggy]

Incremental with fulls once a year - This seems like a favourable option, however I am assuming that as we reach the maximum restore point period for the backup the "full" will be moved to the oldest restore point so we will essentially be uploading a lot of data everyday anyway.

You're probably confusing the retention algorithm with the one for the backup copy job here.