S3 offload

[andyboy16]

Hello there,
I'm looking for a ways to store only a few backups onsite and the rest to S3. I have SOBR setup to our S3 buckets but the retention is what's not working well. I have a backup job (reverse incremental with synthetic full running on Fridays) to retain only 2 days of backups (backup repo is set to the SOBR) and our SOBR is setup to offload as soon as they are created and also move any backup files older than 1 day old. After a week of backups running I'm only seeing 3 restore points (1 full backup, and 2 incremental backups). I also have archival set to keep 5 weekly full, and 24 monthly full. The purpose for this is to use least amount of space on-prem for backups. We have about 77TB of data to backup and as you can imagine, keeping that much on-prem is a tough sell to upper management to acquire more storage hence why we have full power to use as much S3 space as we want.

TLDR: keep a few backups onsite and move all other backups to S3 while using least amount of storage space on-prem for backups. We must also have the ability to restore anything at least 30 days and also archive 5 weekly full backups and 24 monthly backups.

I'm on V11 VB&R.

[HannesK]

Hello,
and welcome to the forums.

I assume that you you use active full backups? Because weekly synthetic full and GFS is only available with forward incremental, or if you have reverse incremental with active full backup.

After a week of backups running I'm only seeing 3 restore points (1 full backup, and 2 incremental backups).

that sounds correct. One full plus two VRBs always stay in performance tier

I'm not 100% sure what your goal is. Can you maybe explain a little bit more? The most recent backups are always on-prem to allow fast restore.

Thanks,
Hannes

[andyboy16]

I'm looking to store 2-3 days worth of backups on prem. And I want everything else to be stored on the S3. It's been over a week now and I'm only seeing the last 3 days worth of backups when I attempt a restore...though my retention policy is set to 2 days, but if I set that to 7 days, I see 7 days of backups sitting on-prem...which I don't want. I want only 2 days worth to sit on-prem and everything else to sit in S3.

[HannesK]

if you remove the active full it will only keep the latest two days.

If you use REFS / XFS as filesystem (recommended), then you could also use forward incremental with synthetic full and enable synthetic full every day. That's what customers do since many years to offload after a few days. The advantage of forward incremental is, that your backup finished faster and snapshots are open for a smaller amount of time.

[andyboy16]

Here's my settings, and I've changed my retention to 1day.



Even with the previous setting of 2 days retention, I only had 3 backups on-prem and nothing was restorable from S3 even though SOBR kicked in to offload.

[HannesK]

yep, for days where you have Mon, Wed, Fri, there I would just do every day.

and nothing was restorable from S3 even though SOBR kicked in to offload.

I cannot believe that without a support case number. With that backup settings, restores from S3 must work because the full backup(s) exists in S3. With reverse incremental it's a different story. In that case there might be no full backup in S3.

[andyboy16]

The problem with that is that my datastore is storing double full backups and it's filling up my storage. I don' t have enough room to store 3 Full's.

[HannesK]

then you are not using XFS / REFS as I recommended earlier

By using just "Reverse incremental" you will have one full and a two increments in performance tier. If you disable the "copy" in the SOBR settings, then you just have to be sure that you never lose your performance tier. Because the moved restore points in capacity tier are useless without the data from performance tier.

https://www.veeam.com/kb1799 has animations for the backup modes. Maybe that helps.

[andyboy16]

Isn't REFS just a file system type? What does that have to do with moving backups into S3 and only keeping 2 backups on-prem? .....unless I'm totally misreading what you are stating.

[Gostev]

This was in reference to:

my datastore is storing double full backups and it's filling up my storage. I don' t have enough room to store 3 Full's.

Veeam integrates with ReFS or XFS block cloning to ensure that on such backup repositories, synthetic full backups take no physical disk space. Thus, enabling them cost nothing from disk space usage perspective and you never have to worry about "room to store" those fulls.

[cpfleger]

@HannesK @Gostev :

wouldn't copy with immutability (let's say for the time (retention) period andyboy16 needs to have be a workaround? As S3 consumation seems not to be the cost driver?

Second: any way to better protect moved "meta" data in the the performance tier?
Either setting up a hyperscaler vm to run an extent or maybe a file cop job to get those data backed up?

Are you both aware of any improvements here planed for the near future as this seems to become a practical problem (recently had a harsh discussion with a big german partner regarding that).

Thx and regards!

[HannesK]

Hello Claus,
where do you like to set immutability? On Hardened Repository or in S3 object storage? In general: "abusing" immutability periods to "solve" issues is a bad idea.

Move after 2 days works fine with reverse incremental or daily synthetic full. So not sure what kind of workaround would be needed.

Second: any way to better protect moved "meta" data in the the performance tier?

what's the goal? If performance tier gets lost, importing object storage will re-create all necessary files from object storage.

any improvements here planed for the near future as this seems to become a practical problem

which problem?

Best regards,
Hannes

[cpfleger]

Because the moved restore points in capacity tier are useless without the data from performance tier.

So could you then elaborate a little bit more what you then mean here?

Otherwise: do I understand you right that even after a total loss of the performance tier you can always simply reimport all data from object storage? Even with move to capacity option?

Regards!

[HannesK]

that sentence was about "move" without "copy" with reverse incremental backup mode. If the VBK from performance tier is lost, then then data in capacity tier is useless (because it has only the increments without the full)

As long as the customer uses "move" and "copy" with reverse incremental, everything is fine. Also if a customer uses "move" without copy with daily synthetic full, everything is fine.

yes, Veeam backups are "self-contained". That means a new VBR server can import all valid data "from scratch" without any data from performance tier or configuration database.

[cpfleger]

Thx a lot Hannes for the clarification!

Regards!

[andyboy16]

So if I understand this correctly. If I have all of the following setup:

1) Backup incremental job with Synthetic full for Monday, Wednesday and Friday, (with archival set for 5/weekly and 24/monthly full) and
2) Have my retention set to 1 day, and
3) Have my SOBR set to copy AND move (after 1 days), and
4) Set Archive GFS full to 90 days and place in Glacier,

I should have only 1 full and 1 incremental on site and everything else will be moved into S3. Does that sound right? I guess my goal is to try to make room for the full's when the backup chain completes.

[HannesK]

more like two fulls and up to three increments depending on the day (weekend has two increments). To save disk space, recommend again using a file system with block cloning support (REFS / XFS)

[andyboy16]

more like two fulls and up to three increments depending on the day (weekend has two increments). To save disk space, recommend again using a file system with block cloning support (REFS / XFS)

Why two fulls when retention is 1 day? I thought each full completes the chain and then will offload into S3.

[HannesK]

because the new full has to exist before the old full + the increment (s) can be moved.

[andyboy16]

I'm still not seeing any restore points before 5 days that's in capacity tier. Looks like anything older is being deleted/removed even though I have immutable set for 30 days.

[andyboy16]

1) Backup incremental job with Synthetic full for Monday, Wednesday and Friday, (with archival set for 5/weekly and 24/monthly full) and
2) Have my retention set to 1 day, and
3) Have my SOBR set to copy AND move (after 1 days), and
4) Set Archive GFS full to 90 days and place in Glacier,

[Gostev]

But it's been just a few days? You need 1 week (7 days) to get a new weekly GFS created. Wait longer for more weekly and monthly synthetic full backups to appear, and these should remain as available restore points. There's really no reason for the retention policy to delete your GFS backups before each one gets 5 weeks and 24 months old respectively. Backup age is a simple check with very little wiggle room for any errors by the way, each backup deletion by the retention policy is reflected in job logs along with all the associated details, so invalid deletions are easy to see happening and troubleshoot.

Don't even worry about backup placement with SOBR for now, as it's a very perpendicular topic. As long as your GFS full backups remain under retention and are seen as available restore points, SOBR will tier them correctly.

[andyboy16]

It has actually been 2 weeks running this schedule. I would think I'd see at least weeks worth of restore points.

[Gostev]

Why would it be so, when your retention for recent backups is just 1 day? With this, you explicitly instructed Veeam to delete all restore points which are older than 1 day.

As it stands now, almost all of your restore points will be deleted from the configuration just a couple days later after their creation, except those selected as weekly and monthly GFS. In other words, with the current retention policy, for the next 3 weeks you should only see 1 restore point (weekly GFS) added to the list of available restore points once a week.

If you want them to remain and keep traveling your SOBR tiers, you should increase the retention policy to actually let them live longer. For example, if you want to have 2 weeks worth of recent restore points available (in addition to GFS), your retention policy should be 14 days.

[andyboy16]

That's the problem, we can't sustain keeping 14 days on-prem because of space issues hence we have to offload them into S3 as much as possible to save space.

[Gostev]

But why would 14 days of backups remain on-prem, when you have your SOBR configured to move them off to object storage as soon as possible?

[andyboy16]

I'm sorry, I misinterpreted that!

[andyboy16]

Hello again,
So this is working as expected now...thank you. But now, I'm facing an issue where a synthetic full job for one server (that's over 35TB big) is taking over 80hrs to complete which cuts into the next day's backup. Is there a better option such as using Active full backups instead of synthetic full?

[Gostev]

If you don't use ReFS or XFS backup repository, synthetic fulls will require backup storage with much IOPS (many spindles) to complete in a reasonable time. So for low end backup storage, active fulls are usually a better option (even if they put more stress on production environment).