New to tape backups and confused how it works.

[morty]

I'm trying to implement AWS Storage Gateway VTL.

I have it all setup and seemingly working but I don't really understand what the strategy I should use.

My regular backups have a "Daily" backup to disk job that does a backup once daily of 20 VMware Virtual Machines using the forever-forward incremental method using per-vm files and has 30 days of restore points. The size of this data for 7 days of restore points (1 full and 6 increments) is about 4.5TB.

I'm required to have at least 7 business days offsite (7 restore points). I'm not archiving past 7 days if I don't have to so I haven't been exporting tapes.

Right now I have 10 tapes 1TB in size each all in one media pool. So I have a media pool of 10TB (I don't want to use more than 10TB because I have an AWS budget of $250/month). I really don't understand media sets either but I've created the media pool to never create a new media set and to always use the same one. Is this right? Also I set it to protect data for 7 days. I can't seem to set how many restore points to keep for the tape job so does the tape job just forever keep copying data until it fills the entire media pool and then start overwriting old data assuming it was written past the protection time (7 days)?

What tape strategy should I be using (number of tapes, size of tapes, media pools, media sets, protection, etc)?

Thank you in advance.

[Dima P.]

Hello Ryan,

I really don't understand media sets either but I've created the media pool to never create a new media set and to always use the same one. Is this right?

Yes. This option allows you to either append or not append to previously written tape media (in case it's still in the library). Keep it always continue if you export your emulated media to amazon on a weekly basis.

Also I set it to protect data for 7 days. I can't seem to set how many restore points to keep for the tape job so does the tape job just forever keep copying data until it fills the entire media pool and then start overwriting old data assuming it was written past the protection time (7 days)?

This is a retention for your tape media and not the backup file. Tape job pulls the entire backup chain from your disk storage and keeps it for the set amount of time. If you are exporting tape media to amazon, I'd rather use 'do not overwrite' option. After tapes are exported to amazon most likely you wont export them back for tape retention to apply (better to kill the tape from amazon to avoid the download costs and then remove it from Veeam B&R console)

[morty]

So if I never want to export tapes, I just want to keep using the same tapes and media pool over and over to only have at least 7 days of restores, how would that look? Is that even possible? I feel like maybe tape is the wrong choice now if it's only for archiving. I just simply want 7 restore points off site.

[morty]

What I'm trying to achieve here is essentially just having a 10TB volume (media pool) with the latest 7 restore points from the backup. I looked into AWS Storage Gateway Volumes (both cached and stored) but they seem risky to use because if Veeam reads data from the volume (for example to merge an incremental into a full) then I might get charged for requests which are very expensive. Is tape capable of achieving what I'm looking to do while taking advantage of forever forward incrementals and virtual full backups so that I'm not copying my full data-set more than once (the initial run)? We have a 100mbit upload connection so xfering an entire full of ~4.5TB takes several days.

[L33T]

If your full backup is 4.5TB and your media is 1tb you`d need 35 tapes to run a 7 day rotation, depending if they get compressed it could be a bit less. We use a 28 day rotation with a monthly tape that gets pulled out of the media pool at the end of each month. The retention policy is set for 28 days so in your case 6 days would need to be set, we create a new media set for every backup session (in the job settings).

Ours is setup very simple but we use a LTO7 media library so capacity and performance isn`t an issue. I get what you are saying about using a pool to rotate X amount of days within it but personally an individual full backup per day on to tape is the most fail safe way of ensuring consistency.

[Dima P.]

What I'm trying to achieve here is essentially just having a 10TB volume (media pool) with the latest 7 restore points from the backup. I looked into AWS Storage Gateway Volumes (both cached and stored) but they seem risky to use because if Veeam reads data from the volume (for example to merge an incremental into a full) then I might get charged for requests which are very expensive.

Yes, secondary copy to Gateway VTL is better as your offsite backup chain won't be modified.

Is tape capable of achieving what I'm looking to do while taking advantage of forever forward incrementals and virtual full backups so that I'm not copying my full data-set more than once (the initial run)? We have a 100mbit upload connection so xfering an entire full of ~4.5TB takes several days.

If you want to keep the last 7 days of backups you have write virtual full backup anyway at the beginning of every new week.

I just want to keep using the same tapes and media pool over and over to only have at least 7 days of restores, how would that look? Is that even possible?

Sure, but in case you want to cyclically overwrite the tape i'd recommend you to use 'Creating a new media set for each backup session' option with 7 days retention (with such setup every tape is going to expire on the planned day). Ideally one job run will use one tape but you might need to create additional tapes if more capacity is required for specific job run. Additional you must schedule virtual full backup to run at the begging of the week. The workflow should look like this:

Week 1: Tape 1 (Virtual Full) < Tape 2 (Increment) < Tape 2 (Increment) < Tape 3 (Increment) < Tape 4 (Increment) < Tape 5 (Increment) < Tape 6 (Increment) < Tape 7 (Increment)
Week 2: Tape 1 (Virtual Full) is expired and overwritten by virtual full (at this point all tapes with incremental from Week 1 are not operational since full backup was erased) < Tape 2 is expired and overwritten by incremental job run < etc

[morty]

If your full backup is 4.5TB and your media is 1tb you`d need 35 tapes to run a 7 day rotation, depending if they get compressed it could be a bit less. We use a 28 day rotation with a monthly tape that gets pulled out of the media pool at the end of each month. The retention policy is set for 28 days so in your case 6 days would need to be set, we create a new media set for every backup session (in the job settings).

Ours is setup very simple but we use a LTO7 media library so capacity and performance isn`t an issue. I get what you are saying about using a pool to rotate X amount of days within it but personally an individual full backup per day on to tape is the most fail safe way of ensuring consistency.

Keep in mind this is a VTL in the cloud. We definitely can't be copying a full backup to the cloud everyday.

Is tape capable of achieving what I'm looking to do while taking advantage of forever forward incrementals and virtual full backups so that I'm not copying my full data-set more than once (the initial run)? We have a 100mbit upload connection so xfering an entire full of ~4.5TB takes several days.

If I instead chose to keep 30 days offsite then I'd only have to do a virtual full once every 30 days then, is that correct?

[Dima P.]

If I instead chose to keep 30 days offsite then I'd only have to do a virtual full once every 30 days then, is that correct?

Yes. Basically, you will get the my schema from the post above extended from 7 days to 30 days.

[morty]

Great, I'll give that a try then, thank you.