GFS for primary backup jobs

[stephen.loera]

Hello,

Just wanted to add that I also would love to have GFS ability to be built into primary backups jobs.

Maybe this is a separate feature request, but I would like the option of having a, let's call it a "simple backup copy job." This type of job would literally just copy the backup files to another repository, like a file copy job. I have clients who want to send backup copies to a CCB Service Provider that exactly mirrors the source backup job.

Why? For example, a primary backup job can be configured to backup only on weekdays, but backup copy jobs are setups as "intervals". If a source or primary backup jobs only runs on weekdays, but the backup copy job runs everyday, backup copy jobs report errors on the weekend because there are no new restore points.

It would be nice to have a simple backup copy job that doesn't have all of the I/O overhead of a traditional backup copy job that operates in forever forward incremental mode.

[DGrinev]

Hey Stephen,

Thank you for the feedback and feature request!

You should keep in mind that simple file copy can deliver a backup file to the target with the corrupted data due to numerous reasons (i.e. unstable connection).
Right now you are protected, since if the source is healthy the backup copy job can guaranty that the target backup will be healthy too, because it's working on the data block level.
Please review this thread for additional information.

[evilaedmin]

So I wanted to ask everyone here: if it helps us to deliver this functionality sooner, would it be OK to only have GFS available for backup modes with periodic synthetic or active fulls - at least initially? Because then, all functionality comes down to NOT deleting some backup files during the retention policy processing.

This means it won't be available for forever forward incremental backup, like it is available with Backup Copy jobs (which are effectively forever forward incremental). Or for reversed incremental backup. There are significant extra complexities there, since these type of chains do not have "native" fulls - they need to be created. And this functionality is not easily portable to primary jobs.

Backup appliance user here: having GFS available for period synthetic fulls would be just about perfect and how I wish the product worked .

[RGijsen]

+1 for us as well. We are a small company hosting RDS end Exchange services for a number of clients. We rent a rack in a datacenter, in which we have all our hardware, including a large Veeam repository machine. We have multiple gbit internet connections, but unfortunately a slow l2tp/ipsec site-to-site VPN which because of some latency on the remote-backup location I can't get past 70mbps.
We use want to use our remote-backup (copy jobs) to only store the last week of backups. We'd only need the copies if something went terribly wrong in the main datacenter. Because of the greatly bigger speeds and also much more available space in our primary repository, that's where I'd like to store the GFS backups, as restoring them from our remote machine if I needed to, it's just way, way too slow.

Apart from that, apart from Veeam I've never worked with a backup product so far that does NOT support GFS of some kind in the primary store.

So long story short, +1 for us too

[nwbc]

Hey there,
is there an update currently available on the progress of the GFS request?

[DGrinev]

Thank you for keeping interest to the feature!
We cannot share any ETA for the upcoming features, but you can be sure that the work is in progress! Thanks!

[Sean.Lucas]

I also would like to see GFS retention on primary backup jobs.

In our scenario we would like to be able to use the primary backup job to directly backup to our DR site's repository instead of having to first create a local backup then a copy job out to our DR site.

We would also like this for Cloud Connect, having the option for our customers to backup directly to the Cloud Connect repository with GFS retention would be great!

[deduplicat3d]

+1 It would make my life a lot easier to have GFS on primary jobs. Backup copy would then be used just for emergencies where primary failed

[Gostev]

Backup appliance user here: having GFS available for period synthetic fulls would be just about perfect and how I wish the product worked.

Sounds good, thanks everyone for confirming. I am doing my best to make sure this implementation gets into the immediate update. Thanks!

[Regnor]

In our scenario we would like to be able to use the primary backup job to directly backup to our DR site's repository instead of having to first create a local backup then a copy job out to our DR site.

+1 We're already doing this for some customers as backup copy is just too slow for them.

[dhayes16]

Yes. Add us to the list who would love this features. Veeam is awesome on so many levels but we went round and round logically on this thinking we were missing something. As an MSP the backup solution we have offered with storagecraft was exactly that. Several years of data onsite in gfs type format and only a few retention points on the cloud for DR. Veeam seems to be flipped around where they focus on offsite being gfs long term and local being short term. I am glad I am not the only one not understanding why it is not there.

For now we are doing primary backups with 7 day retention. Then a backup copy job based on that to a veeam cloud connect provider to give us the offsite insurance policy. Then we have a second backup copy job to local storage with the gfs functionality to give us the long term local retention. Obviously it is not efficient since we will have a duplicate vbk in there eating up space as well as wasted cpu.

So I am hoping this makes it in soon.

Thanks Veeam for listening to your customers.

[Gostev]

All, as we're working on this feature I wanted to ask your preference on GFS backup expiration settings. How would it be more convenient for you to specify the time period of how long those GFS restore points should be kept?

The current working idea is as follows:

Keep weekly full backup for X weeks
Keep monthly full backup for X months
Keep quarterly full backup for X years or quarters or months?
Keep yearly full backup for X years

Originally, I wanted to make it consistent everywhere (using the same period as the actual GFS type), however I realized that for me personally, counting the time period in quarters is somehow inconvenient. Is it just me?

An alternative I considered would be just to use days everywhere:

Keep weekly full backup for X days
Keep monthly full backup for X days
Keep quarterly full backup for X days
Keep yearly full backup for X days

But this will result in quite big numbers at the top end which are not immediately obvious what do they mean.

I guess my question is, how does your business define SLAs on how long the particular backup should be kept? It would be logical to match that, so that you don't have to do any conversions at all.

[mkaec]

Another application I have experience with solved this problem by including a dropdown to allow the user to specify the unit of measure for each retention point. One user could specify "3 months" and another could specify "90 days". I would set up mine in days because I would set the oldest backup to drop off a few days before the new backup came in.

For the current working idea, I agree that specifying quarterly retention in quarters is a bit awkward. "Months" makes more sense to me there. Though, I probably wouldn't use quarterly retention. I typically use weekly and monthly.

[ChrisRoad]

@Gostev

I do like

Keep weekly full backup for X days
Keep monthly full backup for X days
Keep quarterly full backup for X days
Keep yearly full backup for X days

Yes, numbers will be big, but will remove the inherent uncertainness of what 3 weeks or 2 months really means. (i.e. does "two weeks" now mean that the backup is kept for 14 days and deleted on the 15th, or kept for 20 days and deleted on the 21st?)

Is "quarterly" still a requirement from customers?
We got rid of that long time ago and just use 6 or 9 monthly backups in cases we do somehow still have to cover the requirement of a traditional "quarterly" backup. (bust most often customer want monthly backups for up to a full year anyway)

[Gostev]

@Marc more options is always an easy way out. However, more options => more code => more bugs => worse reliability, so we always strive to NOT provide options unless absolutely necessary. Flexibility to set the time period using both "90 days" and "3 months" ways does not seem essential at all (does not add extra value), so let's just make a call which way of the two is more convenient?

@Chris, good feedback that matches lots of things we were talking through with developers.

1. Yes, dropping quarterly completely is another option - less is better.

2. Yes, "2 months" can be anywhere betwween 59 and 62 days, depending on the month - so not deterministic. But in the retention logic, we can simply consider each month to be 31 days, to make sure backups are never deleted before any 2 months have passed (think about July + August scenario), so that it is not an issue. While deleting a backup 1 day later in some cases is obviously not a problem. This way, UI remains clean and simple (2 months), and backups are guaranteed to be stored for the specified period of time no matter what.

3. Days are great until you see something 183 days or 1825 days (hint, it means 6 months and 5 years). This will look really messy and unclear in UI for someone who did not set up the retention originally when creating the job, but has to manage one.

Answering your question, "keep backup for 2 weeks" means it will be deleted on 15th day since its creation (as soon as given backup age exceeds 2 weeks = 14 days). In other words, users will set the retention exactly according to the business requirements, without the need to translate "business" language into "technical". For example, for backups of a system maintaining the credit card payments log, they will set retention to 3 years - which is exactly what the corresponding law mandates.

BTW, in talking to some Veeam folks with backup administration IT background in parallel, so far the following UI is winning in their mind:

Keep weekly full backups for X weeks
Keep monthly full backups for X months
Keep yearly full backups for X years

Just about everyone agree that quarterly retention is dead.

[mkaec]

Quarterly retention is currently available in BCJs. The real trick will be able to figure out how to remove it from there in order to maintain consistency.

[Gostev]

Right. We won't touch GFS in Backup Copy jobs for now at all (no time to make this change in the next release). We'll deal with it later, but end state is BCJ retention becoming the same as what we're discussing here, of course.

[dhayes16]

Personally I like..

Keep weekly full backups for X weeks
Keep monthly full backups for X months
Keep yearly full backups for X years

I really do not think we sacrifice anything by keeping it simple here. This is really how our SLA for on prem would look.

[Gostev]

Thanks Dave!

[GreenAlpha55]

I would like to vote for GFS on backup jobs as well. Because I can't I am forced to have a backup copy job (and use double the storage for no reason).

[engedib]

Personally I like..

Keep weekly full backups for X weeks
Keep monthly full backups for X months
Keep yearly full backups for X years

I really do not think we sacrifice anything by keeping it simple here. This is really how our SLA for on prem would look.

This is exactly what we need as well Easy, simple.

[DerOest]

Personally I like..

Keep weekly full backups for X weeks
Keep monthly full backups for X months
Keep yearly full backups for X years

I really do not think we sacrifice anything by keeping it simple here. This is really how our SLA for on prem would look.

I would like to vote for GFS on backup jobs as well. Because I can't I am forced to have a backup copy job (and use double the storage for no reason).

+1

This will make configuration a lot easier (less jobs to take care of etc.) and save the additional copy space (which is either wasted, and/or puts more load on the storage system running deduplication etc.)

Btw, our take on "3-2-1" is like:
a) Primary Data (2 Datacenters, redundant/mirrored storage)
b) Backup to onPrem Storage at a third location (multiple 10Gb lines available, with the whole GFS scheme)
c) Backup to Tape as Desaster Recovery/Offline/Airgap (so not keeping a lot of tape around, just the last 4 Weekly & 2 Monthly for DR case)

[GreenAlpha55]

I have a hard time understanding "3-2-1" with Veeam. Veeam essentially forces "4-2-1". If you're not doing "4-2-1", then if your offsite data is destroyed your GFS backups are gone.

"4-2-1" meaning;
Production > Backup Job > Backup Copy w/GFS > Backup Copy w/GFS (offsite) = necessary for GFS at 2 locations.

"3-2-1" currently;
Production > Backup Job > Backup Copy w/GFS (offsite) = GFS only offsite. If offsite goes down, GFS backups are gone.

"3-2-1" would only be possible when GFS is implemented on backup jobs. = GFS at 2 locations. Eliminates need for local backup copy. YES!
Production > Backup Job w/GFS > Backup Copy w/GFS (offsite)

[KelKnowles]

I constantly run into a need for this feature. One customer I have hourly backups on disk with daily synthetic fulls but they also want long term past 2 weeks or so on the same primary storage. I already adhere to the 3-2-1 rule with cloud and tape copies so having to do a local backup copy uses up additional space and also disk IO that could just be a fast clone / spaceless full.

[petben]

I see this has been discussed since 2011, when are Veeam going to allow multiple schedules to be set on the same job. so for example a user can set a daily/weekly/monthly backup schedule. this seems like a basic, core part of backup.

Yes you can setup multiple jobs but there's no deduplication across jobs.

[foggy]

Hi Pete, please see discussion above for the answer. Thanks!

[arkroyal]

Love VBR for the most part, but the backup and backup copy retention options are not exactly the most intuitive to try and figure out. I have got it to the point where I am (reasonably) happy with my retention policies, but having the backup copy job running for no purpose, i.e. no new backup to copy, seems to me to be a waste of resource. Another vote here for:

Keep weekly full backups for X weeks
Keep monthly full backups for X months
Keep yearly full backups for X years

[Gostev]

Yes, this ended up to be the final design for this feature.

[mjm599]

Hi,

Does anyone know when this functionality will be available? Or any advancement on the current configuration options available for Azure object storage integration?

Thanks,

[veremin]

We're working on this feature, but cannot provide ETA at the moment. Thanks!