Comprehensive data protection for all workloads
Post Reply
ag611
Service Provider
Posts: 3
Liked: never
Joined: Aug 20, 2018 7:59 pm
Contact:

Help understanding GFS and Insider Protection

Post by ag611 »

We want to make sure our VCC configuration is in line with best practices, and part of that is whether we need to enable GFS for our backup copy jobs to Veeam Cloud Connect. We have Insider Protection enabled and set to 7 days for each tenant.

I know Veeam recommends enabling GFS in order to really take advantage of Insider Protection, and it even shows a warning if GFS is disabled, but I don't understand why. I can't figure out what kind of attack would be mitigated specifically by having GFS enabled.

The way I see it is this:

- If an insider manually deleted all cloud backups in one go, all the restore points, including the full backup, are recoverable from the recycle bin
- If an insider reduced retention on the backup copy job to 1 restore point, and let it run so older restore points are aged out, and then deleted off-site backups, we'd still have the original manually-deleted full + aged-out incrementals in the recycle bin
- If we enable GFS, and an attacker did everything in #2 AND disabled GFS, we'd still have the GFS full backups in recycle bin, plus everything else

So what attack or threat is addressed by enabling GFS? What am I missing here?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Help understanding GFS and Insider Protection

Post by foggy » 1 person likes this post

Hi, I recommend reading this blog post and apply to VCSP forum user group to get access to the dedicated private forum for service providers, where similar questions are discussed. Thanks!
wessexit
Influencer
Posts: 13
Liked: never
Joined: Oct 13, 2017 5:27 pm
Full Name: Richard Maynard
Contact:

Re: Help understanding GFS and Insider Protection

Post by wessexit »

I've been getting my head around this as well and found the mentioned blog post very useful in doing so.

I can see why GFS is needed, but I still have two questions

1. Aren't forward-incremental recovery points in the recycle bin useless?

With forward incremental, 3 recovery points after 3 days the chain looks like this:

Code: Select all

            | FULL > INC1 > INC2
RECYCLE BIN | REPOSITORY
On the 4th day it looks like this:

Code: Select all

     INC1 > | FULL > INC2 > INC 3
RECYCLE BIN | REPOSITORY
Isn't INC1 now part of a different chain?! Surely only recovery points should be kept in the recycle bin where they form part of a valid chain (such as in the event of them all being deleted by an insider).

2. Do you still need GFS with a reverse incremental chain?

With reverse incremental, 3 recovery points after 3 days the chain looks like this:

Code: Select all

            | INC1 > INC2 > FULL
RECYCLE BIN | REPOSITORY
On the 4th day it looks like this:

Code: Select all

     INC1 > | INC2 > INC3 > FULL
RECYCLE BIN | REPOSITORY
As the aged-out incrementals (INC1 in this case) are part of a valid chain they can still be used, right? For an insider attack to be successful in this case (i.e. using reverse incremental) they would have to carry out the attack/damage and for that go go unnoticed for longer than deleted recovery points are retained at the service provider - meaning that GFS does not need to be enabled.

Is this right, am I missing anything?!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Help understanding GFS and Insider Protection

Post by foggy »

Hi Richard, I'm not sure I fully understand your concern in the first case - yes, restore points that were deleted according to retention form a sort of (a part of) a backup chain in the recycle bin as if they existed on disk (there's no full backup though, only incremental points).

As for the second case, your understanding seems to be correct, but having GFS in place you do not need to worry about the fact that recycle bin can be periodically cleaned up, for example.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Help understanding GFS and Insider Protection

Post by veremin »

1. Yes, mostly they are
2. No, you do not
wessexit
Influencer
Posts: 13
Liked: never
Joined: Oct 13, 2017 5:27 pm
Full Name: Richard Maynard
Contact:

Re: Help understanding GFS and Insider Protection

Post by wessexit »

Thanks foggy and v.Eremin for your responses. It appears both of my suspicions are confirmed. Two final questions:
  1. If GFS isn't needed with a reverse-incremental CC copy job do you still get the job warning on the source VBR installation about GFS not being enabled? Obviously having some sort of GFS is better, but we're talking about whether it's necessary for insider protection (which we've established it is not).
  2. Is a reverse-incremental CC copy job equivalent in terms of bandwidth to a forward-incremental one all else being equal?
The reason this is all so important is that the insider protection with CC copy jobs is a huge selling point for clients. It's undermined though if there is a massively increased storage requirement (due to FGS and the additional full backups it creates). If it works just as well with reverse-incremental backups then the only downside is reverse-incrementals and not the increased storage requirement!

Thanks!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Help understanding GFS and Insider Protection

Post by foggy »

Will your questions still be relevant considering the fact that backup copy jobs are always forward incremental? Reverse incremental mode is available for regular backup jobs only.
wessexit
Influencer
Posts: 13
Liked: never
Joined: Oct 13, 2017 5:27 pm
Full Name: Richard Maynard
Contact:

Re: Help understanding GFS and Insider Protection

Post by wessexit »

Crikey @foggy, is that right?!

Can you confirm the minimum safe GFS configuration is 2 weekly? I think this is right as an insider attack on a Sunday could cause you to end up with a compromised GFS weekly *and* also the rest of the chain compromised too... I can't see how a 2nd GFS weekly backup could also be compromised unless your live systems were damaged for over a week without anyone noticing...

This then causes a 2-3 fold storage increase on the Cloud Connect end, right?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Help understanding GFS and Insider Protection

Post by foggy »

Depending on what kind of attack do you mean and the moment it was performed.
wessexit
Influencer
Posts: 13
Liked: never
Joined: Oct 13, 2017 5:27 pm
Full Name: Richard Maynard
Contact:

Re: Help understanding GFS and Insider Protection

Post by wessexit »

I'm referring to an attack on a Sunday just before a recovery point is selected as that week's GFS point and after n incremental backups have been run by the attacker after where n is the number of recovery points retained by the copy job.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Help understanding GFS and Insider Protection

Post by foggy »

In this case yes, if you have 2 weekiles configured, you will have an additional GFS restore point safe.
wessexit
Influencer
Posts: 13
Liked: never
Joined: Oct 13, 2017 5:27 pm
Full Name: Richard Maynard
Contact:

Re: Help understanding GFS and Insider Protection

Post by wessexit »

Thanks foggy. It's a real bitch a client has to pay for 2-3x the storage though...
wessexit
Influencer
Posts: 13
Liked: never
Joined: Oct 13, 2017 5:27 pm
Full Name: Richard Maynard
Contact:

Re: Help understanding GFS and Insider Protection

Post by wessexit »

I had an idea...

If you could configure a maximum copy job frequency on the CC server (e.g. once per day) then it would mean an attacker *cannot* purge the clients "good" data out of the chain by continuously running copy jobs... it would take the attacker x days achieve it by which time you'd know about the attack...

Any chance we can get a viewpoint on this from @Gostev ? Sounds to me like an easy thing to implement that would reduce CC storage requirements by a factor of 2-3...

Thanks!
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Help understanding GFS and Insider Protection

Post by Gostev »

Not if you use ReFS.
wessexit
Influencer
Posts: 13
Liked: never
Joined: Oct 13, 2017 5:27 pm
Full Name: Richard Maynard
Contact:

Re: Help understanding GFS and Insider Protection

Post by wessexit »

Interesting... presumably you're referring to dedpupe functionality within ReFS?

Also wouldn't the client's quota still need to be 2-3x despite dedupe savings within the filesystem or does ReFS report the size of a given folder including dedupe savings?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Help understanding GFS and Insider Protection

Post by Gostev »

wessexit wrote: Apr 11, 2019 2:21 pmInteresting... presumably you're referring to dedpupe functionality within ReFS?

Also wouldn't the client's quota still need to be 2-3x despite dedupe savings within the filesystem or does ReFS report the size of a given folder including dedupe savings?
Yes, that's exactly what I am referring to.

ReFS reports actual size of all files in the given folder (as opposed how much disk space this folder consumes). So, "dedupe savings" is something you benefit from as the service provider, which in the end allows you to provide a lower price per TB for your customers.
Henrik.Grevelund
Service Provider
Posts: 160
Liked: 18 times
Joined: Feb 13, 2017 2:56 pm
Full Name: Henrik Grevelund
Contact:

Re: Help understanding GFS and Insider Protection

Post by Henrik.Grevelund »

Hi,

This is an old tread but the subject is still valid.
So this is just an update on the newest version behaviour.

After upgrading to version 10 i noticed a reduced number of files being moved to the _RecycleBin folder.
I called support and they explained that in version 10 this is the rules for moving files to the _RecycleBin:

1. Backup copy jobs simple retention(non-GFS retention) is always forever forward incremental(expect one scenario, see below):
https://helpcenter.veeam.com/docs/backu ... ml?ver=100
Increments(.vib) for such backup chains that were removed by retention shouldn't be moved to the recycle bin

2.In case a customer has GFS backups enabled or his jobs are configured with forward incremental retention -> all the deleted backups will be moved to the recycle bin after they are deleted by retention.
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

3. Backup copy job simple retention will turn into forward incremental in case "Read the entire restore point from source instead of synthesizing it from increments" option is selected in the job settings.
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

4. In case Backup copy job has GFS backups enabled, but doesn't have "Read the entire restore point from source instead of synthesizing it from increments" -> Incremental files will not be moved to the recycle bin, only removed GFS backups(.vbks') will get there.

5. If files were deleted via "Files" view of a tenant VBR console or backup were deleted using "Delete from disk" option -> files will be moved to the recycle bin first regardless of retention model.


Veeam suppport explained that the fact that version 9.5U4 moved .vib files from a normal forever incremental jobs was considered a bug, and has been fixed in version 10.
But the documentation still states : it should enable GFS retention
Have nice day,
Henrik
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Help understanding GFS and Insider Protection

Post by veremin »

Technical writers team will review the said page and apply the necessary changes. Thanks!
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Help understanding GFS and Insider Protection

Post by veremin »

The documentation has been corrected. Thanks!
Post Reply

Who is online

Users browsing this forum: abdul_bari, Google [Bot] and 205 guests