Help understanding GFS and Insider Protection

[ag611]

We want to make sure our VCC configuration is in line with best practices, and part of that is whether we need to enable GFS for our backup copy jobs to Veeam Cloud Connect. We have Insider Protection enabled and set to 7 days for each tenant.

I know Veeam recommends enabling GFS in order to really take advantage of Insider Protection, and it even shows a warning if GFS is disabled, but I don't understand why. I can't figure out what kind of attack would be mitigated specifically by having GFS enabled.

The way I see it is this:

- If an insider manually deleted all cloud backups in one go, all the restore points, including the full backup, are recoverable from the recycle bin
- If an insider reduced retention on the backup copy job to 1 restore point, and let it run so older restore points are aged out, and then deleted off-site backups, we'd still have the original manually-deleted full + aged-out incrementals in the recycle bin
- If we enable GFS, and an attacker did everything in #2 AND disabled GFS, we'd still have the GFS full backups in recycle bin, plus everything else

So what attack or threat is addressed by enabling GFS? What am I missing here?

[foggy]

Hi, I recommend reading this blog post and apply to VCSP forum user group to get access to the dedicated private forum for service providers, where similar questions are discussed. Thanks!

[wessexit]

I've been getting my head around this as well and found the mentioned blog post very useful in doing so.

I can see why GFS is needed, but I still have two questions

1. Aren't forward-incremental recovery points in the recycle bin useless?

With forward incremental, 3 recovery points after 3 days the chain looks like this:

Code: Select all

            | FULL > INC1 > INC2
RECYCLE BIN | REPOSITORY

On the 4th day it looks like this:

Code: Select all

     INC1 > | FULL > INC2 > INC 3
RECYCLE BIN | REPOSITORY

Isn't INC1 now part of a different chain?! Surely only recovery points should be kept in the recycle bin where they form part of a valid chain (such as in the event of them all being deleted by an insider).

2. Do you still need GFS with a reverse incremental chain?

With reverse incremental, 3 recovery points after 3 days the chain looks like this:

Code: Select all

            | INC1 > INC2 > FULL
RECYCLE BIN | REPOSITORY

On the 4th day it looks like this:

Code: Select all

     INC1 > | INC2 > INC3 > FULL
RECYCLE BIN | REPOSITORY

As the aged-out incrementals (INC1 in this case) are part of a valid chain they can still be used, right? For an insider attack to be successful in this case (i.e. using reverse incremental) they would have to carry out the attack/damage and for that go go unnoticed for longer than deleted recovery points are retained at the service provider - meaning that GFS does not need to be enabled.

Is this right, am I missing anything?!

[foggy]

Hi Richard, I'm not sure I fully understand your concern in the first case - yes, restore points that were deleted according to retention form a sort of (a part of) a backup chain in the recycle bin as if they existed on disk (there's no full backup though, only incremental points).

As for the second case, your understanding seems to be correct, but having GFS in place you do not need to worry about the fact that recycle bin can be periodically cleaned up, for example.

[veremin]

1. Yes, mostly they are
2. No, you do not

[wessexit]

Thanks foggy and v.Eremin for your responses. It appears both of my suspicions are confirmed. Two final questions:

1. If GFS isn't needed with a reverse-incremental CC copy job do you still get the job warning on the source VBR installation about GFS not being enabled? Obviously having some sort of GFS is better, but we're talking about whether it's necessary for insider protection (which we've established it is not).
2. Is a reverse-incremental CC copy job equivalent in terms of bandwidth to a forward-incremental one all else being equal?

The reason this is all so important is that the insider protection with CC copy jobs is a huge selling point for clients. It's undermined though if there is a massively increased storage requirement (due to FGS and the additional full backups it creates). If it works just as well with reverse-incremental backups then the only downside is reverse-incrementals and not the increased storage requirement!

Thanks!

[foggy]

Will your questions still be relevant considering the fact that backup copy jobs are always forward incremental? Reverse incremental mode is available for regular backup jobs only.

[wessexit]

Crikey @foggy, is that right?!

Can you confirm the minimum safe GFS configuration is 2 weekly? I think this is right as an insider attack on a Sunday could cause you to end up with a compromised GFS weekly *and* also the rest of the chain compromised too... I can't see how a 2nd GFS weekly backup could also be compromised unless your live systems were damaged for over a week without anyone noticing...

This then causes a 2-3 fold storage increase on the Cloud Connect end, right?

[foggy]

Depending on what kind of attack do you mean and the moment it was performed.

[wessexit]

I'm referring to an attack on a Sunday just before a recovery point is selected as that week's GFS point and after n incremental backups have been run by the attacker after where n is the number of recovery points retained by the copy job.

[foggy]

In this case yes, if you have 2 weekiles configured, you will have an additional GFS restore point safe.

[wessexit]

Thanks foggy. It's a real bitch a client has to pay for 2-3x the storage though...

[wessexit]

I had an idea...

If you could configure a maximum copy job frequency on the CC server (e.g. once per day) then it would mean an attacker *cannot* purge the clients "good" data out of the chain by continuously running copy jobs... it would take the attacker x days achieve it by which time you'd know about the attack...

Any chance we can get a viewpoint on this from @Gostev ? Sounds to me like an easy thing to implement that would reduce CC storage requirements by a factor of 2-3...

Thanks!

[Gostev]

Not if you use ReFS.

[wessexit]

Interesting... presumably you're referring to dedpupe functionality within ReFS?

Also wouldn't the client's quota still need to be 2-3x despite dedupe savings within the filesystem or does ReFS report the size of a given folder including dedupe savings?

[Gostev]

Interesting... presumably you're referring to dedpupe functionality within ReFS?

Also wouldn't the client's quota still need to be 2-3x despite dedupe savings within the filesystem or does ReFS report the size of a given folder including dedupe savings?

Yes, that's exactly what I am referring to.

ReFS reports actual size of all files in the given folder (as opposed how much disk space this folder consumes). So, "dedupe savings" is something you benefit from as the service provider, which in the end allows you to provide a lower price per TB for your customers.

[Henrik.Grevelund]

Hi,

This is an old tread but the subject is still valid.
So this is just an update on the newest version behaviour.

After upgrading to version 10 i noticed a reduced number of files being moved to the _RecycleBin folder.
I called support and they explained that in version 10 this is the rules for moving files to the _RecycleBin:

1. Backup copy jobs simple retention(non-GFS retention) is always forever forward incremental(expect one scenario, see below):
https://helpcenter.veeam.com/docs/backu ... ml?ver=100
Increments(.vib) for such backup chains that were removed by retention shouldn't be moved to the recycle bin

2.In case a customer has GFS backups enabled or his jobs are configured with forward incremental retention -> all the deleted backups will be moved to the recycle bin after they are deleted by retention.
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

3. Backup copy job simple retention will turn into forward incremental in case "Read the entire restore point from source instead of synthesizing it from increments" option is selected in the job settings.
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

4. In case Backup copy job has GFS backups enabled, but doesn't have "Read the entire restore point from source instead of synthesizing it from increments" -> Incremental files will not be moved to the recycle bin, only removed GFS backups(.vbks') will get there.

5. If files were deleted via "Files" view of a tenant VBR console or backup were deleted using "Delete from disk" option -> files will be moved to the recycle bin first regardless of retention model.


Veeam suppport explained that the fact that version 9.5U4 moved .vib files from a normal forever incremental jobs was considered a bug, and has been fixed in version 10.
But the documentation still states : it should enable GFS retention

[veremin]

Technical writers team will review the said page and apply the necessary changes. Thanks!

[veremin]

The documentation has been corrected. Thanks!