Availability for the Always-On Enterprise
Post Reply
CloudMSP
Service Provider
Posts: 30
Liked: 11 times
Joined: Jul 16, 2017 5:39 am
Full Name: Veeam MSP
Contact:

Deleted Backups Protection - Urgent Questions!

Post by CloudMSP » Dec 30, 2017 7:04 pm

Hello.

I don't quite understand this:
If the tenant plans to create off-site copies of backed-up data with a backup copy job, it should enable GFS retention settings in the job properties. This way, Veeam Backup & Replication will be able to protect backups created by the job against an attack when a hacker reduces the job's retention policy and creates a few incremental backups to remove backed-up data from the backup chain.

Without GFS retention settings enabled, the backup copy job will complete with a warning. In the job statistics window, Veeam Backup & Replication will display a notification advising to use the GFS retention scheme for the job. Please note that the warning is displayed only if the tenant backup server runs Veeam Backup & Replication 9.5 Update 3. In earlier versions of Veeam Backup & Replication, the warning will not be displayed, and the backup copy job will complete with the Success state.
We keep are the service provider and we keep 7 days (7 Restore Points) of offsite backups. I want to enable the deleted backup protection for say 7 days (we should know by then if something has gone wrong).

Why must I also setup a GFS scheme? Won't any deleted backups be protected for 7 days anyway ?

And if I do setup say keeping just 1 weekly backup in the BCJ? What protection is this giving me? This just means I might end up with just my old weekly backup if a hacker does indeed get get access to the client console? Also can't the hacker just uncheck those boxes?

Please help me understand the possible outcomes and the way Veeam can protect me. Thanks.

tdewin
Veeam Software
Posts: 1301
Liked: 426 times
Joined: Mar 02, 2012 1:40 pm
Full Name: Timothy Dewin
Contact:

Re: Deleted Backups Protection - Urgent Questions!

Post by tdewin » Dec 31, 2017 10:31 am

Well I haven't tested it myself but it does make sense to me now. The regular chain of a backup copy job is forever incremental. So it means that the VBK is not really deleted but updated on a daily basis. Keeping track of the changes and then "unmerging" them for recovery would be a real challenge to program.

For a GFS point, keeping and recovery is rather simple. Let's say a weekly full is being deleted, well you just keep the full a bit longer and no real action needs to be taken to recover from it. You can just import the VBK and that's it.

BTW we have a separate subforum for cloud providers and this kind of questions. Go to user Control Panel > Usergroups (tab) and apply for the "Cloud & Service Providers" group

CloudMSP
Service Provider
Posts: 30
Liked: 11 times
Joined: Jul 16, 2017 5:39 am
Full Name: Veeam MSP
Contact:

Re: Deleted Backups Protection - Urgent Questions!

Post by CloudMSP » Dec 31, 2017 10:38 am

Ok I will try to get in those forums.

I still don't get it though, so does this mean that, if a hacker did get access and ran a bunch of incremental backups, I might lose all my daily restore points and be forced to rely on the single GFS weekly backup I had saved? So I still wont truly be protected?

tdewin
Veeam Software
Posts: 1301
Liked: 426 times
Joined: Mar 02, 2012 1:40 pm
Full Name: Timothy Dewin
Contact:

Re: Deleted Backups Protection - Urgent Questions!

Post by tdewin » Dec 31, 2017 10:54 am

Well I guess the PMs will be able to shed more light on it but based on this:
https://helpcenter.veeam.com/docs/backu ... tml?ver=95

I would say:
If the hacker "deletes the backups from disk" directly, the files are being protected and moved to the recycle bin folder. So your incremental points would be protected.
If the hacker knows how Veeams work, he could adapt the simple retention from the Backup Copy Job. In this case, because a backup copy job merge takes place, it is not just deleting files, it is updating the VBK and in doing so removing the data without actually deleting files (blocks are just being overwritten in the merge process). Thus the protection will only protect against GFS deletion.

Think of it is this way. The protection will work if a file system command is given to delete a file from disk.

Theoretically, we could protect against this by keeping the "removed data" from the fulls in reverse increments ("VRB" format), but this would cause a huge IO burst. Another way would be to delay merges and keep longer forward chains, but I guess it would also create a false idea of having Full GFS points that are not actually there. (eg, customer says 7 day of simple points, the provider side however keeps 7+<x> protected days of simple retention). A final way would be to grow the VBK, and not overwrite blocks, but this would require the tracking of blocks that are not being deleted because of this mechanism. But like I said, I guess all of them are adding another level of complexity and I guess it would confuse more people.

But again, I'm just an SE, let's wait for the PM's to confirm.

anthonyspiteri79
Veeam Software
Posts: 223
Liked: 29 times
Joined: Jan 14, 2016 6:48 am
Full Name: Anthony Spiteri
Location: Perth, Australia
Contact:

Re: Deleted Backups Protection - Urgent Questions!

Post by anthonyspiteri79 » Jan 04, 2018 6:55 am

Just wondering if you have tested the above scenario on your platform or in your lab? I'm putting this same config through my Cloud Connect Backup Lab and I'll hopefully be able to clarify by doing. Just takes time with the GFS policy to kick in etc.

While that's sorting it's self out, the full message that you get when you don't enable GFS for Backup Copy Jobs on Update 3 is:
Your service provider has implemented backup files protection against deletion by an insider for this cloud repository. To protect against advanced attack vectors, we recommend that you configure your cloud backup jobs to keep multiple full backups on disk (as opposed to forever-incremental chain with a single full backup).
Anthony Spiteri
Global Technologist, Product Strategy | VMware vExpert
Email: anthony.spiteri@veeam.com | Mobile: +61488335699
Twitter: @anthonyspiteri | Skype: anthony_spiteri

Gostev
Veeam Software
Posts: 22802
Liked: 2801 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Deleted Backups Protection - Urgent Questions!

Post by Gostev » Jan 04, 2018 4:02 pm

Yes, Timothy is correct.
CloudMSP wrote:does this mean that, if a hacker did get access and ran a bunch of incremental backups, I might lose all my daily restore points and be forced to rely on the single GFS weekly backup I had saved?
That is correct. However, it is important to realize that in reality, it is impossible to quickly "run a bunch of incremental backups" over the wire - because technically speaking, those will be incremental backups with a size close to one of the full backup (as ransomware encryption will completely changes virtual disk contents). So, we're talking pretty much initial full over the wire - most attacks will be detected well before even the first such incremental backup completes.

anthonyspiteri79
Veeam Software
Posts: 223
Liked: 29 times
Joined: Jan 14, 2016 6:48 am
Full Name: Anthony Spiteri
Location: Perth, Australia
Contact:

Re: Deleted Backups Protection - Urgent Questions!

Post by anthonyspiteri79 » Jan 10, 2018 4:32 am 1 person likes this post

CloudMSP, i've completed a blog post on Deleted Backup Protection. You can find it here: https://anthonyspiteri.net/deeper-look- ... -update-3/

Hopefully you find it beneficial. Also, let us know how you are going.
Anthony Spiteri
Global Technologist, Product Strategy | VMware vExpert
Email: anthony.spiteri@veeam.com | Mobile: +61488335699
Twitter: @anthonyspiteri | Skype: anthony_spiteri

Post Reply

Who is online

Users browsing this forum: Bing [Bot], DGrinev, Google [Bot] and 42 guests