Delete files, folders from within Backups

[Gostev]

"Not restoring" something is not really an option when talking about image-level backup and restore. Which is why Staged Restore is the only technologically viable approach. Only the specific application itself knows how to correctly handle the content of its database, which is why the application needs to be made running and then instructed to erase sensitive data, during which process you can indeed think of your backup being "read-write". After which, the actual (post-cleanup) machine state can be placed back into production environment.

[BackupBytesTim]

Not to argue the methodology of how Veeam is designed, but the explanation just doesn't make sense to me. It seems like it's just not a feature, not that there's a technical problem with making it a feature.

If we assume a simple disk image, well if I make a disk image with other software from a disk of my computer, I can then edit that disk image, removing or adding things to it, then I can write the disk image back to a physical disk. So the explanation that a disk image can not be modified is not accurate, maybe Veeam can't do it, but that's just then "not a feature", not "technically impossible".

If we assume Veeam's data is stored in Veeam's format (which, it is, that format may involve one or more "disk images", but it is a Veeam proprietary format) (unless that's totally wrong, in which case, do correct me on that), but then since Veeam created the format and Veeam's software added data to the file, why can Veeam's software not also modify the file? Seems again to just be a lack of a feature, not something technically impossible to do. My understanding is Veeam doesn't even have a feature to do actual raw disk images, and even if it did it should still be able to read the contents afterwards provided it was formatted in a way Veeam understood (such as NTFS). But even with a not-raw disk image, ultimately the disk has partitions, formats, volumes, files, metadata, all of it things that Veeam can read and write, so it should be fully capable of modifying the contents of the file as much as it can selectively restore only parts of the backup file to the disk.

I do understand the explanation Gostev gave regarding "specific application(s)" however, outside of specific database contents or Active Directory elements, other app specific things of that sort, the disk ultimately holds files, and if a human wants to remove files from the backup that should totally be doable, now if a human deletes some file that renders an app-specific database restore impossible, well that's the user's fault, but it shouldn't mean Veeam doesn't allow modification of the backup file entirely. Other software can modify backup files just fine, it would be nice if Veeam could do the same.

[Gostev]

Let's approach this differently then. How about you share an example of other image-level backup software that is able to modify guest files directly within an image-level backup then, and we go from there? Because there are a number of pragmatic issues such software would need to solve. For example, I'm naturally curious how it is possible for such functionality to play nicely with incremental block-level backup. When you would be effectively modifying "the past" at the point when "the future" has already happened and is stored as a delta from the unmodified past.

[BackupBytesTim]

My specific regular comparison for Veeam functionality is with Acronis, which does offer the ability to remove past versions from the backup chain, it removes entire versions (incremental, full, or differential backups along a chain, everything captured at the time a backup was performed), not just specific individual files from that backup "version", but functionally I imagine that part doesn't make a difference to how it works with block-level backups. I assume it keeps track of what blocks are associated with what files in each backup "version". So I can remove an entire version and it just deletes the data used exclusively in that version, if certain blocks are needed for recovery of other versions in the chain then those blocks are not deleted. I assume. Honestly I never questioned too technically how it works because it always did what I wanted it to do.

So if I were to explain how I would make that work with modifying the "past" when the "future" relies on data from the past, it'd be just to not delete blocks that are needed for recovery of a future point in time. I'm assuming Veeam already tracks what blocks from what VIB files are needed for recovery, to my knowledge it doesn't restore the VBK, then restore the entire contents of each VIB file up to the desired point in time, so to some extent it must know what data belongs to what files at each point a backup "version" was created.

Similarly though Apple's Time Machine (which is in my opinion the only good backup software for macOS, I wouldn't highly recommend anything else, not Veeam, not Acronis, nothing) can actually delete specific files from specific "versions" from a backup file. And of course if I just clone a disk to a VHDX file or something I can mount it and edit the contents same as I would an actual physical disk.

Side note, the Time Machine recommendation is not a fault of Veeam or any other backup software, it's really just due to quirks about Apple's security that make any other software much less useful in that regard (without significant changes to security settings and very specific procedures and limitations for performing the recovery), I don't expect Veeam to make itself better there, I do understand entirely the issues are with Apple's security preventing other applications from just having full disk access on their own or being able to boot up from recovery media without disabling things, among other things.

[Gostev]

I think I now understand the key piece you're missing: the process of deleting a file from disk does not actually touch disk blocks backing said file. They remain as is and still contain the file's content, which for example makes it possible to "undelete" files. This is the case with pretty much all modern file systems.

When you delete a file, the only thing that is updated is its record in the MFT (Master File Table). You can actually think of MFT content as a "specific database contents of a specific application" from your previous post. It is a complex structure that can contain thousands of file records in the single disk block backing that part of the MFT.

So even if a software does manage to orchestrate individual file deletion from an image inside backup and "save" the modified image state back to the backup, which in case of a single file deletion would mean saving just a single disk block backing the affected part of the MFT, the consequences will be... interesting.

For example, the very next (already created) incremental backup will most likely contain a different version of the same disk block again, but obtained from production storage in the world where the file in question was NOT deleted. So if you restore from this restore point, your file will magically reappear - as again, its actual content stays intact on disk when you perform a file deletion.

And that's the easiest scenario. The other are potentially much worse. Say, the disk block backing the modified MFT part is not replaced by the following incremental backups, however they will certainly replace a bunch of other MFT blocks taken from the world where the file in question was NOT deleted. This means the resulting MFT would be "glued" from disk blocks taken from two different worlds and will mostly likely be inconsistent, with totally unpredictable consequences if you were to restore such machine - for example, the whole volume might fail to mount due to the file system driver seeing an MFT checksum issue or a USN inconsistency.

Of course, all these issues are specific to image-level backup only. For file-level backup, deleting a particular file from backup should be a trivial task.

[BackupBytesTim]

I do understand how all that works, but I don't understand why any of that affects the ability to delete a file, it sounds like it would only be a problem if the software deleting the file doesn't do it properly...

I certainly understand if I were to delete a file from the backup, and it's completely gone, no record of it anywhere, and then I back up the computer again, and the file is still on the computer, and so it gets readded to the backup in the next version, then if I restore a new version of the backup (newer than where the file was removed from the backup) the file will be restored, because it's in the backup.

But your scenario where you mentioned the "block backing the modified MFT part is not replaced by the following incremental backups" sounds like that's a bug in the backup software then, whichever software is being used, Veeam or otherwise. I would think the MFT is an important thing in full disk image backups, does Veeam not back it up each time? I guess I don't see why that would even be a problem that needs to be considered. I mean, if we're going for "What happens if the backup doesn't contain every block needed for proper recovery?" then there's lots of other issues I could bring up too... but I always assumed it just backed up everything that changed, or just everything regardless of changes, depending on the settings for that particular backup job occurrence.

Following the same logic, if I delete a file on my computer, yes, typically the file's contents are left on the disk until they're overwritten for one reason or another, but Windows and the filesystem handles that properly, if I reboot the computer or something the file doesn't just "magically reappear" for any reason, so I guess I don't see why Veeam wouldn't be able to modify the backup file the same as Windows modifies the live disk.

[Gostev]

Veeam or any other backup software will not backup blocks which were not changed since the last backup run. This is obviously not a bug, contrary to that its the only right way to do incremental image-level backups

Now, if you seriously suggest backing up the entire MFT each time, then you should educate yourself about it's structure and what it contains... at which point it should become clear why it's a really bad idea

[test1234]

Hi Veeam

This is a critical and legally significant feature request. This post is being created after discussions with the Veeam engineers and their request for me to make this feature request.

Reference: Veeam Support - Case # 07708755

Context:
We have 40 clients. 2 of of our clients are leaving us and migrating to new provider. There is now a legal requirement to remove ALL TRACE of their data from our datacenters. We have deleted their raw data across all of our servers, however we are unable to remove their data from our backups/restore points.

These are unencrypted backups in this case. I am able to view the files in Guest File Restore etc, however after discussions with veeam techs it became clear I cannot delete anything from within Guest File Restore AND there are no scripts whatsoever that allow selection/deletion of data from restore points. We are now being threatened with legal action, and we remove to remove the restore points of DB backups for our other clients.

Notes: the concept of backing up different clients with different backup jobs was raised to me, it was mutually agreed this cannot always be achieved depending on server design and RPO for backups and data sizes.

Thus, the decision was made that this is a suitable feature request and totally valid/required in the eyes of Veeam already. Please review and let me know any questions

[PetrM]

Hi Marco and Welcome to Veeam R&D Forums!

At first glance, it looks like a potential vulnerability—either due to intruder actions or user mistakes—and furthermore, it contradicts the definition of a restore point: it is no longer a snapshot of the system state at a specific point in time if you modify it. Why can't you delete backups from disk using the existing capabilities?

Thanks!

[pybfr]

On what legal basis is the customer request based ?
Because it comes in contradiction with many important regulations related to data protection like NIS 2 or DORA. It is even a recognized exception to GDPR in Europe…
The trend across all data protection is directed toward more and more immutable which is exactly the opposite of what your customer demands…

[RobertoM]

We operate in Switzerland and you can all imagine this is always a sensitive subject.
Even though, truth to be told, our data protection and privacy law (even the new one) may be considered less strict than GDPR.

Anyway, it is a given that deleting specific files from restore points is not feasible.
Also not feasible is to somehow alter the database backups of an ERP solution to make a former customer disappear.
It is not only unfeasible from a technical standpoint, it does not make any sense.

The normal thinking is that I can delete from our production servers anything related to a specific person or company.
With the exception of what I am legally obligated to keep (some accounting data for example).
Since it is also illegal to keep any kind of data (backups included) forever without reason, the expectation is that sooner or later the restore points will not contain the things I deleted anymore.

Until such time, it is understood that we must keep track of the persons and companies we "soft-deleted" and have a system in place to immediately delete them again in case of restore from backup.
That is the way the GDPR (and swiss) "right to be forgotten" is theoretically handled.

Imagine post-restore scripts to perform database queries on the ERP and some kind of powershell scripts to re-delete the customer's files before the restored system is made accessible to anyone.


I would think that anyone is free to threaten and initiate legal action;
but it wouldn't really go anywhere, because it is widely accepted in the tech/legal field that it is not possible or expected to have a backup solution that allows tampering with previous restore points.
What year is it, 1998 with .zip archives as restore points?

[MoritzG-Seidemann]

I can't find the specific feature atm but I distinctly remember a way for veeam to delete the data post restore with something like a script that executes after the restore is completed.
But that might only be related to restored database data

[test1234]

Hi all

Seems there a lot of questions about legality rather than backup software functionality.

We are a Swiss based company with datacenters in multiple countries. As end-to-end service provider in the life insurance industry, we host the primary and only data of every client. The data laws and business laws apply are dependant on both the country where the data is hosted and the country where the client is based (our South African DC hosts data from 25 countries, all with different data laws, especially in AFRICA.)

When the clients leaves they migrate to a different provider. These are the first 2 significant clients to leave. We hosted their data for 15 years.

Fact 1: each backup of our DB server is historical, as such, even 1 restore point contains the sensitive data of millions of people since go-live. I am confused as to why this is feature request is being argued from a legal position. This specific data centre is in South Africa, there are contractual and legal agreements between us and the customer regarding data store and deletion.

Why is it the opinion of forum users and Veeam that such a request has no legal significance? I have presented a clear situation. In life insurance we are LEGALLY REQUIRED to store 5 years of backups containing financial accounting transactions and any contracts or processes relating to them. Now we are being required to remove the backups of 2 clients, but we cannot go that without violating the legal financial data laws. Africa is not Europe - every country has their own data laws. No EU shared laws

So I am simply requesting Veeam users be given the ability to delete data from backups. Is the issue here that a feature request needs to serve multiple purposes? Goes without saying this feature request can also save disk space if used properly

Otherwise, yes, since the client left the latest monthly backup chain doesn't have their data anymore. In 5 years their data won't exist but they can't wait that long from a data perspective

IF this is not developed then yes, obviously the discussion shifts and it becomes about us convincing them we can't delete their data, but it's strange to see people assuming legal requirements when I was told by Veeam this is a valid request and it makes sense. Also, if this isn't possible in the long term the instruction from my CEO/CTO is simply to migrate away from Veeam because this was possible for us on Crashplan and is certainly possible on other backup solutions.

[Marijn]

Hi test1234,

Let met start by saying that the ability to delete data is a valid but the manner in which you want to do so is a bad idea.

What you're bassicly asking is for Veeam to develop a tool to change all your backup data. As you can imagine that might cause serious issues in the restorability of those restore points and would also be quite a tricky task to perform. Depening on your restore method Veeam would need to mount and index all restore points, remove the data-to-be-deleted and then rebuild all full and incrental restore files? How to handle immutable files? etc. It's not going to happen.

What you should have done, and need to do now, is rethink your overall design. If there is a business request that all customer data must be deletable when a customer leaves then you need to design a solution for that. And no, "asking Veeam to build a feature to do so", should not be part of that solution

Make sure the backups you make are split for every customer. This might also mean that every customer needs to get their own set of databases / vm's in order for them to get their own set of backups. You can then safely delete a backup set without impacting the backup chain of other customers.

edit:
I know you wrote that making seperate, per client, backups isn't an option but this only means that your current overall design does not meet the requirements.

[test1234]

Hi Marijn

Going to focus on your statement "Let met start by saying that the ability to delete data is a valid"

I have no preference or expertise dictating the method to delete data from backups. I am merely requesting this be reviewed for development because it is a need and it seems possible. What other methods are there available?

Regarding block-level storage, encryption etc - I do not understand this. There are various veeam functions and features available for each specific backup job or storage format. For example you cannot make synthetic backups on Minio object storage, but you can on ZFS because it allows block cloning.

I do not accept that you cannot develop certain features because they will only be possible on certain backup formats.

If deleting backup data from unencrypted backups on ZFS storage is POSSIBLE and has VALID use-case I would like to continue to investigate what options we have available. I cannot implement the backup design suggested of different backup jobs for each client and I doubt all your users do this - file level backups are incredibly slow within Veeam and I can't imagine a server design of 1 volume per client in a 100 - 200 client set up

I am not asking Veeam to build a feature to handle a unique request for us. This is something I've always wanted in Veeam that we lost when code42 crashplan was discontinued and now we need it with legal connotations and I believe this would be a useful tool if developed properly for all variables (as each tool within Veeam already is)

[test1234]

Not sure why certain characters are broken in my response, at least do me i see ddd-ddd - you can copy and paste into notepad to read if you see the same pls

[Marijn]

Hi test1234,

Feel free to request any feature you like.
Also, I don't work for Veeam, I just use Veeam to backup. So I do not know anything about your situation other then what you told us here.

I'm just suggesting that if I would have such important data sets with important legal requirements I would have split the data in production too.

Also crashplan still seems to exist, but I would stick to Veeam and fix my design.

[pirx]

I think is was just a bad decision to put different customers data in same jobs and restore points. You are also not using per VM backup chains? I do not yet really get it. The different customers have dedicated VMs that are backed up in jobs that are mixed by customers? And the VM do not have their own backup chain? Because even if you have one job with many VMs, you can delete individual backups.

[Gostev]

I can't find the specific feature atm but I distinctly remember a way for veeam to delete the data post restore with something like a script that executes after the restore is completed.
But that might only be related to restored database data

It's called Staged Restore. Here's a quote from the original What's New document I wrote for version 9.5 U4, which believe or not is over 6 years old now:

Staged Restore enables organizations to make required changes to the database content or application/OS
settings before restoring a VM into the production environment. This is achieved by starting up the required
restore point in a DataLab directly from backup files, injecting and running a custom script in the guest OS while
preserving the changes made, and finally moving the modified state to the production environment to finalize
the restore. Use cases for Staged Restore include:
• GDPR compliance around the right to be forgotten, by enabling you to use the same applications or scripts
designed to clean up personal data from your production systems, when performing the restore of VMs
containing personal data from older backups.
• Streamlining IT workflows by enabling administrators to make the required OS settings changes (such as IP
address, DNS or firewall settings), installing or removing applications and performing any other operations
required to make the restored VM compliant with the target environment before it is actually placed there.

[test1234]

Hi

Our servers are not VMs but physical servers. Our clients DBs total 10-20TBs across 15 servers and one could argue our servers existed in this layout before Veeam existed.

We migrated our exact Code42/crashplan backups to Veeam. Veeam lacks functionality. I am awaiting feedback from Veeam themselves who I met with, reviewed, and they told me to make a feature request. I realise now you are all forum users! I will escalate to Veeam engineer who suggested this

[Gostev]

I'm not just a forum user. Please review the full discussion including all my earlier posts in this thread.

The TL;DR is, it's not practically doable to granularly delete data from image-level backups (requires an extremely complex implementation that comes with recoverability implications); not possible in principle when backups are immutable (which is essential these days); and more importantly is not required by any regulations (what is required is not restoring "forgotten" data back to production).

For your specific case, I recommend you change your deployment according to what vast majority of our service providers are doing:
1/ Use dedicated backup jobs per client, so you can easily delete all of their backups after off-boarding (for example, what if they go bankrupt).
2/ Ensure all incoming backups are encrypted at source, so your staff can't access their content in principle (at least not without your client willingly sharing a password).

[RubinCompServ]

we lost when code42 crashplan was discontinued

Crashplan did file-level backups, not volume/VM level backups; the two are very different. Your request is like saying, "I really like my automobile but I miss a feature that I lost when I gave up my helicopter, so please make my car able to fly"