Cannot Delete Old Backups From a DataDomain Repository

[Greg Lamb]

Hi Everyone,

I have a bit of a problem that I want to run by you. Here is a high level overview of the architecture.

We are using a Data Domain as our primary landing space for backups using DD Boost over Fibre Channel. To support this configuration a Data Domain local user account is required that is granted DD Boost access (permissions) to the Storage Unit (MTree) and this credential is used by Veeam to connect to the backup repository. The MTree is then replicated to our DR site Data Domain and a read-only CIFS share of the replicated copy is mapped to the DR site Veeam B&R/Proxy servers for restore scenarios.

When we first deployed Veeam we performed a number of test backups using different jobs names, configurations etc. as we learned the about the software. Once we developed a standardized approach we went into the Backup & Replication>Backups>Disk view and removed all of the backups from disk however at the time it threw a soft error and removed the backups from the disk view. (I wish I remember the exact error, it has been too long)

As part of our DR and audit requirements we routinely restore the MTree replicated backups at our DR site and vice versa to validate...

The Problem:

Recently I reviewed the replicated CIFS share via Windows Explorer and discovered that the "testing" backups still live on disk and were not deleted. At that point I created a temporary CIFS share on the source side to validate if it was a replication issue. Sure enough the data lives on both the source and replicated MTrees. Naturally the next step would be to delete the test backups however we receive an "Access Denied" permission error. After reviewing the folder NTFS permissions it lists the local Data Domain account with Special (Full Control) and Everyone read-only permissions (which we have restricted from the Data Domain ACLs to a specific user/defined server). Since the Data Domain local user is the only user that has elevated permissions I cannot delete the files. I also cannot take ownership of the folder. When trying to map the network drive using the local Data Domain user account it says that it cannot because "it is in use" (by Veeam).

Is there a way to leverage the Veeam application or Data Domain integration so that we can delete the test backup files. The key is that from within the Veeam Backup & Replication>Backups>Disk view the backups are not listed (the DB reference was purged).

P.S. I love the design concept around securing the backup files using the repo credentials

Support Case #01759874

[foggy]

Looks like only database records have been removed, however actual data have not (probably, due to some access issue the displayed message was talking about). Have you tried to rescan the repository from Veeam B&R?

[Greg Lamb]

Hey Foggy,

Unfortunately a rescan didn't do the trick. I believe that it appears to be a permissions related problem.

After further review and playing around I have got to the bottom of it.

Once upon a time when first testing out Veeam and the Data Domain integration, we created a temporary Data Domain local DD Boost user account and this credential was used to connect to the Data Domain from within the Veeam Backup Repository configuration. Later on we deleted this local account and created a new account with the proper naming convention etc...

When Veeam performs a backup (to a Data Domain, unsure if this applies to all repos) the backup files NTFS permissions are set to the backup repository credentials at the time of backup. When this credential is changed at the Backup Repository level and the old account is deleted from the DataDomain, Veeam essentially no longer has permissions to delete the backup files. I could see this when reviewing the NTFS permissions on the backup files as it had a SID for a deleted account.

In order to delete the files, I modified the ACLs on the MTree CIFS share so that a Domain Administrator could access the files via Windows Explorer. At this point I was able to take ownership, add the domain admin user to the folder with NTFS full control permissions replacing the contents permissions and then delete the files. As the MTree replicated the replicated copy also was cleaned up.

I hope that this info helps someone else one day

[foggy]

Thanks, Greg, for letting us know. Much appreciated.

[arslanali07]

Hello,

If you have Data Domain connected to Veeam with DDBoost connection and you want to delete some backup job files from the storage unit, then simple way is follows:
1) Create a NFS share related to Storage Unit Directory location,
2) Mount this NFS share to and Linux machine,
3) Browse and delete related jobs files

It is usable for who delete some jobs but not in repository,

Cheers,

AAA
Support Case #01759874