Veeam and EMC Data Domain Retention Lock

[simo77]

This feature is not in a short-term roadmap. For protecting backups against cyber threats, we recommend deploying V11 Hardened Repository as your primary backup repository. Thanks!

Hello Gostev,
there are many companies that have invested in Data Domain (and use different backup products) and don't want to create additional repositories in other technologies.
Some of your competitors support DD retention lock.
With the ransomware emergence this functionality has become very important.

[Andreas Neufert]

You can use DD retention lock with Veeam. It can be independently configured.
Here is the Dell EMC guide.
Look at the "Fast Copy" approach there. This is completely transparent for us. As it is completely handeled on the storage, you even do not have the attack vector that someone changes settings within the backup software and wait for a longer time until they attack.
https://educationstg.dellemc.com/conten ... cation.pdf

[SE-1]

Take a look at the cyber recovery solution that comes with DD.

The design is pretty simple:
CR comes as an ova which you need to deploy and connect to the replication target DD.
In CR you can schedule an automated fast copy in combination with retention lock (governance/compliance)

It copies the complete content of your replicated mtree to another standalone mtree via fast copy, and places a retention lock on it with a duration you define. It basicly protects you when someone would delete your backups from within veeam.

When using VTL, you can script it your self, which is also very straight forward to do.

https://www.dell.com/support/kbdoc/en-u ... -documents

[barry.cairnsCC]

Hi Andreas, Thanks for the suggestion of fast clone on the DD. I would imagine that then becomes a separate location that Veeam is not aware of in the backup history so nothing for anyone to be able to discover to delete?. How would you suggest the fast copy jobs are initiated would it simply be one fastcopy per day of the entire Mtree used as a backup repository with the relevant retention settings applied to it?? Do you have any more details from a Veeam perspective as the Doc covers the Dell products mainly. Thanks again for the solution I was beginning to struggle to see how we can deliver a solution easily and keep the backup jobs intact.

[Andreas Neufert]

Yes, the fast copy would place the "snapshot" of our backup files somewhere else and make it immutable.
We do not see it and even if we would see it we/someone could not remove the data.
Of cause the unprotected actual data would be deleted when a hacker comes and delete them.

In case of an attack you would create a new fast copy of the immutable data and mount this to Veeam so that we can "see" the data and be able to import the data.

Scheduling needs to be done on the dell emc side for the fast copy. If I remember well the document gives an example how to schedule this.

[massimiliano.rizzi]

Hello Community and good day,

we are in the process of putting back to work a DD6300 Appliance with the DD Boost integration as a secondary repository target for an existing customer. Basically, we will start from scratch with an empty DD6300 file system after a full appliance upgrade to the latest supported version.

The Veeam backup server itself (a physical box) is running the current Veeam Backup & Replication 11 build.

I just wanted to check whether the recommendation below still applies today or whether there is something new:

You can use DD retention lock with Veeam. It can be independently configured.
Here is the Dell EMC guide.
Look at the "Fast Copy" approach there. This is completely transparent for us. As it is completely handeled on the storage, you even do not have the attack vector that someone changes settings within the backup software and wait for a longer time until they attack.
https://educationstg.dellemc.com/conten ... cation.pdf

Wish you a great rest of the day.

Thanks!

Massimiliano

[Andreas Neufert]

This is the correct way to go.

[massimiliano.rizzi]

This is the correct way to go.

Thanks!

[massimiliano.rizzi]

Hello Andreas,

one more question here with regards to the "Fast Copy" approach.

The excerpt below from the Dell EMC guide recommends that we give due consideration to the time when the scheduled Fast Copy operations should be performed:



Based on my understanding, the primary goal here is ensuring that the backup data stored on the Mtree used by Veeam (which is then "Fast Copied" on the "Retention Lock" Mtree) must be consistent in order to be recovered (which makes perfect sense).

For this purpose, I was thinking a possible solution to accomplish this on the Veeam side could be to:

==================================================
1. Use the Disable-VBRJob cmdlet in order to disable the Backup Copy Jobs targeting the Mtree used by Veeam prior to the scheduled "Fast Copy" operation towards the "Retention Lock" Mtree.
2. Use the Enable-VBRJob cmdlet in order to enable the Backup Copy Jobs targeting the Mtree used by Veeam after some time in order to allow the "Fast Copy" operation some time to complete
==================================================

Does it make sense to you ?

Thanks!

Massimiliano

[Andreas Neufert]

I don´t think that this is a good idea as you would have to monitor this closely to not have 0 backups.
We write in a way that we have the data of the current restore point always in a consistent state (so fahr as the backup ran).
I would define a time when all backups needed to be completed and monitor this.
Then perform the FastCopy independant in a way that no one can mess with it.

[massimiliano.rizzi]

I don´t think that this is a good idea as you would have to monitor this closely to not have 0 backups.
We write in a way that we have the data of the current restore point always in a consistent state (so far as the backup ran).
I would define a time when all backups needed to be completed and monitor this.
Then perform the FastCopy independant in a way that no one can mess with it.

I got you. Thank you for taking the time to provide me with the information. It is very much appreciated.

Have a nice day!

[brkdncr]

I think Veeam is in a better position to handle this. The DellEMC guide suggests setting up and using a linux server to script things. Veeam already can access data domain repositories, and already has a scheduling system, and already has a File Copy job (does this use DataDomain FastCopy? If not then that would be a welcome feature). All that's needed is the script to handle the fast copy and the script to handle cleanup.

My feature request for supporting immutable data protection with data domain would be as follows:

Documentation indicating that a 2nd mtree needs to be created.
Documentation indicating that retention lock needs to be configured on that new mtree
a method to receive variables from the Veeam admin that include: retention period settings, how often to run the fast copy scriptt, how long after the retention period to perform cleanup.
A veeam-supported scheduled script that performs a Fast Copy of the backup repository folder to the retention locked mtree with a unique folder name (date/time recommended).
A veeam-supported scheduled script that performs cleanup of the retention locked NFS folder.

[Andreas Neufert]

A Veeam integration would go a different route like the one we release with Catalyst in v12 where we handle immutability of each object and the dependencies in our backup chains correctly.

The current fast copy approach is something that customers can use today. Yes a small Linux system like a NUC or so should be directly connected with the datadomain without additional network access, so that no attacker can manually change anything.

[SE-1]

There is a software that does all that automatically , The software tool is called cyber recovery and comes with DD for free.
The idea is that you replicate your mtree to a vault DD in another physical location, Regarding network access you complete isolate both environments
The CR will create an airgap by closing the replication & network interface
you can schedule when the air gap opens and starts the replication, once the mtree is in sync, it closes the airgap and disables the replication of that mtree and creates a fastcopy to a third mtree + puts retention lock on it (governance or compliance)

[claudiofolu]

CR It`s not free.

[Andreas Neufert]

If you want to use mtree replication, reach out to your DD specialist as Veeam does random IO and force cache flushes you need to deactivate there 2 of the optimizations as well you need to schedule/trigger the replication after backup for fastest processing. You need a bit more bandwith than usual because of the 2 optimizations can not work with Veeam.

As well please make sure that the retention lock is not needed on the primary data for your suggested solution. I do not know the details there.

[Pierpa]

Hi,
I've seen some other entries in this forum related to integration with Datadomain retention lock, but thay are pretty old.
Is this DD retention lock for ddboost storage repos already usable in v11a P20220302 or is this feature in the list for enhancements for the future??

Thanks a lot for any reply.

PIerpa

[foggy]

Hi PIerpa, this went unnoticed - no, Retention Lock support is not yet implemented in Veeam B&R, it is planned for future versions.

[Andreas Neufert]

Hi Plerpa, please see above for a method on how to use it today with Veeam.

[Pierpa]

Hi all,
I've known that finally on Datadinaub DDOS 7.8 the AUTO RETENTION LOCK FEATURE IS SUPPORTED ON DDBOOST.
Additionally, I've seen that DDOS 7.8 is supported on Veeam v11a P20220302, last patch.

Therefore the question now is, since Veeam does not integrate Retention Lock ( as EMC Networker for example ) :

if my backup method is simple "Active Full- Incremental" and is a per-VM backup, may I use auto retention lock on the Datadomain mtree acting as Backup Repository in order to have my backup protected "immutable" since the retention lock time ends ??

For sure, and obviously, I have to set my Automatic retention period LESS than my restore point cycling days in order to let Veeam delete oldest retention points when the ARL period is already ended.

Have I to take care about something to set on the DD ( Automatic lock delay for example ) ??

Any suggestion is well accepted.

P.S: please, add the integration of ARL in Veeam for the next releases... and eventually update me about your plans. As far as I know, nothing is new in Veeam12 about this point.

Have my best regards
Pierpa

[Andreas Neufert]

You can use Veeam with the backup vendor independent retention lock "fast copy" method. See my link to the Dell documentation above.

Any Autolock method that is not integrated into our product as the risk that a backup chain becomes marked as "need to be repaired" and as you can not heal that situation a new backup chain needs to be created. Please let me explain. Veeam is closely monitoring what is happening on disk and if we have access to our backup files. If Veeam for whatever reason can not delete a backup file, it will mark the whole chain as "need to be repaired" and we try whenever you start a backup to heal the situation (delete the file again). As we can not do this and do not know why, our product will not do anything with the backup chain to protect customers and the data integrity. This means whenever you place backup files as undeletable and somehow we try to delete, then you end up in a chain that is in repair mode. The only way to heal this, is by creating a new active full to heal this situation. The old chain is ignored then and you need to manually delete it over time. For this reason WORM capabilities are only supported with specific integrations or if they work independent from Veeam (See DDBoost Fast Copy Rentention Lock or Exagrids similar way. As well we have in v12 the HPE StoreOnce Catalyst Immutability integration).

[Pierpa]

Hi,
concerning this discussion, the main point is that <Jobname>.vbm file HAS NOT to be retention-locked when the job starts.
While backup files are always named in a different way, the <jobname>.vbm file is always the same and is necessary to find a trick to "unlock" it.
The trick is to unlock it via a command on ddomain "mtree retention-lock revert /data/col1/VEEAM_DDBOOST/<myjob>.vbm". It works and allows Veeam to run the job again.

I've seen that Veeam can run a script before the job run, and from here you can spawn a ssh command on the ddomain.
But the bad news is that the "mtree retention-lock revert" command is interactive... and I've not found a way to run it from the Veeam server ( windows ... ) even with powershell ..

Even if I know that in this forum the question is "out of scope" , have you any idea if this remote command can be run in some way from Veeam server??

Any suggestion will be really appreciated. And, please, take this integration with ARL in your features list for the next releases ... I think that it is really needed by Veeam&Ddomain users !!!

With my best regards
Pierpa

[Andreas Neufert]

You can use the Windows SSH backend for this and use it in a script. Or use a independent system that just performs the fast copies scheduled (maybe even hourly).

You can NOT work with the retention lock by default. It will NOT work and you potentially bring your backup chain into the sitaution that it jumps in repair mode until the Veeam operation went through (delete operation that could not be executed). You have to go with the fast copy approach!

[SteveK821]

Just to confirm what Andreas has said: we are a Veeam and DD vendor and we use a custom script that implements SSH commands to create fast copies to another mTree. It runs as a scheduled task and needs SMB activated on the target mTree (to delete expired copies, similar to how brkdncr describes above). The script runs outside of normal backup activity. The hard part is setting up the users to run this without interaction.

You cannot use auto-retention lock on a normal Veeam DD Repository as the VBM file needs to be changed regularly. The fastcopy itself does not require much time to run (seconds).

[Pierpa]

Ok,
I've setup a linux machine with NFS share and the script to version and run the fastcopies. The script fascopy.sh well documented in Dell whitepaper works well if I run it from the linux machine.
I've trusted the Veeam server to run ssh commands vs linux machine, and it is ok.
I can also run the command "ssh root@rhrlbr /home/pierpa/fastcopy.sh" from Veeam server to make the fastcopy on the dd through the linux "bridge".
I can also put the same command on a powershell script and it runs ok.
But if I use this powershell script as a post-command for a Veeam job, the ssh command hangs forever...

Someone can give me a suggestion on that ???

Thanks
Pierpa

[Andreas Neufert]

Hi Pierpa,
tanks for the follow up.
First please allow me to mention that I would not use root for the Linux pass through. Create a user that has only read only access to the script. Otherwise an attacker on the Veeam server side could alter the script.

Anyway for the SH script on Windows.
Did you use a windows editor to edit the SH script? Windows has another line feed character at the end of the line which Linux can not interpret. So write the SH script on Linux and copy it to Windows or use an editor that can switch to Unix mode to edit the script.

Let me know if this was not the root cause.

[Pierpa]

Hi Andreas,
the reason is that Veeam B&R service run as SYSTEM user. Of course, SYSTEM user cannot be trusted vs a linux machine, since you cannot create ssh-keys...
to check, I've started Veeam B&R service as "local Administrator" and not as SYSTEM. I've created ssk-keys for Administrator and now the post-command runs like a charm.
But now the question is: have I to start all the Veeam services as Administrator ( or equivalent local/domain user with the correct rights ) or is enough to change the user for Veeam B&R service?
The user changed to start Veeam B&R service can have side effects on the overall behaviour of the application?

Thanks in advance for any comment
Pierpa

[Andreas Neufert]

I think it does not really matter as long as the users have the local admin rights.

[almac]

Is there an update on Veeam native support for DD Retention Lock? This thread has been open for 7 years and Veeam still requires a convoluted solution to make sure our backups are immutable with Data Domains!

[Gostev]

Veeam simply does not have a business relationship with Dell like we do with other companies, for example HPE, which in turn affects the prioritization of pending feature requests around Data Domain in particular... so purely business considerations, as opposed to "bad PMs" completely ignoring this need for 7 years now.

I mean, HPE resells hundreds of millions worth of Veeam, so anything we do around their storage helps our business. While Dell... well, let's just say they seem to have always seen us as their competitor. And in such circumstances, would you personally prioritize integrations with Dell storage, if you were with Veeam?