Copy backups to rotating drives for offline storage

[bjesteadt]

I am trying to introduce a rotated offline hard drive media pool for taking four weeks of backups and storing them in an offsite and offline location, but we don't want to do so with our primary backup repository. Wasn't sure if this is a good issue for here or tech support, so I opened case ID# 03144837, and then posted here as well.

We looked at the backup copy job option, but the backup copy job option doesn't support rotated media pools. Could we use a post-processing script on the backup copy job to replicate the files to a different drive that could be rotated? I found a forum post that describes something similar to this, but thought maybe there is a newer version of the script available.
https://forums.veeam.com/veeam-backup-r ... 96-15.html

The other wrinkle is that I want to use Windows Server 2016 deduplication to reduce the footprint of the backups. I presume that all drives would need deduplication enabled. What else would be required to keep it "deduplication-aware" so it doesn't try to rehydrate the files during a copy? On a related note, is there still an issue with veeam backup files larger than 1TB and using Windows Server 2016 deduplication, or did a patch fix all that?

[foggy]

We looked at the backup copy job option, but the backup copy job option doesn't support rotated media pools.

They do, unless you enable GFS retention.

What else would be required to keep it "deduplication-aware" so it doesn't try to rehydrate the files during a copy?

Data will anyway be rehydrated, since you're copying it between different systems (you cannot copy deduplicated file from one Windows server to another, deduplication metadata is volume-specific).

[bjesteadt]

What if the deduplicated data would be simply copied from one drive to another drive on the same server?

From what I've read, Windows Server 2016 deduplicated drives are atomic units, meaning the metadata is stored on the drive itself, and can be dismounted and mounted on any Windows machine with the deduplication feature enabled. I can't say I've tried it, but there are claims that Windows Backup does not have to remove deduplication in order to copy files. I wasn't sure if Veeam had a similar dedup-aware functionality?

[Mike Resseler]

Hi Brian,

Let me shine some light on the deduplication engine in Windows Server 2016. One thing is absolutely correct. The metadata is stored in a hidden folder (The system volume information, administrator's best friend ) and if you dismount a volume and attach it to another windows server 2016 with dedup engine enabled, the data is readable again (Please note patch level because I had some bad experience when I forgot that at a certain moment )

For Windows backup, it is true that you can take an image level backup (VSS Snapshot) and that it remains "small" However, when you do recovery, dedup engine must be there again in order to read the files and recover items. When you don't do an image level backup, the situation is different. In that case, the files are going to be rehydrated first, moved to the backup storage and then it depends if dedupe is active there again or not.

Now the reason that this happens has nothing to do with Windows Backup itself but because there is a filter driver hidden underneath the file system. That filter driver works with VSS together (hence the reason why an image level backup can leave it as is).

hope it helps
Mike

[bjesteadt]

I understand what you're saying. Besides, the more I think about it, since Windows dedup doesn't kick in until a file is three days old, the original backup and the backup copy aren't modified until later anyway. So I suppose it is a bit of a non-issue.

Is the 1TB file size (or larger) still a concern for Windows 2016 dedup, or was that issue patched?

Are there any other concerns I should be aware of when using Windows 2016 dedup on Veeam repositories?

[Mike Resseler]

Brian,

The 1TB is not that much an issue anymore, but honestly, from my experience, don't go over 4 TB files. It will take too long for that engine to dedupe it.
For best practices: See here: https://bp.veeam.expert/architecture-ov ... uplication

(PS: It will tell you in that guide about the 1TB but it has mainly to do with server 2012 and its single processing. If your repository server has enough CPU and MEM it can do parallel processing so higher files sizes can be obtained)

[bjesteadt]

So it is no longer a matter of large files getting corrupted, but instead just a matter of there being enough time and resources to process the large file without interruption?

[Mike Resseler]

Indeed. And the biggest issue remains the supportability of Microsoft (hence why we state 1 TB). But as said, I succeeded in larger files, but that was only 1 FILE though. You need to look at the churn and calculate that you can do around 6 to 7 TB per 24 hours of dedup activity. Which is rather low if you have 2 larger files...

[bjesteadt]

How would I monitor the deduplication success (or failure) status to see if I am able to fit it all in a 24 hour cycle?

[Mike Resseler]

Unfortunately, that would be event viewer or PowerShell...