Encryption and Deduplication with mix of NTFS, ReFS and Object Storage.

[ConradGoodman]

I am currently writing a scope to start using Object Storage in S3 with SOBR as a 3rd copy of our backup data.

Currently we have 4 B+R servers with Local Windows Storage in each site.

2 of these B+R servers are running new hardware, with 40TB of ReFS in one volume, Server 2019.

File system Dedupe:

Server 1 W2019 REFS:

Capacity 40TB, Free Space 37TB, Deduplication rate: 16% - Stores Windows Agent SQL backups and vSphere VM backups

Server 2 W2019 REFS:

Capacity 40TB, Free Space 33TB, Deduplication rate: 37% - Stores Windows Agent SQL backups and vSphere VM backups


2 are running older hardware, Server 2012, 2 x NTFS volumes of 15TB and 7TB.

Server 3 W2012R2 NTFS:

Disk 1 Capacity: 15TB, Free Space: 9.9TB, dedudplication rate: 1% - Stores Windows Agent based SQL backups
Disk 2 Capacity: 5TB, Free Space: 2TB, deduplication rate: 8% - Stores vSphere VM backups.

Server 4 W2012R2 NTFS:

Disk 1 Capacity: 15TB, Free Space: 12.4TB, dedudplication rate: 27% - Stores Windows Agent based SQL backups
Disk 2 Capacity: 5TB, Free Space: 3.81TB, deduplication rate: 37% - Stores vSphere VM backups.


Each site has very similar jobs.

CentOS VM Backups
Windows VM Backups
Other Linux VM Backups
Domain Controller VM backups
Windows Agent SQL Backups

All the above jobs use active weekly fulls with 7 incrementals retained.

A daily copy job from Server 1< - > Server 3
A daily copy job from Server 2< -> Server 4

Copy all VMs in both directions, with a full and 2 incrementals retained.

The next stage is to move the copy jobs into a SOBR with Amazon S3 object storage, backup files must be encrypted at rest.

I understand I can encrypt the bucket and rotate the keys there, but our security team will prefer the backup files are encrypted by veeam.

This presents challenges and a number of questions.

1) What will be the impact on file system deduplication of enabling encryption on the backup copy jobs? I have read lots of conflicting advice on these forums.
1.1) Is it true that ReFS will provide the same deduplication with encrypted VBKs but NTFS will never see the same blocks?
2) Why is server 3 seeing such poor file system deduplication.

[ConradGoodman]

3) Does this have any effect on blocks stored in S3, and the deduplication function of object storage?

[foggy]

1) Encryption has no impact on ReFS space savings since Veeam B&R tracks identical blocks based on their raw content and just tells ReFS to clone this or that block. ReFS itself has no idea of what's inside. This is not the case with deduplication of course.
2) Hard to say, could be compression, for example. Check the job (compression) and repository (decompression) settings.
3) Yes since blocks are copied to object storage as they are stored on the source, in an encrypted format.

Consider also Capacity Tier encryption as an option - it was specifically designed for the case when the local backups are not encrypted. Here's another existing discussion that is worth reviewing.

[ConradGoodman]

thanks foggy.

Regarding 3. can S3 object storage still reference the same blocks for example for GFS fulls, or do we lose this functionality and use a lot more storage?

Thats my biggest concern really, that the archive GFS fulls are not loads of encrypted copies of all the same data.

[veremin]

Capacity Tier works in forever-incremental fashion, meaning only new data gets offloaded to object storage. It doesn't matter what type of source restore point gets transferred (full, incremental, GFS, etc.) - only blocks that have not been transferred before are subject to offload.

Think about our ReFS-integration - the same concept here.

Thanks!