Is deduplication working?

[pshute]

Among the many virtual machines we back up, we have one that is being built to replace an older one. Initially it was nearly empty, and not adding much to the backup. But now that we've started copying files from the old server, the total backup size is growing.

If these files are copies of files on the old server that's also being backed up at the same time, shouldn't deduplication make sure they're only backed up once? If so, why is the total size growing?

I'm not certain this is the reason for the growth, it may be a coincidence. How can I check?

[PTide]

Hi,

Do you use per-VM chains?

Thanks

[Vitaliy S.]

Additionally, keep in mind that deduplication is done within the backup file (VBK, VIB or VRB). So if you had some files stored in the previous restore point, then the resulting (new) backup file would not have a reference to that old data.

[pshute]

Do you use per-VM chains?

No. I now suspect that it was just a coincidence. Can you please confirm that copying files between several servers that are included in a backup should result in no increase in the size of the next backup. Deduplication should find all the copies? Is there any limit to the number of copies of a file that will be deduplicated?

Is there any way of examining the effects of deduplication on the amount of space each server contributes to the total size of the backup? In the log I can see Size, Read and Transferred for each machine, but I only see an overall Dedup and Compression total.

[PTide]

I now suspect that it was just a coincidence.

Would you elaborate on what exactly was a coincidence, please?

Can you please confirm that copying files between several servers that are included in a backup should result in no increase in the size of the next backup. Deduplication should find all the copies? Is there any limit to the number of copies of a file that will be deduplicated?

Veeam deduplication works on VM-disk level, not on in-guest file-level. It is based on identifying duplicate blocks inside a single VM disk or across multiple VMs inside the same job. This is primarily beneficial when VMs are deployed from the same template since the base image is identical, but it's less useful for incremental data.

Is there any way of examining the effects of deduplication on the amount of space each server contributes to the total size of the backup?

Unfortunately, there is no way to get that info.

Thanks

[pshute]

Would you elaborate on what exactly was a coincidence, please?

The possible coincidence is the fact that the backup size increased after some files were copied from one server to another. But I don't know for sure.

Veeam deduplication works on VM-disk level, not on in-guest file-level. It is based on identifying duplicate blocks inside a single VM disk or across multiple VMs inside the same job. This is primarily beneficial when VMs are deployed from the same template since the base image is identical, but it's less useful for incremental data.

Does that mean that, say, a 1GB video file that a user uploads to one server, then copies to another server, will not be deduped? I would have thought that a file's data doesn't share blocks with other files, and that so long as the block sizes of each copy are the same, there should be identical blocks that can be deduped. Is that not correct?

[Regnor]

Additionally, keep in mind that deduplication is done within the backup file (VBK, VIB or VRB). So if you had some files stored in the previous restore point, then the resulting (new) backup file would not have a reference to that old data.

Like Vitaliy said; deduplication works only in a single restore point. So if you run an incremental job, it will grow because only your full backup contains all the known blocks from your old server.
In that case you'll notice deduplication only when the next full backup runs.

[pshute]

I was only comparing the weekly full backup sizes.

[Regnor]

In Full backups you should see the deduplication increasing; it won't be 100% but at least some part of your duplicate files/blocks should be deduplicated.