Discussions specific to the VMware vSphere hypervisor
Post Reply
pshute
Expert
Posts: 209
Liked: 13 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Is deduplication working?

Post by pshute » Jul 03, 2018 11:24 pm

Among the many virtual machines we back up, we have one that is being built to replace an older one. Initially it was nearly empty, and not adding much to the backup. But now that we've started copying files from the old server, the total backup size is growing.

If these files are copies of files on the old server that's also being backed up at the same time, shouldn't deduplication make sure they're only backed up once? If so, why is the total size growing?

I'm not certain this is the reason for the growth, it may be a coincidence. How can I check?

P.Tide
Product Manager
Posts: 5307
Liked: 466 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Is deduplication working?

Post by P.Tide » Jul 04, 2018 2:34 pm

Hi,

Do you use per-VM chains?

Thanks

Vitaliy S.
Product Manager
Posts: 23062
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Is deduplication working?

Post by Vitaliy S. » Jul 04, 2018 4:11 pm

Additionally, keep in mind that deduplication is done within the backup file (VBK, VIB or VRB). So if you had some files stored in the previous restore point, then the resulting (new) backup file would not have a reference to that old data.

pshute
Expert
Posts: 209
Liked: 13 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Re: Is deduplication working?

Post by pshute » Jul 05, 2018 1:09 am

PTide wrote:Do you use per-VM chains?
No. I now suspect that it was just a coincidence. Can you please confirm that copying files between several servers that are included in a backup should result in no increase in the size of the next backup. Deduplication should find all the copies? Is there any limit to the number of copies of a file that will be deduplicated?

Is there any way of examining the effects of deduplication on the amount of space each server contributes to the total size of the backup? In the log I can see Size, Read and Transferred for each machine, but I only see an overall Dedup and Compression total.

P.Tide
Product Manager
Posts: 5307
Liked: 466 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Is deduplication working?

Post by P.Tide » Jul 05, 2018 12:23 pm

I now suspect that it was just a coincidence.
Would you elaborate on what exactly was a coincidence, please?
Can you please confirm that copying files between several servers that are included in a backup should result in no increase in the size of the next backup. Deduplication should find all the copies? Is there any limit to the number of copies of a file that will be deduplicated?
Veeam deduplication works on VM-disk level, not on in-guest file-level. It is based on identifying duplicate blocks inside a single VM disk or across multiple VMs inside the same job. This is primarily beneficial when VMs are deployed from the same template since the base image is identical, but it's less useful for incremental data.
Is there any way of examining the effects of deduplication on the amount of space each server contributes to the total size of the backup?
Unfortunately, there is no way to get that info.

Thanks

pshute
Expert
Posts: 209
Liked: 13 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Re: Is deduplication working?

Post by pshute » Jul 05, 2018 10:24 pm

PTide wrote:Would you elaborate on what exactly was a coincidence, please?
The possible coincidence is the fact that the backup size increased after some files were copied from one server to another. But I don't know for sure.
Veeam deduplication works on VM-disk level, not on in-guest file-level. It is based on identifying duplicate blocks inside a single VM disk or across multiple VMs inside the same job. This is primarily beneficial when VMs are deployed from the same template since the base image is identical, but it's less useful for incremental data.
Does that mean that, say, a 1GB video file that a user uploads to one server, then copies to another server, will not be deduped? I would have thought that a file's data doesn't share blocks with other files, and that so long as the block sizes of each copy are the same, there should be identical blocks that can be deduped. Is that not correct?

Regnor
Service Provider
Posts: 331
Liked: 66 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Is deduplication working?

Post by Regnor » Jul 06, 2018 5:13 am

Vitaliy S. wrote:Additionally, keep in mind that deduplication is done within the backup file (VBK, VIB or VRB). So if you had some files stored in the previous restore point, then the resulting (new) backup file would not have a reference to that old data.
Like Vitaliy said; deduplication works only in a single restore point. So if you run an incremental job, it will grow because only your full backup contains all the known blocks from your old server.
In that case you'll notice deduplication only when the next full backup runs.

pshute
Expert
Posts: 209
Liked: 13 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Re: Is deduplication working?

Post by pshute » Jul 06, 2018 5:21 am

I was only comparing the weekly full backup sizes.

Regnor
Service Provider
Posts: 331
Liked: 66 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Is deduplication working?

Post by Regnor » Jul 06, 2018 10:26 am 2 people like this post

In Full backups you should see the deduplication increasing; it won't be 100% but at least some part of your duplicate files/blocks should be deduplicated.

Post Reply

Who is online

Users browsing this forum: No registered users and 26 guests