Dedupe 1,0x Windows Server 2012 R2

[td007]

Hi
We operate a virtualized file server based on Windows 2012 R2.
Vmware ESXi 5.1.
The capacity of the data partition is 2.0 TB.
Deduplication is turned on and the deduplication ratio is 48% (savings 784GB).

This virtual server is with VEEAM Vers 7.0.0.871 (installed on Windows Server 2012 R2) backed up daily.

The stats:
Backup size 1.8 TB
Dedupe 1.0 x
Compression 1.1 x

Backup mode: Reversed incremental
Enable inline deduplication data is turned on
Compression: Optimal
Storage optimizations: WAN target
Enable application-aware image processing is turned on

The actual amount of data is about 1.4 TB.
Backup size is 1.8 TB.
Dedupe 1.0 x

Why does not the deduplication?
Maybe someone has a tip for me?

Thank you

[Gostev]

Are you saying you have multiple copies of the same files stored on that file server?

[martynuk]

All of our Veeam incremental backups show 1.0x dedupe. Only fulls show a sensible dedupe figure.

[Gostev]

That's right, all incremental data is unique by definition, because we do source-side dedupe before copying the incremental data to the repository.

Deduplication in full backup will normally require multiple VMs in the same job, otherwise there is really nothing to dedupe with. When you are backing up a single VM, you will only see dedupe if it stores multiple copies of the same files, otherwise (again) there is nothing to dedupe.

[td007]

Yes, I have many of the same files in the virtual machine.
Windows itself in the virtual machine can deduplicate 48%.

Only VEEAM can not deduplicate, on the contrary, the backup is larger than the original!

[Gostev]

Windows itself in the virtual machine can deduplicate 48%.

Well, this changes everything. If you already have Windows dedupe running inside the backed up VM, then all VM data is already deduplicated and compressed to the maximum possible amount, so obviously no further data reduction will be possible on top of that. Windows dedupe is very efficient, as you can see from the stats Veeam can only add some more compression on top of it, but nothing more.

As for backups being larger, this could be due to large amount of dirty blocks in NTFS.

[foggy]

Also, please note that Windows deduplication seriously affects the number of changed blocks inside VM and Veeam B&R has to copy all of them, so you can expect large increments for this VM (much larger than the estimated data change rate inside guest OS).

[td007]

The data size within the VM is about 1.4 TB without deduplication.
With deduplication, the data size is about 800 GB.

The backup size of VEEAM for this VM is 1.8 TB.

So one can say that the combination of Windows Server 2012 as a VM with enabled Deduplication and Veeam is not optimal, at least for our constellation.

[foggy]

What is the size of VMDK on VMFS storage?

[td007]

no problem for large increments for this VM.
Transferred 3,4 GB
Duration: 0:11:18

Screenshot

many thanks for your interest

[td007]

The size of VMDK ist 1,82 TB + 200GB (2x VMDKs)

[dellock6]

This number is perfectly in line with all that has been said in this thread. This is the same size of the Veeam backup, and the reason Veeam is saving all the disk without any reduction is because as Alexander said, deduplication changes so many blocks when it starts its activities, that can be almost all blocks of the VMDK has been touched and "dirty", and so for Veeam (remember, we run image-based backup, we do not do file-level) they are all to be saved.
When saving win2012 deduped volume is a good choice to disable Veeam deduplication of that VM...

Luca.

[td007]

Thanks to all.

Now it's clear.
I tought it to me already but unfortunately I found so far no detailed information on it.

greeting
Tiemo

[Butha]

Hi,

I believe an earlier post was answered where Veeam 7.0 on Server 2012 R2 (With Dedupe on windows Volume enabled) always produced a ratio of 1.0x, and I have the exact same issue, except I am backing up to SAN space on a 2008 R2 server - with no dedupe enabled on the OS.

Inline dedupe is enabled on the jobs - many of the jobs include similar servers (grouped for instance by OS) - compression is set to optimal and target " local target". A good friend of mine always show off with his high dedupe ratios - but since using Veeam (3 years?) I have always only seen a dedupe ratio of 1.0x on All increments, and 1.1x on the Full .vrb file.

Using reverse increment backups.

As a test I setup a test veeam BR server with a 2012 server with dedupe enabled target - and the OS was able to get high dedupe ratios (and of course then Dedupe disabled in Veeam) - so the data is definitely not the issue - some are for instance grouped file servers where big amounts of data that are in fact duplicate .iso images etc..

Would appreciate some feedback with regards to dedupe ratios etc that other people are getting.

Regards

B

[Vitaliy S.]

Hello,

Actually I believe it is expected to see these numbers on increments. Check out the explanation from Anton above, this should be applicable for both Windows Server 2012 and 2008.

Thank you!

[Butha]

Good Morning Vitaliy,

I see my post was merged - I believe the circumstances is very different though.

Td007 (And Anton's reply) applies to Server 2012 - where dedupe is enabled natively in the OS. My issue is the opposite. I have 2008, and dedupe is not enable (not available) natively in the os - so don't understand that there is no dedupe found.

Would appreciate some feedback.

Regards

B

[Vitaliy S.]

Anton was referring that all incremental backups are unique be definition, nothing to dedupe, it doesn't matter where you store them.

[Butha]

Hi Vitaliy,

I'm not too concerned about the increments being 1.0x - that part is 100% (as Anton rightly pointed out and explained) - my query is more towards the fact that the full file is 1.1x only. Should that not be higher especially if a lot of similar duplicate files are stored in the vm, or vm's that are part of the same job?

B

[foggy]

Do you have Windows dedupe enabled inside the backed up VMs?

[Vitaliy S.]

Do you have an active full backup available in the backup chain to compare the dedupe numbers with VBK created by reversed incremental backup?

[Gostev]

Perhaps most data is simply unique in your VMs, and the only thing getting deduped are large OS files (remember that Veeam users pretty large block sizes with dedupe for performance reasons), thus giving 10% dedupe.

Big dedupe numbers is normally observed when backing up a bunch of VMs made from the same template (this part is very important), with very few apps installed (so, most of the VM size comes from the Windows OS files).

Thanks!

[Butha]

Just a quick update on this that might be useful to others:

I was looking at a previous issue around when a VM was removed from the job - the full active copy never gets smaller when using reverse increment (all the normal "retention" options are set to remove the vm from the job etc) and came across this post - (apologies for posting a http link - it is written by Luca Dell'Oca who is well respected at Veeam i'm sure)

http://www.virtualtothecore.com/en/veea ... cremental/

At this point it might seem off topic - but please read on - it does relate to the dedupe and compression topic.

My Veeam installation is from a original version 6.0 install and was upgraded to 6.5 and then 7 - and we have in 2 years never performed "reverse incremental maintenance" as indicated by the article. I don't remember reading this anywhere before...

So the short summary:

After running some active full jobs where previously some large VM's were included, but now removed, the base file reduce drastically in size (one example the base backup file was 7.5TB - and now reduced to 1TB) - and both dedupe and especially compression ratios are vastly improved.

Not sure if the old increment chains going back a year or more might have caused the low ratios - or perhaps the algorithm for calculating is not working because of the pointers in the chain to old vm's and data that is no longer present? (almost like it's trying to dedupe or compress data that " used" to exist?)

I'm going through the jobs one by one and running full actives (does take a while!) and will report back after a few days.

Some good examples perhaps might help explain:

Have an exchange backup job that used to include: Exchange CAS server + 1 Mailbox server. The backup size was about 2TB before (CAS - 250GB and the Mailbox server the rest) - and the Mailbox server is now removed. As mentioned the base file never reduced, and the active full now removed the old mailbox server data. So left with only the 250GB CAS server - and it dedupes to 2.0x and compression is 5x - yes a 25.5GB base file from a 250GB source vm. (CAS Server OS - 2008 R2).

Anton - any official recommendation for maintenance tasks on " reverse increment" setups (like Luca's link - every 2nd month?)

Regards,

Butha

[veremin]

Any official recommendation for maintenance tasks on " reverse increment" setups (like Luca's link - every 2nd month?)

You can consider Luca's statement official enough, as he now works in our company. As to other sources of information, you can use these best practices once published in our community forum. Thanks.