Deduplication Single Server

[scottf99]

Hi. I have a Backup Exec background so please excuse if I misunderstand Veeams Deduplication.

I have a client with about 30 Vmware Guests and 2 Hosts. Using Veeam 6.5 I backup 19 of these guests in one job and it all works fine.
The 20th Server has about 4TB of flat file data. I back this up with a single job.
With Backup Exec if I backup this Server to a dedup location it will take up 4TB, same as with VeeamB&R.
If the next day I do another Full backup with Backup Exec it will not take up much more space on the dedup location (as it actually dedupes the flat file data). If i do the same with Veeam it create a whole new 4TB file. This is causing problems as I only have 10TB of space for the Veeam backups and client wants to keep 6 months of backups. Even if I only do a full backup every 2 months i am still screwed. I have looked into Reverse and Syntethetic but it creates a lot of CPU usage and the NAS repository is overwhelmed with IO.

Am I doing something wrong or do I just need to look at how Veeam deduplicates a single Server (I understand and see Veeam deduplication working with 2 or more similar Servers).

Cheers
Scott

[veremin]

If the next day I do another Full backup with Backup Exec it will not take up much more space on the dedup location

I’m wondering what dedup location you’re talking about. Is it an appliance that is capable of deduplication or it’s more about Backup Exec deduplication option? Thanks.

[scottf99]

Thanks for your reply.
The reference to BE was just to show what I am used to.
Veeam is backing up to a NAS through iscsi. The LUN appears as a 15TB NTFS formatted partition on the Veeam Server/Proxy.
Veeam Server/Proxy is Server 2008R2 as are the guests being backed up.

I guess I could simplify my question by asking "where is Veeam deduplication?" It is advertised as being a major feature but if I do 2x 4TB Full backups it takes up 8TB backup space. That is not deduplication???

Cheers
Scott

[veremin]

Hi, Scott. I believe there must have been a slight misunderstanding on how VB&R deduplication works. In fact, deduplication doesn’t work between the VBK and VIB files; instead, it works only within each VBK and VIB.

Other things related to deduplication are pretty good covered in sticky FAQ; so, it might be worth taking a look.

Thanks.

[scottf99]

Thanks again. I am sure it is my misunderstanding, I just wanted confirmation. I have read all the guides and FAQs.

My understanding of deduplication is when a block (byte, bit, whatever) is written to a backup then it is not written again, just a link or pointer to it is used. I hope this is correct.

When Veeam does a full backup and then another identical full backup does it not do this? Does it just create another backup file?

If I now understand correctly then my question is ... why? Why not just write to the same file and actually deduplicate the data. What is happening is a duplication.

Apologies if I am wrong.

With all the terrible problems with BE lately Veeam seems to be the way to turn (and I have purchased it for one client so far) but this is a major failing and I was genuinely hoping I was just doing something wrong.

Cheers
Scott

[veremin]

As mentioned, with forward incremental mode deduplication exists only on one restore point level. So, there is no deduplication between previous restore points and newly-created one. In other words, there will be a deduplication between VMs within one restore point, meanwhile, there will be none between restore points themselves.

Your understanding (though, being not 100% true) is related mostly to reversed incremental mode, where the recent restore point is always a full backup that gets updated after every backup cycle. With reversed incremental changes (new blocks) are injected into the .vbk file to rebuild it to the most recent state of a VM and the blocks that have been already present (latest restore point - .vbk file) are skipped.

However, even with reversed incremental if you decided to perform a new active full backup, there would be deduplication only across VMs that are being backed up, not across existing restore points.

Thanks.

[Gostev]

Use reversed incremental backup mode (recommended backup mode for disk-based targets). This mode is designed to maintain a single full backup on disk no matter of retention policy. This is just a different approach to the same problem (avoiding storing multiple full backups on disk). Symantec uses target-side dedupe to only store full backup data once, while we use reversed incremental backup mode instead. You will find that our approach is much more efficient, because it does not require periodic fulls, which can take a very long time. It is not uncommon that full backup takes a few days in some environments, so periodic fulls are simply impossible there.

[scottf99]

Thank you both for your explanations and for the solution. I will give reverse incrementals a go again.

Cheers
Scott

[veremin]

The best method, indeed, in terms of storage saving. Should any additional help be needed, don’t hesitate to let us know. Thanks.

[benthomas]

Another option would be to upgrade your backup server to Windows 2012 and install the data deduplication role on it. Then you could enable Deduplication on the iSCSI drive and have it run a deduplication run every night and it wouldn't matter how many full backups you did with Veeam, because Windows would detect the duplicate blocks and reduce the required space.

[veremin]

Another option would be to upgrade your backup server to Windows 2012 and install the data deduplication role on it.

Though, in this case the recommended backup method would be forward incremental, instead of, reversed one, since the forward incremental is supposed to guarantee higher deduplication rates. Moreover, if you’re after using restore and/or surebackup options, be aware to set OS to dedupe only files that are older than a specified interval.

As to other users’ experience regarding MS 2012 Deduplication, it’s pretty good covered here.

Thanks.