How to handle large vmdk files?

[ryan1212]

Tyler and Jason,

I have one VM with extremely large vmdk files. These are connected to a virtual appliance which is a "black box" as far as we are concerned because we do not have access to log in\configure it. It has two 1.6TB VMDK files, which are spanned using windows dynamic disks so we cannot split this into two separate jobs.

We would like to be able to hold 4-6 weeks of data for the system, however windows server 2012 de-duplication does not dedup files larger than 1TB, so four weeks of backups on this system will use 12TB of disk space with forward incremental.

Switching to reverse incremental would alleviate some of this issue, but the creation time seems like it would be very long and it would be riskier than I would like.

This job takes about 28hours to run an active full, so that is not an option to do on a weekly basis either. It hosts video and several hundred gb\day are uploaded so the vmware snapshot grows quite large.

Is anyone else using SATA disks with server 2012 deduplication and large backup jobs?

[Vitaliy S.]

Hi Ryan,

Switching to reverse incremental would alleviate some of this issue, but the creation time seems like it would be very long and it would be riskier than I would like. This job takes about 28hours to run an active full, so that is not an option to do on a weekly basis either.

You can switch to reverse incremental backup mode right after forward incremental job run creates another full.

Btw, we've got an existing topic about Windows dedupe 2012 best practices, please take a look, might be useful: Best Practice for MS Server 2012 DeDup Repo

Thanks!

[ryan1212]

Thanks for the reply. I am using the dedup friendly option, but MS documentation (and my results verify) it will not deduplicate large files: http://technet.microsoft.com/en-us/libr ... 31700.aspx "Not good candidates for deduplication: Files approaching or larger than, 1 TB in size"

What mode are others using for large jobs+2012 deduplication?

[Vitaliy S.]

What mode are others using for large jobs+2012 deduplication?

I believe the answer to this question is in the topic I've referenced above, please check it out

[ryan1212]

What am I missing when I look at that thread?

I find this:
"You must use forward incremental, and disable Veeam compression to get the best results."
and:
"First, assuming you are using Veeam compression with reverse incremental, you are unlikely to see much savings for your effort. First, unless you configure the Windows 2012 dedupe to process files that are "0 days" old, it will never actually dedupe the VBK since it's being changed every night, but if you do this, and happen to get good dedupe, you will likely see significant performance degradation of your backups."

So- what is the best way to handle the situation? Forward incremental (as recommended) would take 18TB+ for 6 weeks retention since they cannot be deduplicated and reverse incrementals would take a very long time to create due to the size of the system.

Why is it- reverse incremental can have unlimited roll back points, but forward incremental can only have 7 incremental files before needing a synthetic full created? Would differential backups make this less of an issue?

[Vitaliy S.]

Why is it- reverse incremental can have unlimited roll back points, but forward incremental can only have 7 incremental files before needing a synthetic full created?

It's not exactly like this, you can configure your backup job not to run synthetic fulls at all and schedule monthly active fulls instead, but there are best practices on how often full backups should be run.

Also keep in mind that the most recent VM restore point in case of forward incremental backup mode will the latest VIB file, so you will need the entire backup chain VBK + multiple VIBs to perform the restore operation. The more files you have in the backup chain the more risky it might be.

As to the reverse incremental backup mode, then since your VBK file contains the most recent VM data, then you will only need this file to start the restore process.

So- what is the best way to handle the situation? Forward incremental (as recommended) would take 18TB+ for 6 weeks retention since they cannot be deduplicated and reverse incrementals would take a very long time to create due to the size of the system.

How large are your VIB files?

[ryan1212]

On those systems the .vib files are about 5-10GB/day. I will try switching to reverse incremental and will know how it performs in a few days.

[Vitaliy S.]

Just to make sure I'm on the same page with you - since your increments are only 5-10 GB, Windows 2012 dedupe can process it pretty easy (given that it can do 100GB of new data per hour) without any issues, however full backups might be quite challenging. Is this what you're mostly concerned about?

[ryan1212]

Exactly. Each synthetic\active full is 3TB. 6 weeks of these is 18TB. the vib files dedup well and are not an issue.

Thanks