2012 R2 Dedupe for Veeam Servers Using iSCSI

[mafiasilk]

We are re-evaluating our core backup system for our customers.
And have decided we would benefit from 2012 R2 Dedupe, as our storage device doesnt dedupe at all.

So we are thinking of presenting iSCSI Targets on the 2012 R2 Server from a Deduped volume to the Veeam Servers as backup repositories.

The question is, has anyone had any decent dedupe form this scenario?
Would it look at all the iSCSI targets from all customers and be able to get any decent dedupe data from the Veeam backup jobs?

Our cusotmers are virtually all Windows based (little or no linux VM's)

Was wondering if it's actually worth the rebuild to achieve this or not.

[Gostev]

Windows Server deduplication is per volume (not truly global), and maximum size that I've seen happy users reporting is 10TB, with 4-5TB being more common. This is due to the fact that post-process deduplication needs to keep up with the new data, however its processing speed is quite limited (up to 100GB/hr I believe), and cannot be scaled due to the architecture.

[veremin]

Also, if you're going to use deduped volumes as primary backup repository, you should consider utilizing forward incremental mode with active full, as you won't get any decent deduplication ratio with reversed incremental mode, and synthetic full on dedupled volume is likely to take pretty long time. Thanks.

[mafiasilk]

Hi Guys, Thanks for the input, I set the backup job compression to Dedupe Friendly, and have seen the backup size increase to almost 110% from previous sizes. Will the dedupe settings in 2012 offset this increase and make it viable? or shall I change the compression back to normal?

[foggy]

Only actual test can give a correct answer, but most likely it should (this is the recommended compression setting for dedupe repository).

[Yuki]

I would be very careful about using Win2012 dedupe volume for backup storage. Until you start using it - you can't tell how well it will play in your environmnet. In ours - we found out that we are better off with it turned off. A few reasons for it:
1 - dedupe is a job that runs on a regular basis and it takes time to deduplicate new data
2 - not knowing how fast this process will go and how much data will be deduplicated makes planning somewhat hard
3 - data deleted from the volume won't free up the space right away, but until garbage collection runs. This process also takes time (we had it run for 12 hours before seeing results we expected) and this was a huge pain for u
4 - dedupe does slow down the system due to background processes and extra disk load


Our main deduped server is 16TB and we are moving now to non-deduped 36TB due to reasons above. This is in a company with a lot of data changing on daily basis and large Veeam backups + delays in deduped volumes result in hard to manage and plan storage capacities.

[ryan1212]

One thing to note- performance on 2012R2 does seem to be quite a bit better than it was on 2012.

[mafiasilk]

We are sticking with Dedupe on 2012 R2, but are having a couple of issues with the GFS on the same server using Backup Copy Jobs.

We are running our archive on the same server into an Archive folder, but would like to manage it within the Veeam console, but cannot as this would require a new write of data with the job.
Se what we are doing is manually moving the oldest .VBK file with a powershell script into the archive folder.
The script checks the date of the oldest file in the Backups folder.

Using this method, we are not making the server dedupe the data again, and saving IOPS by not writing new data.

Is this the best way?
Can I put in a feature request for Veeam to have the ability to do this so it can be tracked in the Veeam console?

Does anyone else do this type of thing?

[Vitaliy S.]

Hi Asif,

Do you mean an ability to copy backup files to another folder and then populating it with the backup console?

Thanks!

[mafiasilk]

Sort of, Im looking for a method that will not impact IOPS, but allows GFS type retention.
So Im currently MOVING not copying VBK files from the Backup Directory to the Archive Directory, this also has the benefit of remaining DEDUPED in Server 2012.
The disadvantage is: No Veeam tracking, as the script is in powershell and not inside Veeam.

How can I have my cake and eat it! Maybe a feature request to veeam?

[Vitaliy S.]

We are running our archive on the same server into an Archive folder, but would like to manage it within the Veeam console, but cannot as this would require a new write of data with the job.

If your backup repository and archive repository is on the same server, then it looks like you put all eggs in one basket. In other words, If you lose this server, you lose all your backups even archived ones.

I would suggest using completely different storage/server for backup copy jobs. And if you do this, you wouldn't need to manually move your VBK files to the archive folder.

As to your current strategy, it appears that the only way to achieve this is to use a PS script that would move VBK file using Windows Task Scheduler and then trigger archive repository re-scan operation to import these VBK files to the backup console. Of course, it is not ideal, but I think it should give you what you're after.

P.S. regarding your last question in another thread - your backup copy job will fail if it fails to find the VBK file.

Here you go a bit more details on this (this post talks about rotated drives, but it workarounds the same issue of missing VBK file)

Hope this helps!