Best Practice for MS Server 2012 DeDup Repo

Availability for the Always-On Enterprise

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby Delo123 » Mon Sep 12, 2016 7:35 am

If I understood correctly, Windows 2016 will even be worse for 1TB+ files. Currently 1TB+ files are being deduped but it takes quite long which isn't that much of an issue anymore due to Veeam's scale out repositories. As incrementals now can go to different volumes the dedupe job on the fulls doesn't have to be interrupted. However in Server 2016 i read that windows dedupe will only dedupe the first TB so for a 8TB file, 1TB will be deduped, 7 will not :( That's why will still hope Veeam will come with an option to spilt vbk's in smaller chunks...
Delo123
Expert
 
Posts: 340
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby lightsout » Mon Sep 12, 2016 1:42 pm

I've not heard that, but I guess we will find out!
lightsout
Expert
 
Posts: 183
Liked: 47 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby Gostev » Mon Sep 12, 2016 4:57 pm 1 person likes this post

Delo123 wrote:However in Server 2016 i read that windows dedupe will only dedupe the first TB so for a 8TB file

First almost 4TB actually, although we did not re-test on RTM yet. That 1TB is the true limit for modified files, however backup files are never touched once created (at least with the recommended job settings for deduplicating storage).
Gostev
Veeam Software
 
Posts: 21354
Liked: 2333 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby Seve CH » Tue Sep 27, 2016 12:47 pm

Hi

I have not read the documentation for W2016, but dedup seems not to be supported on W2012R2 with volumes larger than 64TB. It is a VSS issue rather than a dedup problem. Windows will let you activate deduplication and won't warn you, but it will produce all kind of weird errors, mostly of the kind of "The parameter is incorrect."

https://support.microsoft.com/en-us/kb/2967756

Regards.
Seve CH
Influencer
 
Posts: 10
Liked: 6 times
Joined: Mon May 09, 2016 2:34 pm
Location: Switzerland
Full Name: JM Severino

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby Delo123 » Wed Sep 28, 2016 6:32 am

Hi Steve, that is correct :) for that reason most of create a storage pool and carve multiple thin volumes from that each just under 64TB. Backups and even better scale out repositories splitting up fulls and incrementals can de distributed among these giving us multiple dedupe threads etc. :)
Delo123
Expert
 
Posts: 340
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby cheese » Wed Feb 08, 2017 1:18 pm

Hello

Just to save someone else from any pain when formatting very large volumes.

You can use the /q switch combined with /L

format g: /A:64k /q returns (as expecteded):

C:\Users\administrator>fsutil fsinfo ntfsinfo g:
NTFS Volume Serial Number : 0x0422992622991e2c
NTFS Version : 3.1
LFS Version : 2.0
Number Sectors : 0x0000001b491defff
Total Clusters : 0x0000000036923bdf
Free Clusters : 0x0000000036922a0e
Total Reserved : 0x0000000000000040
Bytes Per Sector : 512
Bytes Per Physical Sector : 4096
Bytes Per Cluster : 65536
Bytes Per FileRecord Segment : 1024
Clusters Per FileRecord Segment : 0
Mft Valid Data Length : 0x0000000000010000
Mft Start Lcn : 0x000000000000c000
Mft2 Start Lcn : 0x0000000000000001
Mft Zone Start : 0x000000000000c000
Mft Zone End : 0x000000000000cca0
Max Device Trim Extent Count : 0
Max Device Trim Byte Count : 0x0
Max Volume Trim Extent Count : 62
Max Volume Trim Byte Count : 0x40000000
Resource Manager Identifier : A4A80A8E-ED80-11E6-90E7-00259069050F

However using the /q switch as well:

C:\Users\administrator>format g: /A:64k /L /q
The type of the file system is NTFS.
Enter current volume label for drive G: Veeam Extent 2

WARNING, ALL DATA ON NON-REMOVABLE DISK
DRIVE G: WILL BE LOST!
Proceed with Format (Y/N)? y
QuickFormatting 54.6 TB
Volume label (32 characters, ENTER for none)? Veeam Extent 2
Creating file system structures.
Format complete.
54.6 TB total disk space.
54.6 TB are available.

C:\Users\administrator>fsutil fsinfo ntfsinfo g:
NTFS Volume Serial Number : 0x98c4dbf1c4dbcf9c
NTFS Version : 3.1
LFS Version : 2.0
Number Sectors : 0x0000001b491defff
Total Clusters : 0x0000000036923bdf
Free Clusters : 0x0000000036922a01
Total Reserved : 0x0000000000000040
Bytes Per Sector : 512
Bytes Per Physical Sector : 4096
Bytes Per Cluster : 65536
Bytes Per FileRecord Segment : 4096
Clusters Per FileRecord Segment : 0
Mft Valid Data Length : 0x0000000000100000
Mft Start Lcn : 0x000000000000c000
Mft2 Start Lcn : 0x0000000000000001
Mft Zone Start : 0x000000000000c000
Mft Zone End : 0x000000000000cca0
Max Device Trim Extent Count : 0
Max Device Trim Byte Count : 0x0
Max Volume Trim Extent Count : 62
Max Volume Trim Byte Count : 0x40000000
Resource Manager Identifier : A4A80A99-ED80-11E6-90E7-00259069050F


That's saved my several days formatting 2x 54TB drives - hope someone finds it helpful.
cheese
Lurker
 
Posts: 1
Liked: never
Joined: Wed Feb 08, 2017 9:04 am
Full Name: Dave

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby albertwt » Sun May 07, 2017 10:51 pm

cheese wrote:Bytes Per FileRecord Segment : 4096


Hi,

Does the Bytes Per FileRecord Segment : 1024 must be 4096 ?

I usually do it using the below PowerShell script:
Code: Select all
function FormatDisk([string]$driveletter, [string]$drivelabel)
{
    Format-Volume   `
    -DriveLetter $driveletter `
    -NewFileSystemLabel $drivelabel `
    -FileSystem NTFS `
    -AllocationUnitSize 65536 –Force -Confirm:$false `
    -UseLargeFRS
}

FormatDisk -driveletter D -drivelabel "DATABASE"


That's the best practice for SQL Server Database drive which I usually use.
--
/* Veeam software enthusiast user & supporter ! */
albertwt
Expert
 
Posts: 598
Liked: 19 times
Joined: Thu Nov 05, 2009 12:24 pm
Location: Sydney, NSW

[MERGED] Enabling Deduplication on Veeam Backup Repo NTFS ?

Veeam Logoby albertwt » Mon May 15, 2017 10:31 am

Hi All,

I'm about to turn on Windows Server 2012 R2 deduplication on the Veeam Backup server 2x iSCSI NTFS LUNs (each is 40 TB in size) so that I can run the deduplication job during the business hours, while after hours, the Veeam Backup job is running.

Does this issue with .VBK larger than 1 TB as described in: https://www.veeam.com/kb2023 still persist or it has been fixed in Veeam Backup & Replication 9.5 Update 2 ?

Any kind of help and suggestion would be greatly appreciated.

Thanks,
--
/* Veeam software enthusiast user & supporter ! */
albertwt
Expert
 
Posts: 598
Liked: 19 times
Joined: Thu Nov 05, 2009 12:24 pm
Location: Sydney, NSW

Re: Enabling Deduplication on Veeam Backup Repo NTFS ?

Veeam Logoby vmniels » Mon May 15, 2017 10:37 am 1 person likes this post

The 1TB limit is related to Windows 2012 R2 and not really to Veeam. It is advised to use per VM chain if you want to avoid bigger files then 1TB however big servers may easily create full backup files above 1TB.
VCP-DCV
Veeam Certified Engineer
http://foonet.be
vmniels
Veeam Software
 
Posts: 1475
Liked: 322 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Enabling Deduplication on Veeam Backup Repo NTFS ?

Veeam Logoby albertwt » Mon May 15, 2017 10:40 am

Niels,

That's what concerns me because I've got multiple different SQL Servers that is 2-12.5 TB in size, so do I separate the backup job to be one VM per Veeam Backup Job ?
So is it advisable to enable the Windows Server 2012 R2 deduplication or not really if most of my Veeeam Backup job is larger than 2 TB ?
--
/* Veeam software enthusiast user & supporter ! */
albertwt
Expert
 
Posts: 598
Liked: 19 times
Joined: Thu Nov 05, 2009 12:24 pm
Location: Sydney, NSW

Re: Enabling Deduplication on Veeam Backup Repo NTFS ?

Veeam Logoby vmniels » Mon May 15, 2017 10:45 am 1 person likes this post

You can bundle multiple VM's in the job if you enable per vm backup chains however the default deduplication might become an issue for the large servers. Did you already check the other thread veeam-backup-replication-f2/best-practice-for-ms-server-2012-dedup-repo-t14002-105.html ? There are some examples in there from customers with big VM's.

It might be better to look at Windows 2016 with ReFS if you are looking at space savings.
VCP-DCV
Veeam Certified Engineer
http://foonet.be
vmniels
Veeam Software
 
Posts: 1475
Liked: 322 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Enabling Deduplication on Veeam Backup Repo NTFS ?

Veeam Logoby albertwt » Mon May 15, 2017 10:53 am

Hi Niels,

Thanks for the pointer and sharing for the potential issue.
As at the moment, I do not have the Windows Server 2016 license yet, hence I can only use 2012 R2.

So if that's the case I will just use Deduplication for all of my File Servers VM and not with the Veeam Backup Repository.
--
/* Veeam software enthusiast user & supporter ! */
albertwt
Expert
 
Posts: 598
Liked: 19 times
Joined: Thu Nov 05, 2009 12:24 pm
Location: Sydney, NSW

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby foggy » Mon May 15, 2017 2:27 pm

Please note that this will affect the amount of changes copied during incremental job runs (unless you have BitLooker enabled).
foggy
Veeam Software
 
Posts: 14560
Liked: 1060 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby Delo123 » Mon May 15, 2017 2:38 pm 1 person likes this post

Hi Albert,

the 1TB limit seems to be pretty soft, we have 2012R2 enabled since forever and are quite happy with it. Never had a failed restore or surebackup job from it. Our largest VBK (VM) is around 6TB.
Important however is to use 64k clusters and format the volumes with /L, also never create a volume over 64TB since VSS doesn't support it (we use thin 62TB volumes to be safe). Currently around 4PB of Veeam backupdata combined over our repositories and 400TB on a 2016 Dedupe volume, actually 2016 dedupe seems to be a bit worse regarding actual space savings (performance got better due to multi threading howerver), i believe the reason is in 2016 only the first TB or first 2TB actually get deduped from a file (read that somewhere)
Delo123
Expert
 
Posts: 340
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Best Practice for MS Server 2012 DeDup Repo

Veeam Logoby tntteam » Fri Jul 21, 2017 10:04 am

Hi,

I'm digging "old" thread just to add some infos.
For those saying to activate per-vm on deduped windows volumes, I would advice not to do so. I think windows get lost when there is too much files because I faced this problem and once it happens, any optimization job will faill with "exited unexpectedly" even after full GC and scrub jobs that went successfully.
On the other hand, never got any problem on another deduped volume which do not have per-vm backup file split, dedupe works great and my files are like 4TB.

Also about win2016, I don't think the story about only first TB processed is true, you can check yourself using Measure-DedupFileMetadata cmdlet, check DedupSize and DedupDistinctSize values, on a folder with multiple deduped files over 3, 4TB (cmdlet is slow to run, but it is supposed to count every block of every file in the folder, so...)
tntteam
Enthusiast
 
Posts: 45
Liked: 3 times
Joined: Fri Aug 28, 2015 12:40 pm
Full Name: tntteam

Previous

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Majestic-12 [Bot] and 23 guests