2012 R2 Dedupe Issue

[Anguel]

I already went through different threads but some settings seem to contradict a bit, or are maybe written for older versions of Veeam? I thought there may be a summary for the latest v9 but looks like I will need to go through the threads again.
One question I did not see so far:
I want to store Veeam Endpoint physical workstation backups into a deduped Server 2012 R2 repository and wonder if scheduling the full active backups throughout the week is a good idea, e.g. WS 1+2 on Mon, WS 3+4 on Tue, etc. will improve deduplication capabilities of Server 2012 R2 because the big full active backup files will not come in at once but one after the other.

[Delo123]

Scheduling fulls on different days is a very good idea, not only for the dedupe schedule but also for your backup window. I am not really sure what's best for endpoint backup because if I remember correctly it will transform the incrementals into new fulls which can possibly be an issue with dedupe appliances in general... We do not use endpoint backup very much (everything is virtual here:)) but I would schedule a job and have a look at it a couple of days to see what it actually does.
For V9 however things got much better for 2012 dedupe, not only can backup files now be saved per vm, but also it's possible to setup a scale-out repository and then separate fulls and incrementals
So on your 2012R2 repository you would create 2 virtual disks, one for fulls and one for incrementals, add both as repository in veeam and then create a scale-out repository and choose one for fulls and the other one for incrementals. This way the fulls have enough time to dedupe and dedupe settings on the incremental disks could be set to much less (7-14 days) since they will probably not need to be deduped that much...
We see nearly twice the throughput on per vm backup chains, just over 1GB/s, ever for repositories connected over the network ...

Ps. How big are your fulls anyway?

[Anguel]

Thanks for the hints Delo123!
My fulls will be for 10 workstations and 200-300 GB each.

[Delo123]

2013R2 can process around 100gb/hour per volume so you could create 2 volumes and run dedup optimization on both. I would not expect you would run out of time at all that way. Also be aware should the dedup job be interrupted or run out of time it will simply pick up where it left... Great

[Anguel]

Delo123, you are right about Veeam Endpoint Backup - it seems to use forever forward incrementals only so these are not good for deduplication.
As far as I understand v9 is not changing anything regarding this problem or do I misunderstand?

UPDATE: In https://helpcenter.veeam.com/backup/hyp ... _sobr.html I see that scale-out repositories cannot be used for endpoint backups anyway and they are only available in the enterprise editions of Veeam - that would be overkill for our small business. So I can only hope that they will add new backup modes to endpoint backup...

[Delo123]

You can schedule Veeam.endpoint.manager.exe /standalone on the days you want to have a full and it should do so, if it's a shorter than retention period it should never start transforming if I understood correctly

[veremin]

Currently, Endpoint Backup has only one mode and that's forward forever incremental implying regular full backup transformation that might be slow in case of dedupe volume being used as a target. Thanks.

[Anguel]

You can schedule Veeam.endpoint.manager.exe /standalone on the days you want to have a full and it should do so, if it's a shorter than retention period it should never start transforming if I understood correctly

I'm afraid this will not work (see https://helpcenter.veeam.com/endpoint/1 ... _full.html):

When Veeam Endpoint Backup performs standalone full backup, it produces a full backup of your data in a separate folder in the target location. The standalone full backup is not associated with subsequent incremental backups.
...
The standalone full backup is not removed by retention. To delete it, you must manually remove the full backup file from disk.

[Delo123]

Hmm ok, kind of stupid
Well, in that case you could remove Fulls by some script per date, but you would still have the transforming going on with regular incrementals.
Or, you could only create fulls and no incrementals at all, which is possible best for dedupe but jobs would take longer and you would have to control retention by hand

[Anguel]

Or, you could only create fulls and no incrementals at all, which is possible best for dedupe but jobs would take longer and you would have to control retention by hand

This is what I thought about, however if 10 new fulls (300 GB each) come in every night, will dedupe be able to catch up?

BTW: I have added a feature request for VEB: veeam-endpoint-backup-f33/feature-reque ... 33017.html

[Delo123]

Let's see....
100GB/hour for 10 vm's 300GB each would give you 30 needed hour for dedupe, so that could become a problem
However if you create 2 volumes and distribute the backups between them, you get 2x100GB/hour, in that case dedupe would need 15 hours to finish both, so looks good!

Ps. Vm's are 300GB, but some dedupe / compression will happen from Veeam jobs and also i assume 300GB is not actually "used" within the VM's

[Anguel]

Delo123, to split in two volumes is a good idea and you are right that the backups are actually smaller. And in addition Veeam seems to have accepted my feature request for non-forever incrementals in VEB
One more thing I would like to ask you: When you talk about virtual disks or volumes - do you create these somewhere at the RAID controller level or just as normal Windows partitions?

[Delo123]

If you want to be flexible I would recommend to use storage spaces. In our case we create multiple raids (raid controller level) and offer these to a single storage pool (2012R2 feature). From this storage pool you can now create multiple thin! virtual disks on which you can create volumes and enable deduplication. So you could create 2 "big" overprovisioned virtual disks. The only thing you would need to watch is how much space is actually used on the storage pool. Should you ran out of space more raid volumes can be added to the storage pool later (probably you would need the same amount of disks, it depends on amount of columns in storage pool, do some reading about this!).
Instead using raid controller you could also use storage spaces mirroring or parity but we found parity has very bad write performance (much worse than raid5/6 on raind controller) so we opted for "real" raid.

[Anguel]

I see, your setup is more complex than mine On our small business server I have a Fujitsu D2616 (LSI) RAID controller and can attach 8 drives to it. Until now I had 2x256GB SSDs for Hyper-V host in RAID1 + 2x512GB SSDs for VMs in RAID1 (all Samsung 840pro) + new 4x4TB Hitachi NAS 7200 rpm SATA HDDs in RAID10 for the Veeam backups. As I am really impressed by the performance of the Hitachi drives in RAID10 and SSDs are not good for RAID anyway I am now thinking about replacing the 4 SSDs with additional 4x4TB Hitachis and bundling all 8 HDDs into a single big RAID10 array of 16 TB. This way speed will stay high, costs low and I can better utilize the available space. However, for now in my case storage spaces will be probably overkill and will also add an additional possible point of failure so I will probably just use good old NTFS partitions (with deduplication for backups) that should be robust enough and can be resized if needed with a good partition manager like Paragon.

[Delo123]

Nothing wrong doing that. The benefit with Storage pools is you can actually create thin provisioned disks, other than that there is not a big difference in what you are doing now. Please be aware nothing will change on the file system, it will still be NTFS Resizing partitions while you are going is an option, but can become a problem when the first partition is full (partition manager would first have to move space from the seconds partition to the back), also since storage spaces is an integral part of 2012R2 there shouldn't be added complexity... But if you don't feel safe with it of course always give it a try first in a test-lab! But as you say, you should be perfectly fine with what you propose

Ps. Should you do storage spaces you might create 4 Raid 1's instead of 1 Raid 10, so you would have 4 "physical" disks for storage spaces and add more disks later...

[Anguel]

Delo123, many thanks for all the information!

Should you do storage spaces you might create 4 Raid 1's instead of 1 Raid 10, so you would have 4 "physical" disks for storage spaces and add more disks later...

But wouldn't that be much slower than RAID10?

I now also noticed that I will probably have to keep a small extra drive for the Hyper-V host. My idea was to migrate everything to a big RAID10 GPT-formated drive but now I remember that Fujitsu was not able to implement UEFI properly and I had to install Server 2012 R2 in BIOS mode with MBR instead of GPT

[Delo123]

Storage Spaces will stripe data (like Raid 0) when you add multiple extents/disks, so no

[Anguel]

Storage Spaces will stripe data (like Raid 0) when you add multiple extents/disks, so no

Sounds interesting