Veeam Next Big Thing - REFS

[mkretzer]

Hello,

will the new REFS functionality provide any benefit for ACTIVE FULL backups? Or will they just take all the space as before?

Markus

[vClintWyckoff]

Active fulls, no since they go out and read off the production environment and actively write out a full backup file. The ReFS enhancements are obviously specific to 9.5 and having a Server 2016 Repository formatted with ReFS and will be seen on any of the transform operations - GFS, Reverse Incremental and Synthetics Fulls.

[mkretzer]

Reverse incrementals? How will it help there? Are the new blocks injected from the incremental files by REFS?

[vClintWyckoff]

The full restore point is always latest on disk with Reverse Incremental backup mode. So during this process the changed blocks of data read from the production environment are injected into the VBK. The replaced blocks are read and written out to a resulting .VRB file. During that process that the BlockClone API would be leveraged resulting in much less IO as well as space usage on the ReFS repository.

[mkretzer]

This is sooooo cool and i believe the best feature of 9.5. Why in the world did they only take 10 seconds for that in the presentation but talk forever about stuff noone wants to know??

Can you also tell us about the backup/restore acceleration technologies? Why should it reduce load on the primary storage?

[foggy]

Some details can be found in the corresponding breakout session.

[mkretzer]

Sadly no. There are infos about the processing engine enhancements but not about the backup/restore acceleration technologies...

[nmdange]

Would the performance enhancements with ReFS also apply to Merge operations with Forward Forever Incrementals?

[dellock6]

Yes, as only metadata needs to be updated, without any real block movement on the disks.

[foggy]

Sadly no. There are infos about the processing engine enhancements but not about the backup/restore acceleration technologies...

You can actually get some regarding backup/restore acceleration as well: intelligent source VM data fetching and custom-tailored approach to reading data from different types of the backup storage during the restore (these are mostly under-the-hood code optimizations, so not much to enlarge on them), broker service, direct restore from tape.

[crazzyeddie]

And ReFS support will require Server 2016. No support for 2012R2?

[DaveWatkins]

As far as I understand it, all these new features are only available in Server 2016 because it has ReFS 1.3, which isn't available in previous Windows versions

[mkretzer]

Can an REFS volume be upgraded to the new version?

[veremin]

And ReFS support will require Server 2016. No support for 2012R2?

Correct.

Can an REFS volume be upgraded to the new version?

You'd better ask this question on MS community whether it'd possible to update existing volumes to new ReFS version during 2012 (R2) -> 2016 upgrade.

Thanks.

[vClintWyckoff]

Well its really a best practice to not upgrade operating system versions (2012R2->2016) and rebuild on the newest OS version. However, if you're talking about 'upgrading' Hyper-V hosts MS can in fact perform zero downtime, rolling cluster upgrades.

Can an REFS volume be upgraded to the new version?

As of TP5 the version of ReFS is not updated when upgrading from previous OS versions. Not sure what will make it into RTM, so we'll just have to wait and see.

[crazzyeddie]

Well its really a best practice to not upgrade operating system versions (2012R2->2016) and rebuild on the newest OS version.

It sounds like Microsoft took a page out of the Windows 10 update book and are now "recommending" the 2012 to 2016 upgrade as an in-place install, not rebuild.

[gary.martin7]

I did a quick fag packet calculation and according to the info shared in the breakout session I could be looking at backups currently using around 46TB of space going down to around 20TB.

I noticed in the breakout session the backup option repository option to split VMs into different files was required. Will backups with ReFS repository support Veeam compression and deduplication and multiple VMs per backup file?

Would be a shame if we get the promise of zero space synthetic backups only to have a load of caveats with the tech.

[crazzyeddie]

Would be a shame if we get the promise of zero space synthetic backups only to have a load of caveats with the tech.

Agreed. What about encryption? That's extremely important for many of us.

[Gostev]

I am not aware of any functionality limitations at all.

[nmdange]

So since ReFS still won't support Dedup in Windows Server 2016 (I think), I guess we'll have to pick between getting to use the better dedupe in 2016 or all the other benefits with ReFS

[gary.martin7]

I too am using Windows 2012 R2 Dedupe for Veeam repositories. The Scale Out repositories helped there because I split full backups and incrementals to different drives and only dedupe the fulls disk. However, even though the space is reduced using this method it does load up the server (I process the data outside of the backup window) when the dedupe is in progress. You also have to take care of garbage collection when you delete files.

I see it like this.

Dedupe folds the fulls (and incrementals I guess) on top of each other to remove redundant data (seen it before). This is post process and takes time to achieve. Also not great if any single full is over 1TB. It is very good when the same data exists in multiple backups on the same disk.
ReFS is pretty much doing the same thing for synthetic fulls only, but rather than removing redundant data it never creates it (I got the blocks, I'll just pretend it is a new file). Where you loose here is when your backup files contain the same blocks across files. Where you gain is very fast and io light creation of synthetic fulls.

I guess what you need to do is leverage as much of Veeam's own deduplication in the backup job (bundle like machines together) then you can get the best of both with ReFS. < This statement very much open to discussion.

[nmdange]

For most of my backup jobs, there is only 1 full backup on disk with a number of forward and/or reverse incremental files to meet the short-term retention requirements (usually 14 days to 60 days). In this case, dedupe either using Windows or an appliance doesn't buy that much because there is minimal duplication already. Dedupe makes the most sense when you have long term retention requirements and you are using GFS with Backup Copy jobs.

I suspect if you have a bunch of full backups created via GFS, that ReFS would not provide the same benefits. So perhaps to achieve the best setup, using ReFS on most repositories but then having a separate NTFS formatted repository with dedupe enabled to hold long-term GFS backups would be the best option?

[gary.martin7]

I have a couple of retention policies configured here in my environment. I have a 14 day backup for things like app servers that does run from a single full and I agree that no help would come from ReFS. Also, dedupe is not good here as the incremental rolling into the full nightly would mean at lot of processing to keep up with the changes. However we also have a policy for backing up fileservers and sql servers which has a 90 day retention. I find that extending increments past about a month results in a long recovery time (has to read full then every incremental in the chain). As ReFS synthetic backups are just pointers to existing blocks this means that you can introduce fulls at any time at no storage cost which should have the result of speeding up recovery times (instant recovery should benefit too).

Not sure what you mean by GFS, not a term I have come across before. I do use Backup Copy jobs, but these are set to 2 restore point (minimum available) in order to stage for tape backup.

[dellock6]

Yes Veeam "source" deduplication and compression is always there, so you get the benefits regardless which storage you use.
Then, I see to be honest ReFS even better potentially than deduplication. As it has been said, why do I want to waste IO to first write and then delete with deduplication a block? Way better to never store it in the first place when it comes to create synthetic backups.

On the topic of data reduction, whatever type of job you are running, the savings are all going to be there, let me explain:
- any incremental backup during a day has almost unique data, so chances to be reduced by deduplication are lower. Dedupe will not help here just like ReFS will not help
- when you reach the retention however, there will be a transform operation, and this is the place where ReFS will shine. Instead of rehydrating a full to inject the new blocks ReFS will just write new metadata in his filetable. I see this scenario better for ReFS than deduplication
- GFS: same thing. At every interval, Veeam has to create the new full. In the dedupe appliance, again the file has to be created for real, while in ReFS it's just a metadata update. And since blocks are not copied, they don't waste space exactly like in the deduplication scenario, with the benefit of lower IO as deduplication will have to read and write tons of blocks to create the new full

I'm honestly wondering if deduplication will be needed AT ALL with this new technology, especially when you add to the discussion the "restore" performance. ReFS is not deduped, so there's nothing to re-hydrate during the restore.

[nmdange]

@gary GFS refers to long term retention on disk with Backup Copy jobs by keeping weekly, monthly, yearly backups. Since each of these is actually a full backup the disk space requirements are quite large. https://helpcenter.veeam.com/backup/hyp ... y_gfs.html

@Luca
I understand how ReFS helps with forward forever incremental. When the oldest incremental is merged into the full backup file, instead of copying the data from the vib file into the vbk file, you can just update the metadata in the vbk file to point to the same physical blocks as the vib file.

If you are saying this also applies when you have multiple synthetic fulls with GFS, then this new ReFS functionality is even better than I thought!

The only argument for dedupe would be if you could get additional space savings when deduplicating between multiple backup jobs (or if you have per-vm backup files). However if you group VMs that tend to have similar data into the same job, there would be very little commonality between jobs anyway. With all the performance issues, overhead, cost, etc. of having to use a dedupe appliance, I see very little reason to use them at all. It's also another reason to stay away from NAS devices. I've always thought a Windows server with a ton of cheap internal storage was the best choice for backup repositories and this just confirms that

[Gostev]

If you are saying this also applies when you have multiple synthetic fulls with GFS, then this new ReFS functionality is even better than I thought!

The only argument for dedupe would be if you could get additional space savings when deduplicating between multiple backup jobs (or if you have per-vm backup files). However if you group VMs that tend to have similar data into the same job, there would be very little commonality between jobs anyway. With all the performance issues, overhead, cost, etc. of having to use a dedupe appliance, I see very little reason to use them at all. It's also another reason to stay away from NAS devices. I've always thought a Windows server with a ton of cheap internal storage was the best choice for backup repositories and this just confirms that

Ditto. You are spot on.

[ds2]

Hello,

does it also work for Endpoint Jobs and Endpoit Copy Jobs when target is a B&R Repo with REFS?

Greetings

ds2

[skrause]

Will the ReFS support in 9.5 work with ReFS on 2012R2 or will it require an update of components to 2016?

The ReFS support sounds cool and I would like to do some testing, especially on copy jobs, but I likely won't be able to move our infrastructure to 2016 until the middle of next year at the earliest.

[foggy]

As mentioned above, ReFS support will require Windows Server 2016.

[szwicker]

And the benefits of Veeam9.5/REFS3.0/Win2016 WILL also take place on the Cloud Connector side?
So when I have those big offsite copy jobs running (which are basically like forever incrementals and take forever( I'll get the performance boost as well?
I'm asking because I'm contemplating putting dual Quantum appliances and letting them dedupe and replicate.