Comprehensive data protection for all workloads
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: VBR 9.5 - REFS

Post by antipolis »

quick question : from the way I see it having a ReFS repository on top of storage spaces (on w2016 + vbr 9.5) makes active fulls and backup files health check not needed, is this assumption correct ?
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VBR 9.5 - REFS

Post by Mike Resseler »

Mark,

I perfectly understand your concerns on ReFS and S2D being mature enough or not. However, MSFT is advising this as THE preferred setup for Hyper-V which means it should be tested intensively, otherwise they would not take that risk (at least that is what I assume...). Also don't forget that ReFS is already version 3.1 so it is not a v1 solution anymore.

Mike
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VBR 9.5 - REFS

Post by Mike Resseler »

Hi Antipolis,

I certainly wouldn't leave health checks away. Having a ReFS repository on top of S2D gives you a defense against so-called bit rot.

On the active fulls that might be indeed something that you can leave away, because the savings are done on the synthetic fulls.

However, as always... Don't forget surebackup and test on a regular basis your restores :-)

Mike
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: VBR 9.5 - REFS

Post by antipolis »

well I thought the point of backup files health check was precisely to detect bit rot corruption
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VBR 9.5 - REFS

Post by Mike Resseler »

The S2D / ReFS solution does checking on the file system but does not look to the source anymore... that alone would be a reason for me to continue to perform health check as an additional safety measure (but I might be a bit over concerned when it comes to backups :-)).

Some people might / will have a different opinion on this, but for now I would like to have both of them, just in case. But again, that is my thought on it
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VBR 9.5 - REFS

Post by Gostev »

Mike Resseler wrote:The S2D / ReFS solution does checking on the file system but does not look to the source anymore
But neither health check does (aside of when it needs to actually fix the corruption).
antipolis wrote:well I thought the point of backup files health check was precisely to detect bit rot corruption
To be honest, we have an internal argument here. Some people, me included, are convinced health check is not required on ReFS because across integrity streams and logical backup structure check performed at the beginning of each backup job run you have it all covered. Other people (some smarter than me) push for still doing it because they believe health check can potentially catch issues caused by some unknown Windows kernel bugs for instance.

The good thing is that as a user, you still have the flexibility to do it based on your paranoia level ;)
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VBR 9.5 - REFS

Post by Mike Resseler » 1 person likes this post

And so my boss just called me paranoia :-D
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: VBR 9.5 - REFS

Post by antipolis »

Gostev wrote:But neither health check does (aside of when it needs to actually fix the corruption).
I must admit I got really perplexed when Mike said health check "looks in the source"
Gostev wrote: To be honest, we have an internal argument here. Some people, me included, are convinced health check is not required on ReFS because across integrity streams and logical backup structure check performed at the beginning of each backup job run you have it all covered. Other people (some smarter than me) push for still doing it because they believe health check can potentially catch issues caused by some unknown Windows kernel bugs for instance.

The good thing is that as a user, you still have the flexibility to do it based on your paranoia level ;)
my point exactly... getting rid of health check would be a relief, especially when backing up 10Tb+ virtual machines...

imho s2d+refs makes health check redundant, especially if using surebackup
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VBR 9.5 - REFS

Post by Mike Resseler »

Antipolis,

Yeah my apologies... I meant what Gostev wrote but was a bit too fast... It looks in the source IF IT needs to fix corruption, something S2D and ReFS doesn't do... My bad, sorry
push3r
Enthusiast
Posts: 36
Liked: 6 times
Joined: May 17, 2013 11:54 pm
Contact:

Re: VBR 9.5 - REFS

Post by push3r » 2 people like this post

Here's what I have observed after some testing with regards to Veeam 9.5 integration with ReFS and Storage Spaces.

For my environment.

### ReFS is only good for any method that synthesizes VBKs. Use cases:
1) Backup Copy Job with GFS policy as the GFS VBKs will be synthesized and they will also benefit from space saving. Thanks to Veeam's integration with Windows 2016 BlockCloning technology.

2) Primary Backup Job that synthesize VBKs instead of Active Full. I don't trust synthetic VBKs for my primary because I am also doing replication and BCJ from this source. Active Full does not take that long in my environment, a little over two hours for all the VMs.

### Since I am using method #1 (BCJ with GFS) above, creating full VBKs from scratch (requirement for Veeam to work with BlockCloning) on the newly formatted ReFS volume took some workaround as I didn't want to do an active full through the WAN again. If you have your old BCJ VBKs already at your DR Site, you can trick Veeam B&R to use those as source Backup Job Repo for the BCJ. Then once that is completed, you can point the BCJ's source back to your Production. If anyone wants to know how to do this, let me know.

### Storage Spaces sucks!
1) Parity Space in Windows 2016 is still as useless as the Windows 2012R2 version. It is super slow in terms of throughput/performance even if you use 2x SSD drives as Write Back Cache for a parity space with around 5 columns (more or less columns didn't matter much). It can not beat my good old hardware RAID with WBC/battery. Even at RAID 6, the hardware RAID still beats Storage Space parity with no sweat. Don't waste your time testing Windows 2016 Storage Space Parity. Search the web, there are updated articles about this. I've run through my own tests and confirmed this.

2) Mirror Space performance is on par with hardware RAID. Good for Veeam primary backup but wastes 50% of hard drive space if used for Veeam archival backup.

3) Storage Space (parity or mirror) with how it works with "column" are totally inefficient in terms of expansion. You have to expand by adding the exact amount of hard drive/columns. Make sure you understand columns thoroughly before venturing into Storage Space. Have to plan your column configuration in relation to expansion and enclosure available slots. With hardware RAID, just add one drive at a time in the case of parity (i.e. RAID6). For two-way mirror, just add two drives at a time. With Storage Spaces, good performance is recommended at around 4 columns, so you have to add 4 drives at a time! Additionally, as an example, say you use 5 columns parity space (5 hard drives) and expand with another 5 hard drive for a total of 10 drives, you are still only able to have one hard drive failed! Are you kidding me? Talk about inefficiency. I will stick with my hardware RAID 6, thank you.

4) What happens when a hard drive failed, does the failed drive on your enclosure blink red? Remember, Storage Spaces is software, so is it smart enough to work with your enclosure? With hardware RAID, even your boss can replace a failed hard drive. :)

Please correct me if I am wrong on the Storage Spaces observation above.

I decided that Storage Spaces is not worth the trouble just to have Integrity Stream benefit with Veeam 9.5 and ReFS. I do active full once a month anyway so I am not worry about the integrity of my backups.

I was looking forward to testing Storage Spaces Direct (aka Hyper Converge) but since expansion is so inefficient, I feel rather disappointed.

BTW, I have a NAS box using ZFS and it's totally awesome in terms of flexibility (iSCSI, FC, NFS) and performance. If only Microsoft would adopt ZFS instead of Storage Spaces...
push3r
Enthusiast
Posts: 36
Liked: 6 times
Joined: May 17, 2013 11:54 pm
Contact:

Re: VBR 9.5 - REFS

Post by push3r » 2 people like this post

Additionally, I would recommend this with regards to Windows 2016 and Veeam 9.5.

ReFS for Repo with Backup Copy Job with GFS. Benefit includes fast VBK synthesis and space saving.
De-duplication for Primary Backup Repo. Benefit includes space saving.

Dedup on Windows 2016 is super fast and efficient now with Multi-thread support. Each thread is tied to each CPU.
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: VBR 9.5 - REFS

Post by antipolis »

push3r, thank you so much for this

I indeed had very poor experience with SS in win2012 and had some big hopes that it would be better in 2016, your feedback will save me hours of work and frustration

also good to know that dedup is better in 2016

big fan of ZFS here as well, but I have only low cost sata drives to run it in my lab so performance is poor (even with ssd slog)
push3r
Enthusiast
Posts: 36
Liked: 6 times
Joined: May 17, 2013 11:54 pm
Contact:

Re: VBR 9.5 - REFS

Post by push3r »

Antipolis,

For your ZFS, do you have:

1) good amount of RAM? I have 64GB on my 6 x 5TB hard drives in mirror vdev.
2) use mirror vdev for VMs
3) your ssd slog device should be super fast enterprise ones, not consumers'. Check this. https://b3n.org/ssd-zfs-zil-slog-benchm ... omparison/
4) sata is fine, just make sure you are using a decent number of drives. the more drives the better for performance, of course.

Yeh, too bad about Storage Spaces as it would be nice to take advantage of Integrity Stream in Windows 2016.
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: VBR 9.5 - REFS

Post by antipolis »

32 Gb Ram
tried both raid z20 (z2 on mirror vdevs) and raid 10
mirrored server grade ssd for slog
nas hdd sata x14 connected via an old enclosure to sas hba

no matter what I do as long as I use sync=enabled performance is worser than what I get from a win2012 iscsi target (/w 24 drives raid 60 sata backend)
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VBR 9.5 - REFS

Post by Gostev »

push3r wrote:I decided that Storage Spaces is not worth the trouble just to have Integrity Stream benefit with Veeam 9.5 and ReFS. I do active full once a month anyway so I am not worry about the integrity of my backups.
Just wanted to note that both the Integrity Streams and our ReFS integration is available for simple ReFS volumes too, they don't have to be backed by Storage Spaces. The only thing you lose without Storage Spaces is bit rot auto-healing from data mirror or parity sets. However, you will still get notified about corruption appearing, which is always good to know in advance and take care of that with an Active Full (instead of finding out about said corruption upon the restore).

Perhaps you understand this, but reading the above may confuse other readers, so I figured I'd clarify this part.
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: VBR 9.5 - REFS

Post by antipolis »

this is what i had in mind... monitoring the refs notifications then either trigger an active full or backup health check when needed
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VBR 9.5 - REFS

Post by Gostev »

Exactly!
push3r
Enthusiast
Posts: 36
Liked: 6 times
Joined: May 17, 2013 11:54 pm
Contact:

Re: VBR 9.5 - REFS

Post by push3r »

Gostev wrote: Just wanted to note that both the Integrity Streams and our ReFS integration is available for simple ReFS volumes too, they don't have to be backed by Storage Spaces. The only thing you lose without Storage Spaces is bit rot auto-healing from data mirror or parity sets. However, you will still get notified about corruption appearing, which is always good to know in advance and take care of that with an Active Full (instead of finding out about said corruption upon the restore).
Thank you Gostev for clarify some of the terminologies. That's great news regarding Integrity Streams support with simple ReFS volumes.
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: VBR 9.5 - REFS

Post by antipolis » 1 person likes this post

so with SS out of the picture it all comes down to :

ntfs for deduplication
or
refs for integrity checks + fast and space efficient synthetics/GFS backups

ReFS just sounds like it brings more to the table
push3r
Enthusiast
Posts: 36
Liked: 6 times
Joined: May 17, 2013 11:54 pm
Contact:

Re: VBR 9.5 - REFS

Post by push3r » 1 person likes this post

look at my recommendation. :)

NTFS with dedup for regular Backup Job repo (forever incremental with monthly full). ReFS won't do anything here as nothing is being synthesized.

ReFS for Backup Copy Job with GFS policy.

As simple as that and it's what I have just implemented in my environment.
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: VBR 9.5 - REFS

Post by antipolis »

yes I agree with your points but I just meant that in the case you can't (or do not want) to run active fulls regularly maybe it's better to use refs integrity checks and run active full only when needed
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: VBR 9.5 - REFS

Post by nmdange »

push3r wrote:
look at my recommendation. :)

NTFS with dedup for regular Backup Job repo (forever incremental with monthly full). ReFS won't do anything here as nothing is being synthesized.

ReFS for Backup Copy Job with GFS policy.

As simple as that and it's what I have just implemented in my environment.
If you are doing forward forever incremental then ReFS will help during the merge process. On one of my backup jobs with large change data the merge process can take hours every day which will go away with ReFS!

You can actually do both using an interesting strategy that is described here: https://charbelnemnom.com/2016/10/how-t ... p-storage/

Basically you have a Hyper-V 2016 host with NTFS, and you run your repository in a VM with ReFS enabled then enable the "Virtualized Backup" Dedupe setting on the host. The post is for DPM 2016 but the concept is the same with Veeam since both are leveraging the same ReFS APIs.
saarq
Lurker
Posts: 1
Liked: 4 times
Joined: Apr 19, 2016 6:55 pm
Full Name: Nate
Contact:

Re: VBR 9.5 - REFS

Post by saarq » 4 people like this post

I just want to say that this has been a godsend. I have a 4TB file server and the synthetic fulls were taking 80hours and still not completing (yes, the repository performance, which is slow, was the major factor in this.) I was looking at having to get some new hardware when 9.5 was released. Ran my first synthetic full on that job this weekend and it only took 30 minutes.
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: VBR 9.5 - REFS

Post by antipolis »

antipolis wrote:this is what i had in mind... monitoring the refs notifications then either trigger an active full or backup health check when needed
now that I think of it it would be really nice to have this integrated in VBR (or a combination of One/VBR) : when integrity streams detects a corrupted block then an active full is triggered
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: VBR 9.5 - REFS

Post by lando_uk »

On this subject.

A typical Veeam Backup Validator task takes me about 8 hrs to validate a single job with 30 VMs / 3TB, this is because of slow RAID6. Would ReFS also speed up this health check? Because if it does, then that would be a big deal.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VBR 9.5 - REFS

Post by Gostev »

No, it will make zero difference. Validator needs to read the entire backup file contents regardless of the file system to calculate its checksum. The bigger question is if you need Validator at all considering the presence of data integrity streams (see the corresponding discussion on the previous page of this topic).
ds2
Enthusiast
Posts: 82
Liked: 19 times
Joined: Jul 16, 2015 6:31 am
Full Name: Rene Keller
Contact:

Re: VBR 9.5 - REFS

Post by ds2 »

Are there also REFs-Benefits when using Foreverforward to a REFS-Target? What about the merge, is it faster by pointing only to exiting blocks? Or are there only benefits by GFS or syntetic fulls?

Greetings

Ds2
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: VBR 9.5 - REFS

Post by veremin » 1 person likes this post

Two main benefits provided by ReFS are space savings and merge/synthetic performance increase. So, answering your question, yes, merge process will be much faster, should you stick to ReFS target. Thanks.
ds2
Enthusiast
Posts: 82
Liked: 19 times
Joined: Jul 16, 2015 6:31 am
Full Name: Rene Keller
Contact:

Re: VBR 9.5 - REFS

Post by ds2 »

Thanks for clarification.
Amarokada
Service Provider
Posts: 92
Liked: 10 times
Joined: Jan 30, 2015 4:24 pm
Full Name: Rob Perry
Contact:

[MERGED] Windows 2016/ReFS magic, how do I know it's working

Post by Amarokada »

Hi All

I've recently done an in-place Windows 2012 R2 to Windows 2016 upgrade on all my Veeam servers, and upgraded Veeam to v9.5. My aim is to utilise the new ReFS block pointer efficiencies so that synthetic fulls don't take all day. Under Windows 2012 R2 my jobs were using full weekly backups with Windows Dedupe crunching them down, but with ReFS I'm expecting a similar deduplication effect as part of the file block linking.

As part of the upgrade, I deleted all local backups (from within the Veeam interface "disk" section) to clear out the historical database. I then formatted the local RAID disk with ReFS (4k allocation size), and then kicked off active-full backups of all my jobs.

All my Veeam servers save their backup files to a local RAID disk (via a local repository setup within a local scale-out repository, so I can add further disks in the future without having to change the destinations in the jobs).

This weekend saw the first synthetic full triggered, and I'm not seeing what I expected I would.

One job has about 3.7TB of backup files to crunch through (1 active-full followed by 3 incrementals). The repository is configured to save each VM as a separate file.
It is still running now and has been for 2 hours and is only 39% through the synthetic full creation. Taskmanager shows the disk at 100% active time with about 200MB/sec reading and writing going on.

Also another job which has finished (took 55 minutes to make the synthetic fulls of 11 VMs), had 396GB of backup files (1 active-full followed by 4 incrementals). The resulting 372GB of synthetic fulls are shown as taking 372GB of disk space when viewing the file properties. The whole backup folder shows 768GB size, and 768GB size on disk in folder properties.

I was under the impression that quite a bit of the synthetic fulls would show up as taking no disk space as the matched blocks would just be pointers in the existing backup files.

One thing to note, I have the repository settings set to allow 8 concurrent tasks. However I don't think that is an issue since the 3.7TB job is the only one running now, and still the disk is at 100% active time.

So with the above information is it possible to know if ReFS is doing its thing right? and if not, what step did I miss when reformatting the disk?

Cheers
Rob
Post Reply

Who is online

Users browsing this forum: Mildur and 152 guests