Comprehensive data protection for all workloads
Locked
JVA@Alsic
Novice
Posts: 5
Liked: never
Joined: Dec 29, 2014 10:00 am
Full Name: Jeroen Van Acker
Contact:

Re: REFS 4k horror story

Post by JVA@Alsic »

Indeed, not all disks support this.
Since we've implemented this all our problems have been resolved.
Maybe there is some other write caching configuration on your system available to speed up the process?
Tuning of the RAID controller perhaps?
mkretzer
Veeam Legend
Posts: 1203
Liked: 417 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS 4k horror story

Post by mkretzer »

For us the option is also greyed out (and disabled). I also can not understand how Veeam still is not warning users!
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: REFS 4k horror story

Post by antipolis »

maybe it has been mentioned/debated before (and I am sorry if that's the case) but for the most impacted users here, I'm curious what raid stripe size are you using ? I put mine on 256k

https://www.virtualtothecore.com/en/vee ... ripe-size/
JVA@Alsic
Novice
Posts: 5
Liked: never
Joined: Dec 29, 2014 10:00 am
Full Name: Jeroen Van Acker
Contact:

Re: REFS 4k horror story

Post by JVA@Alsic »

64K, to align with block size of ReFS.
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: REFS 4k horror story

Post by antipolis »

Gostev wrote:@Richard, honestly it does not seem like your issue has anything to deal with what the issue discussed in this thread. Could be a simple I/O error due to heavy concurrent load on the target volume. By the way, you may consider increasing your RAID stripe size to 128KB or 256KB to better match Veeam workload (avg. block size is 512KB), this will cut IOPS from backup job significantly (and your backup storage IOPS capacity is not fantastic, so it could really use this).
^ this

I'm no expert here and by saying this I might be totally out of line but I'm not sure aligning the raid stripe size to the refs block size makes a lot of sense
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: REFS 4k horror story

Post by antipolis »

https://bp.veeam.expert/job_configurati ... ssion.html
Local – this is the default setting and is recommended when using a disk-based repository. When this setting is selected, Veeam reads data and calculates hashes in 1 MB chunks.
LAN – this value is recommended when using a file-based repository such as SMB shares. When this setting is selected, Veeam reads data and calculates hashes in 512 KB chunks.
WAN – this value is recommended when backing up directly over a slow link or for replication as it creates the smallest backups files at the cost of memory and backup performance. When this setting is selected, Veeam reads data and calculates hashes in 256 KB chunks.
Local (>16 TB) – this setting is recommended for large backup jobs with more than 16 TB of source data in the job. When this setting is selected, Veeam reads data hashes and calculates data on 4 MB blocks.
my two big jobs : 8 TB (configured as LAN target so 512k) and 5 TB (configured as WAN target to 256k)
Gve
Service Provider
Posts: 33
Liked: 2 times
Joined: Apr 28, 2015 3:28 pm
Full Name: Guillaume
Location: France
Contact:

Re: REFS 4k horror story

Post by Gve »

antipolis wrote:maybe it has been mentioned/debated before (and I am sorry if that's the case) but for the most impacted users here, I'm curious what raid stripe size are you using ? I put mine on 256k

https://www.virtualtothecore.com/en/vee ... ripe-size/
128k for me.

but if the problem was on the physical storage layer I will have to have a disk latency or disk queue ?
mkretzer
Veeam Legend
Posts: 1203
Liked: 417 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS 4k horror story

Post by mkretzer »

Correct. You should see latency / queue.

We even tries both - we have one storage system which does only 64 k and another which we set to 256 k. No difference.
Gostev
Chief Product Officer
Posts: 31814
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS 4k horror story

Post by Gostev »

antipolis wrote:I'm no expert here and by saying this I might be totally out of line but I'm not sure aligning the raid stripe size to the refs block size makes a lot of sense
The way you put it, it does not makes sense indeed... however, increasing RAID strip size to more closely match Veeam's typical I/O size makes a lot of sense with IOPS-constrained backup storage (and that is regardless of the underlying file system used by backup repository).
Gostev
Chief Product Officer
Posts: 31814
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS 4k horror story

Post by Gostev » 1 person likes this post

By the way, I checked with the ReFS dev team for updates and apparently they are having a good progress working with the affected customers, with the newest performance fixes looking even more promising. So, thanks to everyone who opened support cases with Microsoft - and continues working with them in testing these latest private fixes! I do encourage everyone to open support case with Microsoft on this issue if you have not already - because the issue is not reproducing easily in every lab, and takes a lot of work to simulate for troubleshooting purposes.
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: REFS 4k horror story

Post by antipolis »

Gostev wrote:increasing RAID strip size to more closely match Veeam's typical I/O size makes a lot of sense with IOPS-constrained backup storage (and that is regardless of the underlying file system used by backup repository).
this is what I meant: that reducing the stripe size in hopes of "aligning" to refs will not bring better performance, and can actually do the opposite
mkretzer wrote:Correct. You should see latency / queue.

We even tries both - we have one storage system which does only 64 k and another which we set to 256 k. No difference.
good to know :|

what really made the issue manageable for me was spreading my jobs synthetics&rebuilds on different days of the week...

I'm really curious how this all ends... just reading about "beta drivers" and registry settings for a filesystem which is supposed to be production ready makes me anxious...
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: REFS 4k horror story

Post by dellock6 »

I still find a bit hard to understand why you all want to run synthetic fulls over ReFS. On regular storage, I totally understand that synthetics are a good way to split a chain and avoid to become too long and risk to have one corrupted incremental to corrupt all the following ones, but on ReFS you are just re-pointing the same blocks multiple times in the new full. Fast clone doesn't do any read or write check as Veeam does for synthetic fulls on regular storage, so there's in my opinion really no point in running synthetic fulls over ReFS. And since it seems that large amounts of metadata deletions are one of the causes of the lack of performance in ReFS, synthetic fulls that are expired and need to be removed will not help for sure, while forever incremental would distribute the much smaller deletions over multiple days.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
nmdange
Veteran
Posts: 528
Liked: 144 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: REFS 4k horror story

Post by nmdange »

Running synthetic fulls helps speed up restore times when you have longer retention periods. But I would agree that this does not make sense unless you have a fairly long retention period. Think about it if you have a retention of 30 days with forward forever. If you want to restore your most recent backup, you have to restore a full and 30 different incrementals that have to be processed. Whereas if you have a weekly synthetic full, you only have 7 incrementals.

My pre-REFS strategy has been to use forever forward with 14-day retention, and weekly synthetic full + transform previous chains into rollbacks for 60 day retention. I believe using the transform previous chains into rollbacks option would help with deletions in the same way that forward forever does.
tsightler
VP, Product Management
Posts: 6035
Liked: 2860 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: REFS 4k horror story

Post by tsightler »

nmdange wrote:Think about it if you have a retention of 30 days with forward forever. If you want to restore your most recent backup, you have to restore a full and 30 different incrementals that have to be processed. Whereas if you have a weekly synthetic full, you only have 7 incrementals.
Veeam does not work this way. It does not restore the full and then the next 29 increments, it opens all 30 files at once, reads in the metadata, and then reads whatever blocks are required from each file. The metadata lets it know which file contains the most recent version of each block so only the most recent version of a block is ever read no matter how many points there are in the chain. The Veeam engine was built exactly for this, just about the only performance difference between a chain with 5 points and a chain with 30 points is the time spent reading the metadata from each file, which is typically about 1/10th of a second per-file.

Now sure, it's possible that really large backups, which have lots of metadata, with really large chains, can start getting significant delays, and also in cases of very slow repositories (like dedupe appliances), where reading metadata can be much slower than normal, but for typical ReFS deployments, which are usually on quite fast disks, there's unlikely to be more than a few seconds difference in restore times between a chain of 5 points and a chain of 30 points. Typically, we don't start seeing measurable performance differences until chains reach near the 100 restore point range, and even there, it's usually measured in 10's of seconds of difference.
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: REFS 4k horror story

Post by antipolis »

ok I'm really confused here... I've read a lot of things on veeam /w refs, watched the videos from Gostev, and I might have missed it but I don't recall seeing anywhere that the new best practice is just stop doing synthetic backups on refs...

anyway synthetics or not doesnt explain the performances issues many are seeing, disabling synthetics on regular backups won't solve the issue merge operations or backup copy job GFS
tsightler
VP, Product Management
Posts: 6035
Liked: 2860 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: REFS 4k horror story

Post by tsightler » 3 people like this post

antipolis wrote:ok I'm really confused here... I've read a lot of things on veeam /w refs, watched the videos from Gostev, and I might have missed it but I don't recall seeing anywhere that the new best practice is just stop doing synthetic backups on refs...

anyway synthetics or not doesnt explain the performances issues many are seeing, disabling synthetics on regular backups won't solve the issue merge operations or backup copy job GFS
I'll try to explain it from my perspective. Gostev is part of product development (specifically I believe his title is currently Senior VP of Product Management), while Luca and I work on the field with customers and their deployments (I'm Principal Solutions Architect in NA, and I believe Luca's title is Cloud and Hosting Architect in EMEA). We all work together closely, but I suppose the best way to say it might be that Gostev, and likely all of us in the early days, provide practices that are best in theory, because there is not enough field practice to provide anything else, but, over time, the field proves what practices are actually best, and sometimes this is different based on various technological issues that were unforeseen.

The Veeam best practices document is created by Veeam architects that work in the field, like Luca and myself (and many others), because we observe what things work for customers and what things cause problems, thus the recommendations/suggestions you see here may actually be based on changes we are planning to make in the best practices going forward, or may be based on a specific situation that exist now (like the current issues with ReFS) that we hope will go away in the future, but we want customers to have the best experience now. This is not to say the Gostev and product development do not also see field issues, they of course do through support cases and other avenues (including speaking with field resources like myself and Luca and others), but from that perspective they're focus is mostly regarding fixing the issues so that hopefully theory and practice get closer together, while we, as field resources, work are to provide recommendations that will be successful even with any current technical limitaitons.

And that's exactly where this specific recommendation is coming from. Sure, in theory, using synthetic fulls on ReFS via block clone should not cause any problems. However, in practice, we've seen that having lots of cloned files with many files referencing the same blocks, has many more negative impacts than expected, one of the more common being when those files have to be deleted. Microsoft continues to work on these issues, so hopefully, at some point in the future, theory will get closer to practice, but for now, those of us in the field are saying is that, based on field experience, customers that run synthetics have significantly more performance and stability issues than those that do not so, if you are running synthetics in cases where they offer minimal benefit, it's probably best to not use them.

The reason for this is actually pretty obvious if you think about it. Most of the performance and stability issues around ReFS seem to be with how ReFS accounts for cloned blocks. If I have 45 days of retention with synthetic fulls, that mean I have 6 VBK fules that have blocks referenced with anywhere from 1-6 references in every one of those files. If my full backups are 20TB, that's 20TB worth of blocks that have to have their metadata updated. On the other hand, if I just have 45 days of forever forward backups, the only time there is ever any duplicate blocks is during the merge, and, as soon as the merge is complete, the file with the duplicate blocks (the oldest incremental) is deleted. Since this is not a full backup, it's much smaller, so far fewer blocks need metadata updated during the delete, and the update operation is overall simpler since the maximum times a block will be references is 2, and then immediately back down to 1 again.

In other words, when running synthetics on ReFS, there's a massive increase in the amount of accounting work that ReFS has to do compared to running forever incremental and, in practice, it does impact both the performance and stability of the solution, even if theory says it should not.

Of course you are correct to point out that there are cases, like Backup Copy with GFS, where synthetics are the only option, that's certainly true. We're hopeful that Microsoft eventually gets to the bottom of those specific issues, in which case it probably won't matter anymore more. But for now, we're simply saying that, based on the results of hundreds of customer deployments, and the current status of ReFS, running synthetic fulls in cases where they are not needed and offers little to no benefit is likely to cause more problems than using and forever mode.
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: REFS 4k horror story

Post by antipolis »

tsightler wrote: I'll try to explain it from my perspective.
Thank you for this very informative post !

Just one thing that disturbs me
dellock6 wrote: there's in my opinion really no point in running synthetic fulls over ReFS
tsightler wrote:In other words, when running synthetics on ReFS, there's a massive increase in the amount of accounting work that ReFS has to do compared to running forever incremental and, in practice, it does impact both the performance and stability of the solution, even if theory says it should not.
if synthetics does not make any sense with reFS, and even potentially brings instability, why would we need ReFS in the first case ? just to speed up merges ? I don't see that point justifying refs block cloning by itself (but I guess it depends on deployments and environments)

for backup copy I guess the same logic applies (when not using GFS) : rebuilds = useless with reFS ?

and for GFS would you suggest it's better to have a huge retention on the job (6 mo-1yr) instead ? (granted of course that one can afford to store that time worth of daily incrementals...)

it's just that I deployed refs having in mind that BP was definitely to go synthetics/rebuilds/GFS... in light of your arguments the benefits from refs suddenly looks a lot less appealing to me...

another way to see it (from my current understanding) :

- forever incremental backup chain on ntfs : can get slow over time, possibility to optimize with synthetics
- forever incremental backup chain on refs : can get slow over time, and will stay that way no matter what

are these assumptions correct ?
tsightler
VP, Product Management
Posts: 6035
Liked: 2860 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: REFS 4k horror story

Post by tsightler »

antipolis wrote:if synthetics does not make any sense with reFS, and even potentially brings instability, why would we need ReFS in the first case ? just to speed up merges ? I don't see that point justifying refs block cloning by itself (but I guess it depends on deployments and environments)
Well yes, to me speeding up merges was the number one benefit ReFS provided, as this was one of the most difficult design issues for Veeam at scale. I don't know how many VMs you have, but I work with customers that have 1000's of VMs and sometimes PBs of data, and merges are a big deal at that scale. All you have to do is search for "slow merge" on the forum to see that it was a number one pain point for Veeam for many years.

However, we're not saying synthetics make no sense ever. We're saying that if you are using synthetics in cases where they offer little to no benefit (I'll say, any retention < 100 points although it's hard to set an exact line in the sand), then our suggestion, at least until the performance/stability issues are worked out, is to not use synthetics in those cases. The point was that we were seeing lots of people that have 30 day or 45 day retention using synthetics, and they just don't have much advantage in that scenario, so don't use them and you likely have a better experience with ReFS. It's really that simple.
antipolis wrote:for backup copy I guess the same logic applies (when not using GFS) : rebuilds = useless with reFS ?
I'm not sure what you mean by rebuilds, but I strongly suggest that you enable health checks and defrag/compacts even on ReFS, for any forever backup chain, whether primary backup jobs or backup copy.
antipolis wrote:and for GFS would you suggest it's better to have a huge retention on the job (6 mo-1yr) instead ? (granted of course that one can afford to store that time worth of daily incrementals...)
You are likely to find it to be more stable, but I understand that this is one of the highly desired use cases for ReFS is longer term GFS retention. If you want GFS retention with ReFS then we have to hope Microsoft eventually gets to the bottom of this issue, but you're probably OK for now.
antipolis wrote:another way to see it (from my current understanding) :

- forever incremental backup chain on ntfs : can get slow over time, possibility to optimize with synthetics
- forever incremental backup chain on refs : can get slow over time, and will stay that way no matter what

are these assumptions correct ?
I wouldn't state it this way. Once again, there's no recommendation here that, if you have a year retention, you shouldn't use GFS and synthetics on ReFS. The recommendation to not use synthetics is only for cases where the synthetic offers little to no benefit when used on ReFS. If I'm only keeping 45 days of restores points, you'd be hard pressed to come up with a measurable benefit of synthetic fulls on ReFS. But, if I have a requirement to keep say, 6 weekly, and 12 monthlies, then sure GFS and synthetic makes sense, but you may be more likely to hit some of the ReFS performance issues. Hopefully Microsoft will eventually get to the bottom of those and then it won't matter so much.
nmdange
Veteran
Posts: 528
Liked: 144 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: REFS 4k horror story

Post by nmdange »

Good to know the number of restore points is not as much of an issue as you'd think! I'm hoping being on the new 1709/Semi-Annual Channel build will prove to be better with ReFS than 2016 has been so far.
suprnova
Enthusiast
Posts: 38
Liked: never
Joined: Apr 08, 2016 5:15 pm
Contact:

Re: REFS 4k horror story

Post by suprnova »

The reason we are using ReFS for synthetic fulls is because incremental merges on NTFS were extremely slow (24-70 hours). Also we even have instability with fast cloning merges with ReFS.
tsightler
VP, Product Management
Posts: 6035
Liked: 2860 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: REFS 4k horror story

Post by tsightler »

suprnova wrote:The reason we are using ReFS for synthetic fulls is because incremental merges on NTFS were extremely slow (24-70 hours). Also we even have instability with fast cloning merges with ReFS.
It's certainly possible that you can still have problems even without synthetic fulls, the recommendations are based on the fact that we see far less problems from people who use synthetic fulls, vs those that do not, but less <> none.

When I first started testing ReFS late last year, it was pretty each to crash my home lab environment every 3 weeks with just a very basic setup, and I was able to crash the S3260 in the lab nightly. With current patches from July, and registry key tweaks, neither of these environments have experienced any problems, performance or stability wise, in the last 2 months. Every customer that I've been working with that have been able to be made stable with the beta drivers and/or most July updates, assuming registry keys were set and other minimum requirements were met (proper memory, proper tasks limits, etc). Most customers that are doing GFS points also seem to become stable, at least usable, but, just like in this thread, there are some with performance or stability issues even after all the tweaks, although many have some other questionable factors.

Anyone who is on this thread who would like to become part of the ongoing work, I would love to look at your environment, make sure we have full details of your setup, and continue to track your and our progress. Feel free to PM me your email address and I'll reach out to you. I think all of us hope that Microsoft eventually just gets this thing right but, for right now, I think it's useful to collect information and compare to environments that work, vs those that don't, and share that information when it seems useful for others.
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: REFS 4k horror story

Post by antipolis »

tsightler wrote:I'm not sure what you mean by rebuilds, but I strongly suggest that you enable health checks and defrag/compacts even on ReFS, for any forever backup chain, whether primary backup jobs or backup copy.
I was referring to full backup file maintenance

The description says "Use these settings to cleanup, defragment and compact full backup file periodically when the job schedule does not include periodic fulls", having synthetics enabled on backup jobs, and GFS on backup copy jobs I disabled this... so enabling full backup file maintenance will not bring the same downsides than synthetics on refs ? I mean this still uses block cloning right ?

health checks I left enabled of course
tsightler
VP, Product Management
Posts: 6035
Liked: 2860 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: REFS 4k horror story

Post by tsightler » 1 person likes this post

antipolis wrote:I was referring to full backup file maintenance

The description says "Use these settings to cleanup, defragment and compact full backup file periodically when the job schedule does not include periodic fulls", having synthetics enabled on backup jobs, and GFS on backup copy jobs I disabled this... so enabling full backup file maintenance will not bring the same downsides than synthetics on refs ? I mean this still uses block cloning right ?
Got it, so of course, the compact/defragment option does use block clone during the process, but then the old file is deleted immediately after the compact/defragment is finished, so all the reference counts go back to one. Deletes are definitely one case that can trigger the behavior, but deleting a file referencing a block twice doesn't seem to be as bad as the purge of many different VBK files in one fast action, each of which that are referencing blocks shared many times. So far, I haven't seen file compact/defrag cause an issue and I've had customers leave it on because it is needed to free up unused blocks over time.
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: REFS 4k horror story

Post by antipolis »

tsightler wrote: Got it, so of course, the compact/defragment option does use block clone during the process, but then the old file is deleted immediately after the compact/defragment is finished, so all the reference counts go back to one. Deletes are definitely one case that can trigger the behavior, but deleting a file referencing a block twice doesn't seem to be as bad as the purge of many different VBK files in one fast action, each of which that are referencing blocks shared many times. So far, I haven't seen file compact/defrag cause an issue and I've had customers leave it on because it is needed to free up unused blocks over time.
Thank you for your clarifications, this is much appreciated
Mgamerz
Expert
Posts: 160
Liked: 28 times
Joined: Sep 29, 2017 8:07 pm
Contact:

Re: REFS 4k horror story

Post by Mgamerz »

Can someone post the beta drivers or info on them? While this thread has lots of valuable info, I see many references to beta drivers but the OP doesn't have any useful info in it and I don't really want to search through hundreds of posts to try and find them...
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: REFS 4k horror story

Post by DaveWatkins »

As far as I'm aware the drivers referred to here as beta driver were released in the August Cumulative Update. All the registry entries posted should work on an up to date machine
Cicadymn
Enthusiast
Posts: 26
Liked: 12 times
Joined: Jan 30, 2017 7:42 pm
Full Name: Sam
Contact:

Re: REFS 4k horror story

Post by Cicadymn »

I spoke with Microsoft's ReFS team the other day. Last update was that they were back porting the fix that they believe will solve the issues. Originally they expected it to be out sometime in August or September, however, that wasn't the case. I got the following response from them:

No, this fix wasn’t ported yet. As this isn’t trivial change we want to be sure there is no regression before we release it.

So it sounds like they're really trying to cross their T's and dot their I's. Fingers crossed this will get us to some semblance of normality on ReFS!
nmdange
Veteran
Posts: 528
Liked: 144 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: REFS 4k horror story

Post by nmdange »

The phrase "backported" says to me the fix is part of the next build (1709) which will be out in the next few weeks. I definitely plan on testing backups on the new build, but I haven't really had the same issues other ppl have had. Anyone who's been having problems willing to try out the new release?
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: REFS 4k horror story

Post by DaveWatkins »

On a possibly related note, can anyone confirm this?
ReFS is not supported on SAN-attached storage.
From here https://docs.microsoft.com/en-us/window ... s-overview

Is anyone else running on iSCSI or FC attached disks?
mkretzer
Veeam Legend
Posts: 1203
Liked: 417 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS 4k horror story

Post by mkretzer »

We had it on FC. Does anyone have the same performance issues on non-ISCSI and non-FC disks?
Locked

Who is online

Users browsing this forum: No registered users and 75 guests