How we moved 1,1 PB of Repos from one Veeam Server to another and why ReFS metadata upgrades now haunt my dreams

[mkretzer]

I hope this post helps others who do upgrades of big ReFS repos to stay calm and “trust the system”

Our main Veeam repo is a SOBR consisting of 5 similar-sizes volumes (optimized for ReFS “allowed” volume size back from the time it mattered more), hosted by FC backend storage systems. The repo server was a windows server core 20H2 system. Since this OS is long out of support, we planned to migrate this to a Windows 2022 server (and a new hardware).
The storages are Fibre Channel attached (some support storage snapshots, some not), which means simply moving the Repo to another server is easy in theory, but since we are quite paranoid, we searched for a way to do this nearly risk-free.

We asked Veeam support and the official way of doing this is:
- Put the whole SOBR in Maintenance Mode
- Move backend storages to the new 2022 server
- Mount the volumes in Windows Server (causing ReFS Metadata upgrade)
- Create new Repos for the old volumes/paths
- Create a new SOBR, put in all the new Repos
- Rescan

This procedure has a big drawback for: with 2022 there is no way back if Windows Server trashes all the ReFS volumes because it upgraded the ReFS metadata – which would mean active full for 4700 VM on that SOBR. So we search for a way to do this for a subset of volumes.

We hatched our own plan based on kb3100 (sanctioned by Veeam support):
- Put only two of the volumes/repos of the SOBR into maintenance mode (those whose backend storage supports snapshots)
- Create a storage snapshot for these volumes
- Move these volumes to the new server
- Mount the two volumes in Windows Server (causing ReFS Metadata upgrade)
- Add the two Repos to the old SOBR
- Rescan the SOBR
- If all goes well add the other volumes/repos the same way
- If all goes bad, restore the storage snapshot and reattach to the old server

This procedure worked great in the end. But we did not factor in that the ReFS metadadata upgrade is the scariest part in the whole procedure.
As soon as we took the first two volumes online (which are the newest in the SOBR) and the drive letters showed up disk management and windows explorer froze completely! It stayed this way for nearly an hour, after that all came back as if nothing was ever wrong. We created the Repos, rescanned, tested our backups, and all was fine.
So, we now know that ~500 TB of storage takes an hour to go online and were confident that migrating the remaining 600 TB (which have more spindles and are generally faster) will take also about an hour. But it was not that simple. We still do not know why (fragmentation perhaps?) but Metadata upgrade of the remaining volumes took more than 4 hours to complete!

While we were waiting, we searched for logs and checked performance data on the backend storage – sadly, there is just nothing telling you “ReFS is doing something currently, stay calm, don’t turn off the server”, no log, low CPU and only minimal IO on the backend. We talked to ReFS devs (I am extremely grateful they still answer a ReFS question every now and then – they are just great guys!) and they confirmed this to be normal. With the move to Server 2022 “container btable format“ is upgraded at first mount which seems to be a sequential/single threaded and blocking process!

By the way our XFS is even bigger – we have never seen something like this there…

So - stay calm, don't reboot in the middle of the process (and perhaps use XFS or Object in the long term ).

Markus

[Regnor]

Interesting post Markus.
How many changes actually occurred during the metadata upgrade, or how much did the Storage Snapshot grow?

[mkretzer]

Good question! Interestingly, storage snapshot did grow only a few hundred GB i believe, as it was the metadata that was upgraded. Write rate was very low and no constant writes (long time without any writes at all). Thats the reason we believed something is wrong...

[mweissen13]

Thank you very much for that very relevant post. It is beyond my imagination why ReFS is still in a state like that after >10 years of it's existence. Sometimes I have the feeling that it's still a beta product. Just last week one of my colleagues attaches a ReFS volume by accident to his Windows 11 Workstation. While ReFS ist not even supported there (you're not able to format a volume with ReFS on Windows 11) it still happily mounted it automatically and upgraded the metadata without any confirmation. After that the volume was trashed for the Windows 2022 Veeam Server (could not be used anymore) and also, there is no way back. You can only re-format it and start all backups as active full, losing the complete chain.
That's why we are in the process of migrating all important repositories from ReFS to XFS, where such scary things just don't happen. Plus it's also more performant.

[rendest]

Markus

Long time no see, just as an update, ever since we migrated away from ReFS towards XFS, we have not experienced any issues and the backups are performing as expected with a similar install base (as discussed before).

We will not attempt to run ReFS ever again, we have lost all hope in it.

We are looking into object storage just for storage consolidation of all our Veeam platforms, but for pure block storage we are extremely happy with XFS and it performs as expected.

[mkretzer]

@mweissen13 As far as i know nearly every new release of Windows does ReFS metadata upgrades which makes the Volume no longer able to be mounted from older Versions. From my point of view the problem is that you are just not informed/asked by the OS about the upgrade.

[JaySt]

Thank you very much for that very relevant post. It is beyond my imagination why ReFS is still in a state like that after >10 years of it's existence. Sometimes I have the feeling that it's still a beta product. Just last week one of my colleagues attaches a ReFS volume by accident to his Windows 11 Workstation. While ReFS ist not even supported there (you're not able to format a volume with ReFS on Windows 11) it still happily mounted it automatically and upgraded the metadata without any confirmation. After that the volume was trashed for the Windows 2022 Veeam Server (could not be used anymore) and also, there is no way back. You can only re-format it and start all backups as active full, losing the complete chain.
That's why we are in the process of migrating all important repositories from ReFS to XFS, where such scary things just don't happen. Plus it's also more performant.

to be fair, imho it's more scary you were able to accidentally mount the volume to the Windows 11 workstation. I gave up on ReFS as well going forward, but in this case as you described, it's perhaps not the biggest issue...

[mweissen13]

to be fair, imho it's more scary you were able to accidentally mount the volume to the Windows 11 workstation. I gave up on ReFS as well going forward, but in this case as you described, it's perhaps not the biggest issue...

I see your point. My colleague took the Backup NAS after repairing the power supply and attached it to his computer because he wanted to see what's still on there. He was not aware that this would trigger the metadata update. Sometimes, these things just happen in practice. If the NAS would have been formatted with NTFS, nothing bad would have happened.

[mweissen13]

@mweissen13 As far as i know nearly every new release of Windows does ReFS metadata upgrades which makes the Volume no longer able to be mounted from older Versions. From my point of view the problem is that you are just not informed/asked by the OS about the upgrade.

Exactly. The OS should ask something like "BIG FAT WARNING: The metadata of the volume will be upgraded. It will no longer be mountable in older versions of Windows. Are you sure you want to proceed now?". That's what I would expect from an RESILIENT File System.

[RGijsen]

Yeah, same here. I've stated several times before here, using ReFS for data you care about is not a good decision. When ReFS is used in S2D it shines somewhat, with all the flaws that is has (every FS has). But in S2D you could state it is somewhat resilient, being able to detect errors and sync the right data back, much like a proper RAID or ZFS system if you will.
Standlone though.. oh my. The only tool available to troubleshoot ReFS is REFSUTIL which helps to identify issues, not to solve them. We've had tens of terrabyes that were marked 'inaccesible' by ReFS after a power outage that caused corruption. That's not ReFS' fault of course. After that though ReFS marks the space inaccesible and that's it. No way to recover the data if we wanted to, no way to free up the space of corrupt files. MS support broke their heads on it, but still space was simply lost.
Apart from that, as with all deduplication solutions on spindles (and even flash at some point), while blockcloning sound great, the data hence gets scattered all over your disks. Restores become painfully slow. We used ReFS with blockcloning on a 12x10TB spindle setup with a beafy RAID controller. Initially it was all great, space savings were awesome, but in the end all became so painfully slow... Restores would run at megabytes per second at best. Try to recover a 10TB VM using that. We've had numerous issues, the one after power loss is one (again, that on it's own I can't blame anyone but our datacenter for, but unable to reclaim corrupted space? come on). Issues after OS upgrades. ReFS just crashing, or needing registry values or custom dll's to get uit running correctly.

In the end we decided to just buy raw storage and use NTFS . And we've never been happier since. Backups are stored pretty much sequentially on the actual spindles now, the performance is just the same after two years than the day we started moving back to NTFS. And it's so much more reliable, mature and above that, it's predictable. Especially with standalone (non S2D) volumes, ReFS is way too flakey and unpredictable to put your valuable data on, just to save some pennies on storage. In fact, in one of our offsite copies, we still use ReFS on a full flash system, and when we test restores there, it's still slower than from our spindle-system on our main site.

Just my 2 cents.

[mkretzer]

@RGijsen: ReFSUtil is not that bad for extracting data from corrupt volumes. I already used it after a crypto attack and it was able to restore alot of files. In general you are right, you have to be very careful with ReFS and it lacks a lot of flexibility. But we could never live without block cloning with our 4700 VMs .
If you have the time check XFS its really the best solution for block storage backup repos...

[dloseke]

This is nerve-wracking. I'm not sure I'd have the patience to sit through 4 hours of not knowing if it's doing something or not. Kudo's to you for your patience. That said, my only saving grace would be my fear of corrupting the volume as I recently corrupted a 70ish TB volume when I was having RAID controller issues on a server.

[CaliMSP]

Why XFS and not ZFS for example? ZFS in theory provides better error and thus data protection.

[dloseke]

I believe XFS offers better performance, especially with large files. At least that's what I found when I googled it. Also, I could have missed it, but it looks like OpenZFS doesn't support reflink for block cloning which Veeam takes advantage of as well. Again, I'm not the most well-versed in the Linux world, but that's what a quick search yielded for me.

[mkretzer]

Exactly, XFS and ReFS provide block cloning with Veeam. No other filesystem supports this currently.

[Billy@MSFT]

Hi Markus - this is Billy from the Windows Storage PM team at Microsoft. Thanks for sharing your journey here. Our team wants to address a few of the points highlighted in this thread:

Issue: Long mount time on upgrade to WS2022.
ReFS relies on durability of flushed writes (writes hardened by subsequent HW flush) for correctness. Over the years we have seen violations of this requirement, usually on power loss, and that sometimes results in catastrophic data loss, including ReFS mount failure.
ReFS has inbuilt ability to selfheal from almost all kinds of latent corruptions (i.e., bitflips), and uses the same techniques to handle lost writes.
It works well for some metadata types, and not so much for others, resulting in failures.
In WS2022 ReFS reorganized its metadata layout to make it more resilient to lost writes. Volumes being upgraded to WS2022, have to go through this one-time metadata reorganization, and unfortunately it has to be done on first mount.
We had validated this upgrade time in HCI configurations and found it to be acceptable (<30 secs).
However, for large volumes e.g., 200TB+ on exclusively rotational media with seek penalty, some customers have observed this to be an hour+ operation.
We understand and apologize for this subpar experience. This is the only time in ReFS history, where we had to do this, and hopefully never again.
There unfortunately was no other way to address lost write failures.
Our current recommendation is, to plan upgrade to WS2022 accordingly, if your setup is similar.

Issue: ReFS mount failures.
Some customers have complained about ReFS volumes becoming unmountable down-level, after exposing them to newer OS.
ReFS does not support removable media, so on encountering a down-level volume, ReFS natural logic is to upgrade volume to the latest version.
Customers who want to maintain ReFS volume portability between OS versions, there's a supported mechanism:

Add RefsDisableVolumeUpgrade=1 REG_DWORD key under registry path "HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\FileSystem"

Cmd: "reg add HKLM\SYSTEM\ControlSet001\Control\FileSystem /v RefsDisableVolumeUpgrade /t REG_DWORD /d 1"

Add the above registry key to the uplevel systems (restart required), before exposing a down-level ReFS volume to it.
Please note that in this case, any new features in uplevel ReFS would be disabled, and uplevel ReFS would behave functionally equivalent to down-level ReFS.

Issue: Refsutil general unreliability.
Some customers have reported refsutil.exe being generally unreliable, especially when used on large volumes.
We are aware of this issue.
refsutil's internal architecture doesn't scale well for large volumes.
We are rearchitecting refsutil internals completely to make it scale and work seamlessly for large volumes. Unfortunately we don’t have an ETA to share for that updated version of refsutil today.

Let me know if you have any questions on the above.

[dloseke]

Thank you for all of this information. This is immensely helpful, especially a someone who recently lost a 66TB volume due to ReFS corruption after having a RAID controller fail.

[mkretzer]

@Billy@MSFT Very nice, thank you!
For the future a "upgrade needed" in disk management would be even better. I understand that upgrading per default sounds like a great idea, but it ReFS is currently used in such special szenarios that it might make sense to handle it differently.

[stfconsulting]

@BillyMSFT- I have to say I was impressed to see you comment here. Thanks for taking the time.

[mkretzer]

I must say i was never the biggest fan of Microsoft "Support" - but the ReFS Dev team is one of the most professional, passionate and helpful team from another company i had the pleasure of working with.
Thank you @BillyMSFT!

[n.kretschmar]

Dear,

@mkretzer, thanks for sharing this journey which is helpful for all doing real live things, always with the scalpell close to the heart.
For all, who may say now, why not testing a change before...no producing company can invest in a parallel universe, so a petabyte scale data pool is hard to duplicate. And such insider infos by @BillyMSFT must be public available, and visible in the right moment: Maybe a suggestion to the Veeam who also just advertise ReFS along with other different vendor technologies.

Thanks,

Nico

[JaySt]

very good point on having this important information more visible.
i do think it's one of the difficulties a software-only vendor like Veeam has. Especially in PB scale envs. As unrealistic investing in testing a PB scale data pool is, it's not less unrealistic for Veeam to test it. Same goes for server hardware vendors. The freedom of choice in hardware has its pro's and cons for sure.

[d3k]

While we were waiting, we searched for logs and checked performance data on the backend storage – sadly, there is just nothing telling you “ReFS is doing something currently, stay calm, don’t turn off the server”, no log, low CPU and only minimal IO on the backend.

I'm currently in a similar-sounding situation. We had a lockup of a VM form-factor repo that is part of a SOBR. The repo runs server 2019 LTS which is backed by a fibre channel LUN w/ ReFS on it. After a reboot the system hung during boot; 30 minutes later or so it continued on. The only sign of activity was a very slow read of the FC LUN backing the repo. The next day, it locked up again. I detached the fibre LUN and booted into the OS. After verifying the OS health, etc. I reattached the repo LUN and now the server has been sitting reading the LUN for over an hour. Like you, I have been digging for anything that could give me insight into what it was doing (logs, utilities). We see the drive letter, but that is it. Explorer basically hangs trying to open it, and the disk manager hangs on opening as well. There are no indicators what is happening - it's a total mystery. Perhaps some integrity checking? No changes were made to the system.

Now, not knowing what is actually going on, we're in the unenviable position of possibly losing this SOBR extent.

Consequently: XFS is looking a lot more attractive, so perhaps we'll build out new repos with it and start migrating jobs over.

[mkretzer]

It sounds strange that the system locks up not only once but twice - i have never seen something like this with ReFS.
XFS on the other hand indeed did never disappointed us - and together with LVM its much more flexible!