[Storage Spaces] ReFS "no space left on device" but 8TB free

[okrehan]

Hi everyone,

yesterday we encountered a strange issue. We use VBR 9.5 on two Windows Server 2016 systems. SYS1 acts as primary backup server, SYS2 acts as repository-only server. SYS2 has two volumes, both residing on Storage Pools (simple) and are thin provisioned. Vol1 has 40TB capacity, Vol2 has 43TB. Both volumes/pools consist of three RAID-sets from a HPE P2000 storage array.
We use ReFS with 64kb blocksize and the OS is updated with latest Windows patches. Since we only use reverse incremental backups to this repositories, ReFS space saving technology shouldn't be used hardly.
During yesterdays regular backup intervall, all jobs configured to run against Vol1 failed, all job to Vol2 worked. Error message within Veeam is "no space left on device". Both volumes show ~8TB of free space in Windows explorer as well as Server Manager -> Storage Pools and also on the CLI/Powershell. Indeed it's not possible to copy even a small file to Vol1. We moved some data from Vol1 to Vol2 and instantly we could copy data to Vol1 but only as much as we moved before. It always says no space left on device as we reach the 32TB boundary.
We ran PS optimize-volume with slabconsolidation and retrim option but still have the same error. As Vol2 is setup identically and we do not see this error we think about some kind of ReFS corruption.
Anybody seen this before?

We also opened a case with Veeam support (Case # 04315884) but we think this is a ReFS related issue.

Appreciate any help.
Regards,
Oliver

[soncscy]

Hey Oliver,

https://docs.microsoft.com/en-us/window ... kup-target

Check the limitation here from Microsoft:

Backup targets include the above supported configurations. Please contact application and storage array vendors for support details on Fiber Channel and iSCSI SANs. For SANs, if features such as thin provisioning, TRIM/UNMAP, or Offloaded Data Transfer (ODX) are required, NTFS must be used.

If your backing device requires TRIM/UNMAP, the deletes from ReFS aren't interpreted properly afaik. While it issue involves ReFS, I think it's maybe a combination of your volume + your storage.

How are the volumes from the HPE device managed exactly? (Also, if you can clearly reproduce the issue in Windows, little confused why you'd open a Veeam case. If the OS can't even do it, what hope does an app running on the OS have?)

[okrehan]

Hi,

the storage system is a simple HDD RAID storage, no special features used here so it should be no problem to use it in our setup. ReTrim and SlabConsolidation was just a test if this could solve the problem. The storage itself doesn't use or need TRIM/UNMAP.
We simply created three RAID6-sets with 8 HDDs each and presented it as simple block devices to the repo server. No magic in here.
You're right, probably it's a ReFS problem but as we have a second volume on the same server with the same setup coming from the same storage array and this one doesn't show any problems, it could also be a problem with the ReFS API calls from the Veeam software. Veeam has much more experience with ReFS and how they trigger it so it is worth a try. Perhaps they have some special tools or commands to dig deeper into this issue. If I opena call with MS they will send me back to Veeam support, I'm pretty sure.

Regards

[Henrik.Grevelund]

Hi Oliver,

Did you extend the volume at some point ?
I had the same problem after extending the volume. After a reboot the problem was gone. I think windows ReFS had a problem using the newly added space.

[okrehan]

Hi Henrik,

no, we initially created the storage pool with the three RAID groups and we never touched it since then. It was 40TB from the beginning. We also rebooted the server multiple times so this doesn't solve the problem.

Regards,
Oliver

[soncscy]

Still makes no sense to me.

If the OS cannot even do anything, what chance does an application running on the OS have?

MS can say all they want about applications, but if Windows shows N TB free but refuses to copy a file, I think it's on MS to explain how this situation even occurred in the first place.

I think it's just not a great use of time to address this from an application level when you're clearly dealing with a file system issue. MS support can be tough, sure, but you've proven there's no need to involve any other software besides microsoft's own, and I have no doubt that any solution to allow you to copy a simple file will also allow other applications to work on the same volume.

I'm not saying there's no chance it's an application mishap, but I am saying Microsoft undoubtedly has the tools to do it faster, and you don't even need to mention other applications. Show your MS tech a simple file copy, and anything else they say just swat it down and ask them why it affects even a file copy.

As an MSP who often gets such blame, I guess it's a raw point for me to see people chasing what I see as the wrong direction when you've got clear points that Microsoft has to answer before they can point fingers.

[okrehan]

I totally agree with you that MS is probably the "better" way and I already opened a case with them. They nearly took 8h after a first phone call to send me a link to a diagnostic tool to get some data to analyze.
Nevertheless I hoped somebody else was struggling with this issue in the past and has a hint or a solution.
If Veeam support can't help here, no blame to them, but as this problem currently tears down most of the customer's backup we need a fast solution so I can't only rely on MS support.

[tsightler]

Are you sure that there are no issues with the underlying storage spaces configuration? I've seen the "no space left on device" be returned by the filesystem in cases where customers have allocated thin volumes that have run out of real, physical volume space to allocate.

[okrehan]

The pool consists of 2x 14,5TB and 1x10,9TB physical volumes, resulting in 40,0TB capacity. In this pool there is a thin provisioned 40TB virtual disk. Neither the pool nor the virtual disk are extended in the past. On top of the virtual disk there is a thin provisioned volume with 40TB that shows ~82% usage.
One can say that using a thin volume on a thin disk is not the best way to go and I agree but could that cause our kind of problem, especially if there is still enough physical space left? As we don't use overprovisioning, there shouldn't be no problem in using thin disk/volume.

[tsightler]

I wasn't attempting to speak to the merits of thin provisioning for such a use case, only that I've seen multiple cases in the field where such configuration left the customer unaware of allocation failures in the middle layers that impacted the production filesystem. It didn't really sound like you would have that case, but as there's several layers in your setup where a problem could exist I thought it was worth mentioning, especially since it's the only scenario where I've seen the behavior you describe occur in other environments and you specifically asked if anyone had seen anything like this before. If you've checked every layer and verified free space at each, then that's probably not the issue, but I've watched thin provisioning allocation issues bite so many people over the years it's difficult to not at least mention it when I see it involved.

[shawnperez]

I am going to go out on a limb and say your usable space is likely 3x10.9 and you could make a separate volume out of the remaining space on the 2 14.5 disks. If this is not just setup as a jbod; based on past experience storage spaces act like a raid controller where the smallest drive determines usable space. Most scalable storage acts this way...

[GabesVirtualWorld]

Not sure if ReFS works this way, but wasn't it in the old days that you could have a drive filled with small files where each file uses one block and the rest of the block was rendered useless if you used large blocks? In other words, a 100 files of 4K, would take up 100 x 64K instead of 100 x 4K?

[gummett]

It seems likely that @shawnperez has hit the nail on the head here (something to do with the column layout I expect), but I also wanted to mention that Microsoft don’t support Spaces on top of RAID...
https://www.google.co.uk/amp/techgenix. ... -raid/amp/
It’s straight disks only (or oddly virtual disks in Azure I believe!)

[ITP-Stan]

Also I have worked with HPE P2000 SAN systems before and they do need UNAMP/TRIM.
It's a pretty old SAN box.

[squebel]

Was any solution found to this issue? We are seeing this as well recently. Just today, a ReFS volume went "full" even though Windows said there was never less that 1.6TB free on the filesystem