Where do I see the ReFS space savings in Veeam?

[bjdboyer]

I've set up a test environment and all the SOBR extents are ReFS on Win2016 server. The jobs are set to do weekly synthetic fulls. I've verify the job shows [fast clone]. When I go to the repository server and run a properties on the /Backups directory it shows 260GB size. This is the only thing on that drive. If I do a properties on the E: drive it shows used space of 187GB. based on that I'm assuming almost 80GB of block clone savings on that extent. But when I look in Veeam that extent is showing used space of 260GB, not apparently the reduced size due to the block cloning. Where can I see in Veeam these savings and report on them?

TIA,
Bill Boyer

[Gostev]

You do it in Windows Explorer. Right-click folder with backups, and look at Size vs. Size on disk.

And if I remember correctly, there was also a community script for gathering this info posted here around ReFS integration launch, however you'd have to search for this one yourself, since it was a long while ago already.

Thanks!

[ober72]

Hi Folks,

Luca has a great blog about this and a link to a utility called blockstat https://www.virtualtothecore.com/how-mu ... lockclone/
I found this very helpful. At the end of the day though in my humble opinion the ball on this is really in Microsoft's court. I think I heard that with 2019 REFS3 there would be some better reporting. On a side note don't mount a REFS3 volume created in Windows 2016 on Windows 2019 then try to mount back on Windows 2016 We did it in testing.. the joy of all those little surprises that come and bite you when you did not read the entire manual on changes between versions

cheers

[masonit]

Hi!

I am looking for a way to see refs space savings per folder. Instead of entire partition. Found this thread. Is still blockstat the way to go, or is there a ms supported way? Running refs on server 2019.

\Magnus

[Vitaliy S.]

Hi Magnus, there were no other tools except for the blockstat released since then.

[mweissen13]

Hi!

I am looking for a way to see refs space savings per folder. Instead of entire partition. Found this thread. Is still blockstat the way to go, or is there a ms supported way? Running refs on server 2019.

\Magnus

Hi Magnus, blockstat is apparently the only way. It has a major downside though. On a real-world repo the performance is really poor after the files become fragmented (after some months of usage). It worked great for us first, but after half a year we had to stop using it since analyzing a single folder could take 24+ hours. Note this was on rotating disks, SSDs may be different.

[ober72]

Hi Folks,

I can concur with Michael, on some repos it would take 2 days to get results.

cheers

[Vitaliy S.]

Yes, there are some known performance issues when using blockstat, but thanks for your FRs on a better way for querying this data. We will see how we can address it in our future releases.

[tdewin]

Well the prob is that it has to cross reference every chunk with all files. Reducing the amount of files can reduce the time. For example, don't run it on all the files in your repository (doesn't make any sense anyway since linking is only inside the chain).

But yes, it's not very optimal since it kind of does a "brute force" calculation on each 64k chunk without multithreading.

[mweissen13]

Hello tdewin / Timothy!
I believe you are the original author of blockstat?
I have implemented a similar tool for XFS in Java. It's running several orders of magnitude faster. I have chosen the following approach: I am querying all the block intervals via Linux's "blockstat" utility and I am merging all the metadata in RAM. Maybe that's the key difference to your implementation? As I don't think that ReFS's metadata query performance should be that much different.
I am using a variant of the following algorithm: https://www.geeksforgeeks.org/merging-intervals/
I would be happy to provide you with my source code privately, if you are interested. It's pretty small actually, only around 300 lines of code.

with kind regards,
Michael

[veeeammeupscotty]

You do it in Windows Explorer. Right-click folder with backups, and look at Size vs. Size on disk.

Does this actually work because my size and size on disk are the same, but I know fast clone is working because I have 87TB on disk and the volume is only 47TB.

[Gostev]

Sorry, savings can be seen on the ReFS volume level, see the screenshot here. Thanks!

[masonit]

Hi Magnus, blockstat is apparently the only way. It has a major downside though. On a real-world repo the performance is really poor after the files become fragmented (after some months of usage). It worked great for us first, but after half a year we had to stop using it since analyzing a single folder could take 24+ hours. Note this was on rotating disks, SSDs may be different.

Hi!

Thank you! Sounds like an issue then. How have you worked around the problem? I saw your never post. Maybe you switched to xfs instead?

\Magnus

[mweissen13]

Yes, we actually switched to XFS! But for XFS I had to come up with a custom solution for the problem. I called it "xfsblockstat". It uses the information from "filefrag" to calculate the actual space usage.

[ober72]

Hi Michael,

That is excellent work. I have just been wondering about what to do with xfs now that we are moving to it.

cheers

[masonit]

Hi!

Is there anyone with a solid solution to calculate space savings on xfs? I have tried the scripts in thread below. But the all are pretty off when I compare with the actual usage on the filesystem. If possible I would prefer to see disk usage at file level.

https://unix.stackexchange.com/question ... 499#662499

\Magnus

[DonZoomik]

As the block sharing is at file system level, there is no concept of file-level savings. Of course you can compare the savings for single file or between two files but the only reasonable place to check savings is the same as your indicated post - at job folder level.
That's because Veeam only applies block cloning at job level so you can be sure that there is no inter-job block sharing.

I'm using the scripts (modified for my use case) and it's accurate.

[masonit]

Thanks for reply. Yes I guess job folder level is the way to go. Are you using a modified version of this script?

Code: Select all

#!/bin/bash

for clientDir in `find /backups/disk-01/backups/ -mindepth 1 -maxdepth 1 -type d`
do
    echo $clientDir
    clientSpaceUsed=$(find $clientDir/*/* -xdev -type f -exec xfs_bmap -l {} + | awk '{ print $3 " " $4 }' | sort -k 1 | uniq | awk '{ print $2 }' | grep -Eo '[0-9]{1,7}' | paste -sd+ | bc | awk '{print $1*512/1024/1024/1024}')
    #block sizes of 512 bytes.  Divided by 1024 for KB.  Divided by 1024 for MB. Divided by 1024 for GB.
    echo "$clientSpaceUsed GB"
done

In the same thread someone is trying to see savings on file level with below script. But I can't get that show any good numbers.

Code: Select all

file="/mnt/cache/domains/Windows 10/vdisk1-backup.img"

du -h "$file"

mapfile -t fragments < <( filefrag -ek "$file" | tail -n +4 | cut -d ":" -f 4 | grep -oP "[0-9]+" )
sum=$(IFS=+; echo "$((${fragments[*]}))")
sum=$((sum * 1024))
sum=$(echo "$sum" | numfmt --to=iec)
echo "$sum total"

mapfile -t fragments < <( filefrag -ek "$file" | tail -n +4 | grep "shared" | cut -d ":" -f 4 | grep -oP "[0-9]+" )
sum=$(IFS=+; echo "$((${fragments[*]}))")
sum=$((sum * 1024))
sum=$(echo "$sum" | numfmt --to=iec)
echo "$sum shared"

mapfile -t fragments < <( filefrag -ek "$file" | tail -n +4 | grep -v "shared" | cut -d ":" -f 4 | grep -oP "[0-9]+" )
sum=$(IFS=+; echo "$((${fragments[*]}))")
sum=$((sum * 1024))
sum=$(echo "$sum" | numfmt --to=iec)
echo "$sum unique"

\Magnus

[DonZoomik]

The first script should implicitly targeted (/backups/disk-01/backups/) at repository root as in most circumstances, there is one folder per job.
That's the one I use as well. The modifications are minor (different tools used in pipeline) but the concept is the same.

I won't try to understand the second shell script but per-file sharing makes no sense here. While it's theoretically possible to share blocks within file (with XFS dedup tools), Veeam will not do that as in most cases deduplication is done in backup read phase and data on disk is already compressed. Veeam will only share blocks between files.

[tdewin]

*** VERY IMPORTANT DISCLAIMER *** : I'm not part of Veeam R&D, so you should consider it non-Veeam script but rather an open source, me hacking together something. Don't run it in any production environment. Basically I added MIT license: if you datacenter burns down because of this, that's on you

My guess is that xfs_bmap or filefrag is not aligning the intervals per see. Eg the shell script referenced here say interval 0..100, but in another file the interval could have only cloned half eg 0..50. In this case, the approach above does not find the overlap.

Eg this is output from my test focused on one interval between 2 files

Code: Select all

File size of /mnt/backup/Backup Job 1/forbackup.51D2023-05-15T144450_E57C.vbk is 2590121984 (2529416 blocks of 1024 bytes)
ext:     logical_offset:        physical_offset: length:   expected: flags:
   6:   202472.. 2526791:    3908460..   6232779: 2324320:             shared
   
 File size of /mnt/backup/Backup Job 1/forbackup.51D2023-06-09T104205_FFBE.vbk is 2590060544 (2529356 blocks of 1024 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
 12:   313808.. 2525699:    4020888..   6232779: 2211892

Notice how the end offset in both case is 6232779 but in the newer file, the interval only starts from 4020888, giving = 6232779-4020888 = length 2211892. In the first file 6232779-3908460 = 2324320. So for around 2211892 blocks are actually shared

Most script I have seen just make a uniq test on the line but that would actually completely miss the sharing.

I've tried to hack something together in python here : https://github.com/tdewin/ttozz/

The idea is that it will enumerates the interval not as 0..100, but expands them in smaller fixed size chunks, eg 0..100 becomes 0,10,20,30,40,50,60,70,80,90 and 0..50 becomes 0,10,20,30,40,50. In this case, the interval 0..100 and 0..50 will have 50% overlap. Although not every individual number is compared, the segment or chunks gives a pretty good rough representation of what the savings are. Especially since filesystems try to keep blocks together and we use a default block size of 1024 it is fair not to compare each and every individual block.

I defined a segment or chunk size of 256 (which is 128 kb). But I've tried -s 2048 in my test lab and it still gives very comparable results. It would be nice if somebody has a bigger test lab with multi tb VBKs to see if the numbers actually make sense

[tdewin]

Well I wish I could scratch the previous post completely. It seems @mweissen13 his/her/their suggestion of merging intervals is the best way forward. Also it seems from my test that xfs_bmap surprisingly is underreporting intervals vs filefrag. I don't know which one is right but filefrag output is inline with other tools like "ls", "du" or "df".

So made another 2 stage implementation. I could make a huge post on how it works but I need to test it first on some bigger test installations. Basically it seems smart to cache the parsed output of filefrag in a sorted interval list (sorted at volume offset vs file offset sorting that comes out of filefrag). Once you have such cache, you can quickly parse them together since the hard work of sorting the intervals is already done. If a new vbk file is created, you only need to create a filefrag cache for the new file.

Ironically, I've created a w32filefrag which is basically querying "FSCTL_GET_RETRIEVAL_POINTERS" for a single file and sorting the output all in one go. Just wondering how it would perform on 10TB vbk files. In any case, it would expect speed up because you could run it in parallel for multiple files and it should not consume GBs of memory.

[mweissen13]

Hello @tdewin; I had the same results with xfs_bmap. Filefrag seems to be more accurate.
Also, I would be interested in comparing our results.