I'm planning to migrate 100+TB of backup data from a ReFS-based repository to an XFS-based repository within an SOBR. If all the data currently living on the fast-clone supported ReFS "rehydrates", the XFS-based repository will be quickly overwhelmed given how many fast clone copies exist in the ReFS repo.
Because this is a DR repository far away from the (many) primary backup locations, re-seeding new backup jobs would be an absolute bear, so I'm considering:
Disable a backup copy job at the source
Copying the job's data folder from ReFS over to XFS (which, I assume, will "rehydrate" it)
Run duperemove (I know it may cause performance issues which Veeam support could not help with) to "dehydrate" that data again
Continue the backup copy job
Rinse and repeat for all jobs
I'd love any wisdom people could share on:
Is there a way to avoid this whole thing by telling Veeam to scan and "dehydrate" data as it copies chains over to the new XFS repo?
If not, is there a way to set block size for duperemove to best work with Veeam's XFS reflink usage (e.g. block size) once the initial dedupe is complete? I'll likely never use it again.
Once the data is copied over to the new repo, will Veeam be able to merge the first synthetic full using XFS reflink, or will it require a "true" synthetic full on the new repository prior to creating reflink-based synthetic fulls?
If the latter, will a duperemove run against the first "true" synthetic full preclude Veeam from generating reflink-based synthetic fulls going forward?
Would that not cause the backup file on XFS to not be fast-clone capable? At least with ReFS it was necesarry to be written by a new active full or a compact operation if i remember correctly!
- add new XFS-based extent to existing SOBR
- seal ReFS-extent (restore and retention will work, but now data will be written to it)
- let new data gradually lands new XFS extent
- decommission old extent ReFS, as soon as it doesn't contain any restore points
We could do that, but our retention period is incredibly long (GFS with years of retention), so we'd be keeping the old iSCSI arrays (sucking thousands of watts of power in a reasonably limited DC, with poor performance) around far too long.
I think this might be a kernel of the right solution though - do what you recommended, until we get a good synthetic full on each chain in the new XFS extent. Then, copy the old files into the new extent, rescan to make sure they live in the same place, and deduplicate only those files by reading the first synthetic full into a duperemove hash file, and running duperemove on the "old" files against the hash file based on the first synthetic full in the new XFS extent.
Sound like a reasonable way to keep all our old backups in a space-efficient way without harming the performance or alignment of the new files?
If you do try to use duperemove you're going to want to use the -b option to set the block size to 64K (assuming you were using 64K cluster size on ReFS). It's also important to remember that duperemove can use quite a bit of RAM during it's run so you'll it on small batches of files. I'd suggest moving groups of files that go together, then running duperemove and checking the results. I would suggest doing some test runs to see the results before committing to the migration.
I think the approach you outline above is workable, I have used a similar approach in my lab and I was able to migrate all of my backups from ReFS to XFS and keep the space savings (after running duperemove, obviously space was lost during the copy but was reclaimed by deupremove), but this was on a few TBs of backups in total and it was a lab, so if it didn't work I wasn't really out anything.
@Nightbird they are compressed and encrypted However, for any given chain with multiple full backups, there are plenty of dedupe candidates!
@tsightler, thanks much! The first steps will be against non-critical datasets to test the theory, and we'll get a few weeks of data and restore testing in prior to moving on to others. Of course, 3-2-1, we if it goes bad we're not sunk.
@michaelobrien, We are about to do the exact same thing, move data from ReFS to XFS in a SOBR. We're considering using duperemove as well. Can you share how things went and any learnings or best practices? Any unexpected gotchas?
@tsightler, From a Veeam point of view, is this the best/only way to do a ReFS to XFS migration? Are there any other approaches we should consider?
@cathym The simple answer is that there is no official Veeam supported way to migrate files between repos while keeping ReFS/XFS savings. Of course, if you are using simple forever forward or reverse incremental, it's no real issue, you can just copy the files since they are already aligned they will continue to work.
However, if you are using synthetic full backups or GFS points, any copying of files will expand all files to their full, independent size. I don't have any real world feedback on how well duperemove works with real world cases. In theory there should be no reason that it wouldn't work, but it's completely untested from the Veeam perspective.
By far the best recommendation in this case is to follow the advice of veremin above to add a new repo and seal the old one, allowing the new one to be populated with backups while data on the old one ages out. This is the only fully supported method at this time.
However, if for some reason this isn't possible/practical, I have been recently testing a different "creative workaround" that uses a manual procedure along with existing VBR capabilities to copy the restore points from one repo to another. It seems to work at small scale but I haven't tried to script it yet. Feel free to reach out to me directly if you are interested in discussing.
@tsightler, I'll send you an email to connect next week... our main concern is potential lack of fast clone support/candidacy once the restore points have been moved to XFS due to differences in block alignment and/or other things we're not aware of. Thanks!
Having been in the same situation, I've tried duperemove on a reasonably small backup (~250GB .vbk Files) in a test environment and found no workable set of parameters. After letting it run for about a week with 100% CPU usage i gave up and started with a fresh repo. Luckily this was no big issue for us, since we currently keep 30 incrementals and GFS backups only for 4 weeks. So after 2 months i could simply delete the old ReFS repo. I would be interested if anyone ever had success with that approach.
Thanks for sharing your experience, I was afraid of that based on my own experience with duperemove but I was hoping my experience might not be representative since my test environment had fairly limited resources I did manage to get it to run on a few TBs, but it did take quite a long time and just didn't seem like it would be easy to scale. I'm surprised it took 1 week for just 250GB though, that seems a little extreme!
True. To clarify, I aborted after a week; so I can't tell how long it would have taken.
The system I used for testing had a 6-core E5-2603 v4 @ 1.70GHz Xeon CPU and 96GB of RAM.
Hi @jamesco. Unfortunately, the procedure proved to have too many limitations to really be workable for anything but an experiment. We were able to move a few backups in one very specific scenario, but it was prone to errors and did not allow continuing the backup once it was moved, so I decided it simply was not practical for any real world case and certainly not supportable and thus stopped pursuing it.
However, for any given chain with multiple full backups, there are plenty of dedupe candidates!