FLR on 13TB VBK file - mount extremely slow

[TRE3]

Hello,

one of our customers needs to restore files from a 13TB VBK file every few days (he does not want to implement Windows shadow copies / previous versions). When mounting the file, it needs around 1 hour before he can actually explore and restore files in the FLR Explorer.

These are the specs:
- Dell Hardware backup server with Windows Server 2019, 16 core CPU, 64GB RAM (newest firmware, drivers, Windows updates)
- (Virtual) disk settings: single ReFS 64kb volume / repository (based on a Dell PERC H730P Mini RAID 60 virtual disk with 256kb stripe size / 512b block size / 12 x 8 TB NL SAS HDD)
- Repository settings: per VM backup files + data block alignment
- Job settings: optimal compression + Veeam deduplication in backup job
- no other running backup/copy/restore jobs
- VBK file includes only one huge file server with one huge partition including Windows-deduplicated files
- no running AV on backup server

What settings or configuration can we try to increase the mounting speed?

Thanks!

[Mildur]

Hi Tre3

What Veeam Version is he using? And from where is he doing the FLR Restore?
Directly on the Backup Server or from a personal machine with the console installed?

[TRE3]

The customer starts the restore from a Veeam Backup and Replication 11a that is installed on another hardware machine. The server that I mentioned is only repository server.

[Gostev]

The first step here would be to have support identify through debug logs what specific operations take most of the time. Recommendations on increasing the mounting speed can be made once the bottleneck is known.

[TRE3]

Do you think that using a remote repository server could be a problem when mounting?

[Mildur]

With V11a, restore Point is mounted on the mount server when doing a FLR restore. I would make sure, that the configured mount server for the repo is configured on the same location. If your mount server is on location number 1, and your Backup Repository on a remote location, the backup will be mounted over this slow link. This can take some time to mount.

A support case as Anton told you will definitely help you to find out the bottleneck and where you can optimise.

[Gostev]

Yes, I believe we made this change back in V10 and it was specifically to address the "remote repository" use case. Thanks to mount server, all mount traffic remains "local" while only the result (directory tree) is transmitted over WAN after the backup has been mounted. We saw 20-50x improvement in mount speed from remote repositories with this change, comparing to mounting directly to a backup console machine over WAN.

[rennerstefan]

You are saying the customer needs to restore files every other day from that large backup.
I guess it is always different files and this is due to normal file server recovery tasks right?
In that case @Gostev and @Mildur already give assistance here.
If this is more of an automation thing where the customer is accessing the same file(s) every other day, he could also think about using the "Data Integration API" to automate things.
Just to point that out in case it helps.

[TRE3]

Thank you, for the advice, @rennerstefan. But indeed, the files that need to be restored are always different ones.

[janezk]

Hi,

Sharing my experience "from the field" on the same topic:
recently we had to restore files from 10 different restore points of 32TB big FS. Data was on EMC DataDomain 6800 deduplication device on remote site that we have 2x10G connection to. Veeam 9.5U4 mounts RP where the console is, so we installed te console on proxy server on remote site to expedite mount. All mounts were taking anywhere from 1,5 to 2,5h (probably depending on the size of RP and DD dedup) Eventually we didnt't see and real difference between mounts on remote site vs mount on VBR server on primary site. I guess that the bottleneck was DD in this case and so slow, that all other components didn't really matter.
I will try to test te same changing the source from primary ReFS repo. And after the upgrade to v11a next month from both sources.

BR Janez

[TRE3]

Thank you for sharing your experience, @janezk. I'd suspect the same long mount time when using dedupe devices. But pointing on (virtual) disk on a non-dedupe device, we probably also have an issue, since disk usage and queue length are extremely high (99% / 25) on the repository server when mounting. Thus, we suspect hardware / RAID controller issues and already tried to contact Dell with this info. I'll keep you up to date on the outcome.

[RGijsen]

Yeah, welcome to the world of ReFS. As ReFS uses block cloning (you could see that as a form of deduplication), sooner or later your files will get SO fragmented it's getting slow beyond usable. You say you are using raid 60 with 12 disks, which makes (I've not had my coffee yet, my math might be off) you have only actual 8 data spindles. And they are NL SAS disks too, which are just cheap SATA disks with a SAS converter on the interface port, but with practically none of the features that makes SAS actually SAS. So yeah, being cheap on storage AND using ReFS is not a good thing. We only use ReFS on our pure SSD repository, where random IO doesn't hurt that much, but even there it gets notacebly slower over time. But one of our main repositories has 12x10TB in RAID6, and with ReFS that came down to a crawling stop sometimes when doing restores or even backups, especially when it would merge backups. Added to that the non-availability of ReFS troubleshooting tooling from within Windows itself. You only have refsutil which can do some scanning and salvage some data if you are REALLY lucky. But we've had multiple occurances where ReFS screwed up, or BSODded our system. Especially in 2016 that was a huge issue bure numerous fixes. But also in 2022 there initially were issues with ReFS again, especially when you upgraded your server to 2022 with an existing ReFS volume. It's just not mature enough. Too manby issues. When ReFS marks a file bad, it gets flagged unvisible by the filesystem. There might be a chance to get the data recovered with refsutil, but you cannot free up the space that file occupies. We ended up with about 20TB of simply unusable space on our then 60TB ReFS volume.
ReFS is great when you have Storage Spaces Direct in place. Then when something goes wrong it can recover itself from one of the other copies of the data, a bit like raid 5 or 6. But on a standalone volume, ReFS not the walhalla they tell you it is. Especially on spindles it is not overtime, and even less on slow spindles like yours. When you go for cheap storage (ie large spindles) my advice would strongly be to get MORE of them, just for the raw storage, and move to the much more mature NTFS again. You'll need more space as it doesn't do blockcloning, but backup files mainly remain sequential. And sequential transfers is where spindles shine. We moved our spindle repositories back to NTFS and we'll never look back to ReFS. It's block-cloning savings are nice, but the horror and slowness it brought to us is no match for the cost of just a few extra spindles.

[TRE3]

Just a quick reply: We solved the problem by deleting RAID 60 and creating RAID 6. Maybe, it's an issue with the PERC. Nevertheless, FLR now needs just a few minutes to open Veeam File Explorer on the 13TB VBK.

[RGijsen]

If you remained on ReFS I'd be very curious how your performance is in let's say another year. All data will again be scattered around your disks, and in the end ReFS with block cloning will be a slow as hell on spindles. And actually, on RAID6 it might get even slower as ReFS uses relatively small blocks, which might fit on the stripe size of a single disk, maybe 2. But all other data has to be read from the stripe to calculate the parities. Ah, that RAID overhead thing again. I keep telling people that, save for some specific setups, a standalone ReFS repository with block cloning should NOT be done on ReFS, but on that hardware it should remain on NTFS. On all setups we've reverted to NTFS the performance kept on par, while all ReFS repositories on spindles we've seen so far in time got really, really slow.

So what's the situation at this point? Does it all still perform well for you?

[TRE3]

@RGijsen

Indeed, it's still fast but only two months have passed. Therefore, let's look to the future!

[popjls]

Just opened a ~13TB FLR and it only took 2 mins to open on ReFS also. Has been in place for years, no issues. Going back to NTFS would be a step backwards....