Very slow restore performance & error with millions of files

[dgaccu]

I am using Veeam Endpoint backup for our network video recorder server and in the process of doing a restore after an upgrade gone wrong. The server stores its video files very uniquely, and 60 days worth of video for 25 cameras or so results in 3.4TB of used space on the recording volume (total volume is 4x 4TB drives in RAID10 on a SmartArray controller with 2GB cache), but 1.3 million files. The video recordings are stored in small pieces with a few 1 to 4kb index/ID files per video file. I am using an 8TB USB3 Easystore drive (which is a WD Red NAS drive). Doing a volume restore for just that volume is going to take 50 hours according to the Veeam recovery agent. I am on my third attempt to restore:
1. First attempt was to boot to recovery media and do a volume level restore from there. Relatively slow at 54MB/s, whereas I would expect something closer to 80MB/s for this drive. Got a "low memory, restore may fail" message despite the server having 16GB RAM and booting direcly to the Veeam recovery ISO. It ended up erroring out after 20 hours with these messages "Not enough storage is available to process this command" "Unable to retrieve next block transmission command" "Failed to download to disk. Agent failed to process method (DataTransfer.SyncDisk)"
2. Okay, let's try something different, let's just do a file level restore inside Windows. 5MB/s and going to take nearly 7 days!
3. Okay, now we have to try volume level again, but this time we're doing it inside of Windows after normal boot. Still back to 54MB/s and going to take another two days. I hope it doesn't error again. We need this back up.

Questions: Why did we encounter this error, and how can we get better restore performance? Maybe a different kind of backup media?

[Dima P.]

Hello Danny,

I am using an 8TB USB3 Easystore drive (which is a WD Red NAS drive)

That is your backup repository, right?

[dgaccu]

We're backing up this server as a standalone server and it isn't integrated with our production B&R server. This server just has a standalone Veeam Windows Agent with the USB 3 drive directly attached as the backup destination. I'm not sure if you would still call it a backup repository if it is just a direct target for a Veeam Agent.

[Dima P.]

Thanks! May I ask how your volumes are organized? You have one volume per OS and another for your archive?

total volume is 4x 4TB drives in RAID10 on a SmartArray controller with 2GB cache
...
Got a "low memory, restore may fail" message despite the server having 16GB RAM and booting direcly to the Veeam recovery ISO. It ended up erroring out after 20 hours with these messages "Not enough storage is available to process this command" "Unable to retrieve next block transmission command" "Failed to download to disk. Agent failed to process method (DataTransfer.SyncDisk)"

You can open a case regarding this issue as I assume restore should go well (root cause can be identified only via recovery logs).

try something different, let's just do a file level restore inside Windows. 5MB/s and going to take nearly 7 days!

File level recovery is indeed slower that block level restore. There is a small hack: instead of using backup browser open C:\VeeamFLR folder and simply copy the desired folder - performance should be better than with backup browser.

try volume level again, but this time we're doing it inside of Windows after normal boot. Still back to 54MB/s and going to take another two days.

You can check the volumes I/O metric's while recovery is performed via resource monitor.