Backup agent for Microsoft Windows servers and workstations (formerly Veeam Endpoint Backup FREE)
Post Reply
dgaccu
Novice
Posts: 6
Liked: never
Joined: Dec 18, 2014 5:46 pm
Full Name: Danny Williams

Very slow restore performance & error with millions of files

Post by dgaccu » Aug 03, 2018 2:37 pm

I am using Veeam Endpoint backup for our network video recorder server and in the process of doing a restore after an upgrade gone wrong. The server stores its video files very uniquely, and 60 days worth of video for 25 cameras or so results in 3.4TB of used space on the recording volume (total volume is 4x 4TB drives in RAID10 on a SmartArray controller with 2GB cache), but 1.3 million files. The video recordings are stored in small pieces with a few 1 to 4kb index/ID files per video file. I am using an 8TB USB3 Easystore drive (which is a WD Red NAS drive). Doing a volume restore for just that volume is going to take 50 hours according to the Veeam recovery agent. I am on my third attempt to restore:
1. First attempt was to boot to recovery media and do a volume level restore from there. Relatively slow at 54MB/s, whereas I would expect something closer to 80MB/s for this drive. Got a "low memory, restore may fail" message despite the server having 16GB RAM and booting direcly to the Veeam recovery ISO. It ended up erroring out after 20 hours with these messages "Not enough storage is available to process this command" "Unable to retrieve next block transmission command" "Failed to download to disk. Agent failed to process method (DataTransfer.SyncDisk)"
2. Okay, let's try something different, let's just do a file level restore inside Windows. 5MB/s and going to take nearly 7 days!
3. Okay, now we have to try volume level again, but this time we're doing it inside of Windows after normal boot. Still back to 54MB/s and going to take another two days. I hope it doesn't error again. We need this back up.

Questions: Why did we encounter this error, and how can we get better restore performance? Maybe a different kind of backup media?

Dima P.
Veeam Software
Posts: 9112
Liked: 679 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Very slow restore performance & error with millions of f

Post by Dima P. » Aug 03, 2018 4:19 pm

Hello Danny,
I am using an 8TB USB3 Easystore drive (which is a WD Red NAS drive)
That is your backup repository, right?

dgaccu
Novice
Posts: 6
Liked: never
Joined: Dec 18, 2014 5:46 pm
Full Name: Danny Williams

Re: Very slow restore performance & error with millions of f

Post by dgaccu » Aug 03, 2018 8:03 pm

We're backing up this server as a standalone server and it isn't integrated with our production B&R server. This server just has a standalone Veeam Windows Agent with the USB 3 drive directly attached as the backup destination. I'm not sure if you would still call it a backup repository if it is just a direct target for a Veeam Agent.

Dima P.
Veeam Software
Posts: 9112
Liked: 679 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Very slow restore performance & error with millions of f

Post by Dima P. » Aug 10, 2018 10:40 am

Thanks! May I ask how your volumes are organized? You have one volume per OS and another for your archive?
total volume is 4x 4TB drives in RAID10 on a SmartArray controller with 2GB cache
...
Got a "low memory, restore may fail" message despite the server having 16GB RAM and booting direcly to the Veeam recovery ISO. It ended up erroring out after 20 hours with these messages "Not enough storage is available to process this command" "Unable to retrieve next block transmission command" "Failed to download to disk. Agent failed to process method (DataTransfer.SyncDisk)"
You can open a case regarding this issue as I assume restore should go well (root cause can be identified only via recovery logs).
try something different, let's just do a file level restore inside Windows. 5MB/s and going to take nearly 7 days!
File level recovery is indeed slower that block level restore. There is a small hack: instead of using backup browser open C:\VeeamFLR folder and simply copy the desired folder - performance should be better than with backup browser.
try volume level again, but this time we're doing it inside of Windows after normal boot. Still back to 54MB/s and going to take another two days.
You can check the volumes I/O metric's while recovery is performed via resource monitor.

Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests