Slow Restore

[mcz]

Hi!

I'm using Veeam Endpoint Backup and tried to restore Windows on my Notebook (Lenovo ideapad 700 with i7 and SSD) which uses 130 GB of its 190 GB SSD. There are 8 restore points but allthough the device is quick (and the backup repository has also enough throughput) my restore (bare metal and using the suggested options) takes more than 4 hours!!! This takes too much time if you have a serious problem and should bring the services back to work as fast as possible.

Is this a known issue?

Thanks, best wishes,
Michael

[Dima P.]

Hi Michael,

Can you clarify your setup please? Need to know the exact type of repository you are using? Do you have the same perfomance while running file level recovery? What is the size of the backup file? Thanks in advance.

[mcz]

Hi Dmitry,

the full backup is 40 GB big, the increments have 131 GB in sum. I have tested two backup repositories, one standard external usb disk (8 years old) and one ReFS Datastore with Raid 6 and 8 disks!! Performance wasn't better on the RAID-Repository so I would guess that this problem is related to the application architecture. I haven't tested FLR yet.

Update: From the network traffic I could see that there were times where no data has been transferred from the repository to the client. I could also observe that the throughput at the beginning of the restore was quite fast (80 MB/s) but decreased then very fast after the first 20 GB. At the end I had a throughput of approx. 9 MB/s which is very slow.

[mcz]

...meanwhile I have tested FLR, but it is slowly too.

[Dima P.]

Thanks Michael. What is the average performance of your backup job? Have you checked the said disks I/O latency during backup?

[mcz]

The average backup performance is 60 MB/s and I cannot notice any other latency or bottlenecks during backup.

[Dima P.]

Backup performance looks ok. Can you try a local file level backup to your SSD and then start a file level recovery. From VBR repository side – check your network throttling rules and make sure that you have at least one free task assigned to the repository while performing a recovery. For local USB drive, I’d check the hard disk specifications. Also please clarify how it it’s connected USB2/3, eSATA or FireWire? Thanks

[mcz]

From VBR repository side – check your network throttling rules and make sure that you have at least one free task assigned to the repository while performing a recovery.

The strange thing is that there are not network throttling rules or limited tasks per repository because I'm restoring from a simple SMB share which is not connected to any respository managed by veeam. As I said there are 8 disks in a RAID 6 and performance cannot be that bad because I am able to read blocks from this LUN at a speed of 700 MB/s. I think it must be related to the software architecture and not to hardware limitations because I have the same poor performance if I use my 7 year old USB 2.0 device or the smb share of my RAID 6 LUN.

[Dima P.]

During restore operation VEB is not only reading data from repository but writing it to the source disk. If you would like to investigate the performance further, please open a support case. Our support team can check your logs and identify the bottleneck.

[mcz]

Can you try a local file level backup to your SSD and then start a file level recovery.

ok, I did a backup to my local SSD and it took 14 min. for approx 100 GB which is quite good I think. After that I started a FLR (I took my profile folder) and noticed, that restore was within the first 1-2 minutes quite fast (80 MB/s) and then started to drop significantly. In the end I had only 7-8 MB/s!!! In a second try (but this time I tried to restore the program files) I only got 3-4 MB/s.

So we see that during backup when veeam was reading and writing from and to the same disk, performance was very good. But if we do a FLR or a bare metal recovery, performance will be very bad in the end.
From previouse restores provided by the windows-built-in backup tool recovery was always very fast and I think this should be the goal too.

I have no idea if this issue is related to the big amount of very small files or something like that...

Is it possible to open a support ticket even if I have a unpayed version (veeam endpoint backup)?

[Dima P.]

Michael,

Is it possible to open a support ticket even if I have a unpayed version (veeam endpoint backup)?

Sure, feel free to open support case directly from the product (Control Panel > Support > Technical Support) and do not forget to share the case ID.

[mcz]

ok, I have opened a new ticket, case number is 02078189

[mcz]

ok, so we investigated this case (me and support) and found out that my repository (usb 2.0) is too slow for a fast bare-metal-restore. I did several tests on different machines with a fast repository (raid 6, 8 disks) and there I was able to get a throughput somewhere around 200 mb/s, which is quite fast I think. So there seems to be no limitation if we talk about bare-metal-restore.

But the file-level-restore was ALWAYS slow, on every machine, with every repository (even if the backup is on the source side) and I never got higher than 20 MB/s. So I think this is something which can be improved in the future. I know that veeam backup & replication 9.5 had a big improvement in file-level-restore, so maybe this can be transfered to veeam endpoint (or better said veeam agent for windows).

[Dima P.]

Hi Michael,

Thanks for the heads up! We will investigate the difference between VBR and VEB internally. If possible please check the VAW Beta code – lots of improvements were implemented there, so I wonder if it works better in your environment. Thanks!