Very slow file level restore and problem with permissions

[Fiskepudding]

I did a instant VM recovery, and copied the InetPub folder over the network. The folder is still containing 1,68 GB of data (59137 files, 6324 folders).
The transfer speed was much better. We got 27 Mbps.
However, this is not a practically way to restore files "on demand".

In addition, in a production environment doing a instant VM recovery is not possible. Then we would needed to setup and start a SureBackup job.

Any other ideas?
If not i guess i have to open a support ticket.

[namiko78]

Could you please explain what changes you made to your setup? I'm having similar slow restore issues, but restores are going across the LAN as far as i know...

[Fiskepudding]

Are you refferig to my post?..
I made no changes, just used different methods for restoring.

Instant VM recovery is not always a good option in a production environment, sure backup takes long time to get up and running.
I would lilke the normal "Restoring Guest files (windows) " to just work at decent speeds.

[Fiskepudding]

I recieved an answer from Veeam suport regarding this issue (case 00203915)

Support confirmed that they were aware of the problem, and provided a workaround: OtherOS FLR with FTP
Although this is a bit more timeconsuming to set up, the restore/copy speed was much faster, and totaly acceptable.

Complete answer from Support:

We are aware of the low performance while restoring multiple small-sized files via Windows FLR. This behaviour will definitely be improved in the future versions of Backup&Replication.

As a workaround here I suggest using OtherOS FLR with the FTP enabled. I've just tested it in our test environment and was able to get decent speed and restore some files with Norwegian symbols in their names. Basically, you just need to connect to the FLR aplliance's IP through FTP protocol using Windows Explorer or any FTP client you like and copy the files to original or any other server.

Could you please try this method and tell me if it works for you?

[knuter]

Same answer as I got, though I use "CopyTo" instead of "Other OS FLR with FTP". Easier and decent speed for me.
Here's my short thread by the way: Windows FLR "Restore", max number of files?

[Fiskepudding]

I tried CopyTo as well, that was also way to slow. Only thing that gave me decent speed was with FTP.

This seems like a serrious issue with a basic function that needs be addressed.
We dont restore VM's on dayly basis, but files and folders gets restored almost every day.

[AlexWhit]

I wonder if anyone can help. i am trying to restore some files to a new file server using "copy to" when this is running it seems to be really slow. it will stick on a word document for a long time then carry on. it is at about 1 - 2 MB/s and can drop to 362KB/s. At 138gb that is going to take forever.

thanks for any help

[veremin]

Hi, Alex. Are you running “Windows OS” restore? If so, can you kindly run the “Other OS” type of restore and see what speed numbers you get there.

Thanks.

[AlexWhit]

i am running a windows restore as it is from a windows server to another windows server. i will try that now thanks

[AlexWhit]

I just had a look at that but i need to preserve the Ownership and file access permissions

[veremin]

i will try that

For now this is the best strategy, indeed.

It has been confirmed that there might be some issues while restoring number of small files though Windows OS restore.

According to plan, this problem will be addressed in the upcoming product release.

Thanks.

[AlexWhit]

there are far to many files to reset all the permissions. i will just try robocopy from the old server.

thanks

[veremin]

I just had a look at that but i need to preserve the Ownership and file access permissions.

While doing Other OS restore there is a tick box called “Preserve permissions and ownerships”. So, using it in order to keep original permissions settings for recovered files.

Thanks.

[AlexWhit]

there is a tick box but it is greyed out on my restore on the "other OS" on the windows it is fine apart from it taking 24+ hours to restore

[foggy]

there is a tick box but it is greyed out on my restore

I believe this option is available for Linux hosts only.

[AlexWhit]

that will be why i will just have to put up with the slow restore. saying that it is back up to 7MB/s now

[mschlott]

Are there any known issues with windows file level restore being slow when recovering hundreds of thousands of files? I ran into this once before and after working with support it looked like it may be related to having an orphaned user with permissions to the files. In this case there were no orpaned users. The restore speed is about 1MB/S. I can restore the entire VM very quickly from the same backup so I know it is not a problem with network or disk.

[foggy]

Mike, yes, this looks like a known issue that will be addressed in v7, please review this topic above for details. Actually, this was announced by Anton in his weekly digest less than a month ago:

Speaking about networkless file level recovery (FLR) for Windows, the latest weekly report from R&D brough some excellent news that I am eager to share. We've had quite a few complaints on the forums that networkless restore of a large number of small files can be very slow. Our developers took a closer look at the issue, identified primary bottleneck, and redesigned the way how multiple small files are restored in v7. According to the most recent build notes, this change resulted in up to 40 times restore performance increase! Before somebody asks, Linux FLR had similar issues restoring a bunch of small files over the network, but those were caused by a different problem, and this was addressed in v7 as well (up to 50 times file restore performance increase). Pretty amazing stuff!

[mschlott]

This is good news. Without this improvement, I've got people telling me I need to dump Veeam for netbackup. Will V7 restores run at good speeds using v6 backups or only with backups done with V7?

Right now my only workaround is to do an instant recovery and robocopy the files to the original location. Not exactly the one click restore I need to allow operators to restore files when I am on vacation.

Thanks for the quick reply.

mike

[foggy]

Will V7 restores run at good speeds using v6 backups or only with backups done with V7?

Does not depend on the version the backup was created with.

[Fiskepudding]

Have done some more testing with V7. Same Inetpub folder, where the windows restore failed (slowed down to x KB/s and practily stopped).
This is now much better.

I have a stable sped of about 2 MB/s, so there are major improvements. My guess however is that it would be faster to instant restore the entire VM and retrieve the files.
But if time is not pressing, at least the "Restore guest fiels (windows)" actually works.

6336 folders
59138 files
1,5 GB
took 16 minutes to restore.

[mschlott]

This is not good news to me. A restore speed of 2MB/s is not acceptable when recovering 100G of data. What this means to me is that this type of recover is no longer an operator level action. It will involve escalating the recover to someone with more experience.

Is this considered to be the final resolution or is there on going work being done on this issue? Will this be logged as a bug, or is this to be accepted as the way Veeam does large restores?

[Gostev]

Yes, this is the final resolution. We are talking up to 50x times improvement from previous version here, by the way.

If you ask me, 5000 files a minute is excellent performance, especially considering you are doing the restore from compressed and deduped image level backup file. I can hardly get the same speed copying files on my netbook. And 5000 files a minute should be plenty for most file restore scenarios.

By the way, you cannot really assume you will get 2MB/s on 100GB of data, as the speed will depend on the amount of files primarily. You will get much faster speed restoring fewer of larger files. Because honestly, 60000 files in 1.5 GB are really small files. Now, if you really are in need of restoring 4 million of files (extrapolating Espen's kind of data to 100GB) on a frequent basis, that's a very interesting server workload and disaster scenario there...

[mschlott]

Thanks for the quick reply Gostev. I'll have to give this a try with my data. In my case the average file size is about 1/2 MB, so perhaps my performance will be much greater than 2MB/S. Once I get V7 configured I'll run a test restore to test my performance.

[Fiskepudding]

I think that restore of so many small files in 16 minutes is not bad.

Did test restore of c:\Windows (2003) and got 10 MB/s
16,5MB
Folders: 5903
Files: 43021

TIME: 29min 15 sec.

So things have improved in V7 no doubt.

[natrimac]

Is this still and open issue? i have the same problem with the latest patched version of v7 . i cant get over 2 mb no matter how many files i try to restore. I need to have NTFS rights preserved so other OS is a BAD option.

[foggy]

Will, as it is stated in Anton's post above, everything depends on the number of small files you're trying to restore. Are you stating that the speed for the small number of files is also that slow? Btw, could you please describe your setup briefly, especially regarding the location of Veeam B&R management server and repository. Thanks.

[masonit]

Is there a general issue in Veeam when you restore alot of small files? We just did a restore of 170 000 files ~ total 4 GB with Guest files Windows over lan (7.0.0.839). Average speed aroung 300 - 400 KB/s. No issue when restoring bigger files to the same server (25 - 30 MB/s). I know that big files restore better than small ones but should we not expect more?

\Masonit

[Gostev]

It's a general issue with and without Veeam. Try to perform a regular copy of these files over the network (not from Veeam backup), and you will see that it will not be much faster.

[masonit]

Tried copy exact same files from production vm to another server. Then there is much better performance (9 MB/s). Also tried using "copy to" during restore but the perfomance then is same as with normal restore (300 - 400 KB/s)

\Masonit