Comprehensive data protection for all workloads
Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Very slow file level restore and problem with permission

Post by Fiskepudding » Apr 16, 2013 7:21 am

I did a instant VM recovery, and copied the InetPub folder over the network. The folder is still containing 1,68 GB of data (59137 files, 6324 folders).
The transfer speed was much better. We got 27 Mbps.
However, this is not a practically way to restore files "on demand".

In addition, in a production environment doing a instant VM recovery is not possible. Then we would needed to setup and start a SureBackup job.

Any other ideas?
If not i guess i have to open a support ticket.

namiko78
Expert
Posts: 117
Liked: 4 times
Joined: Mar 03, 2011 1:49 pm
Full Name: Steven Stirling
Contact:

Re: Very slow file level restore and problem with permission

Post by namiko78 » Apr 24, 2013 1:51 am

Could you please explain what changes you made to your setup? I'm having similar slow restore issues, but restores are going across the LAN as far as i know...

Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Very slow file level restore and problem with permission

Post by Fiskepudding » Apr 25, 2013 8:04 am

Are you refferig to my post?..
I made no changes, just used different methods for restoring.

Instant VM recovery is not always a good option in a production environment, sure backup takes long time to get up and running.
I would lilke the normal "Restoring Guest files (windows) " to just work at decent speeds.

Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Very slow file level restore and problem with permission

Post by Fiskepudding » May 10, 2013 8:50 am 1 person likes this post

I recieved an answer from Veeam suport regarding this issue (case 00203915)

Support confirmed that they were aware of the problem, and provided a workaround: OtherOS FLR with FTP
Although this is a bit more timeconsuming to set up, the restore/copy speed was much faster, and totaly acceptable.

Complete answer from Support:
We are aware of the low performance while restoring multiple small-sized files via Windows FLR. This behaviour will definitely be improved in the future versions of Backup&Replication.

As a workaround here I suggest using OtherOS FLR with the FTP enabled. I've just tested it in our test environment and was able to get decent speed and restore some files with Norwegian symbols in their names. Basically, you just need to connect to the FLR aplliance's IP through FTP protocol using Windows Explorer or any FTP client you like and copy the files to original or any other server.

Could you please try this method and tell me if it works for you?

knuter
Influencer
Posts: 16
Liked: never
Joined: Nov 08, 2012 1:14 pm
Full Name: Knut R.
Contact:

Re: Very slow file level restore and problem with permission

Post by knuter » May 10, 2013 1:25 pm

Same answer as I got, though I use "CopyTo" instead of "Other OS FLR with FTP". Easier and decent speed for me.
Here's my short thread by the way: Windows FLR "Restore", max number of files?

Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Very slow file level restore and problem with permission

Post by Fiskepudding » May 13, 2013 5:58 am

I tried CopyTo as well, that was also way to slow. Only thing that gave me decent speed was with FTP.

This seems like a serrious issue with a basic function that needs be addressed.
We dont restore VM's on dayly basis, but files and folders gets restored almost every day.

AlexWhit
Enthusiast
Posts: 40
Liked: never
Joined: Jul 21, 2011 10:10 am
Full Name: Alex Whittaker
Contact:

[MERGED] : Restore Copy to

Post by AlexWhit » May 27, 2013 8:42 am

I wonder if anyone can help. i am trying to restore some files to a new file server using "copy to" when this is running it seems to be really slow. it will stick on a word document for a long time then carry on. it is at about 1 - 2 MB/s and can drop to 362KB/s. At 138gb that is going to take forever.

thanks for any help

veremin
Product Manager
Posts: 16884
Liked: 1433 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore Copy to

Post by veremin » May 27, 2013 8:50 am

Hi, Alex. Are you running “Windows OS” restore? If so, can you kindly run the “Other OS” type of restore and see what speed numbers you get there.

Thanks.

AlexWhit
Enthusiast
Posts: 40
Liked: never
Joined: Jul 21, 2011 10:10 am
Full Name: Alex Whittaker
Contact:

Re: Restore Copy to

Post by AlexWhit » May 27, 2013 8:53 am

i am running a windows restore as it is from a windows server to another windows server. i will try that now thanks

AlexWhit
Enthusiast
Posts: 40
Liked: never
Joined: Jul 21, 2011 10:10 am
Full Name: Alex Whittaker
Contact:

Re: Restore Copy to

Post by AlexWhit » May 27, 2013 9:00 am

I just had a look at that but i need to preserve the Ownership and file access permissions :(

veremin
Product Manager
Posts: 16884
Liked: 1433 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Very slow file level restore and problem with permission

Post by veremin » May 27, 2013 9:10 am

i will try that
For now this is the best strategy, indeed.

It has been confirmed that there might be some issues while restoring number of small files though Windows OS restore.

According to plan, this problem will be addressed in the upcoming product release.

Thanks.

AlexWhit
Enthusiast
Posts: 40
Liked: never
Joined: Jul 21, 2011 10:10 am
Full Name: Alex Whittaker
Contact:

Re: Very slow file level restore and problem with permission

Post by AlexWhit » May 27, 2013 9:20 am

there are far to many files to reset all the permissions. i will just try robocopy from the old server.

thanks

veremin
Product Manager
Posts: 16884
Liked: 1433 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Very slow file level restore and problem with permission

Post by veremin » May 27, 2013 9:24 am

I just had a look at that but i need to preserve the Ownership and file access permissions.

While doing Other OS restore there is a tick box called “Preserve permissions and ownerships”. So, using it in order to keep original permissions settings for recovered files.

Thanks.

AlexWhit
Enthusiast
Posts: 40
Liked: never
Joined: Jul 21, 2011 10:10 am
Full Name: Alex Whittaker
Contact:

Re: Very slow file level restore and problem with permission

Post by AlexWhit » May 27, 2013 2:44 pm

there is a tick box but it is greyed out on my restore on the "other OS" on the windows it is fine apart from it taking 24+ hours to restore

foggy
Veeam Software
Posts: 18251
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow file level restore and problem with permission

Post by foggy » May 27, 2013 3:11 pm

AlexWhit wrote:there is a tick box but it is greyed out on my restore
I believe this option is available for Linux hosts only.

AlexWhit
Enthusiast
Posts: 40
Liked: never
Joined: Jul 21, 2011 10:10 am
Full Name: Alex Whittaker
Contact:

Re: Very slow file level restore and problem with permission

Post by AlexWhit » May 27, 2013 3:17 pm

that will be why :( i will just have to put up with the slow restore. saying that it is back up to 7MB/s now

mschlott
Enthusiast
Posts: 84
Liked: 2 times
Joined: Jan 20, 2011 5:45 pm
Full Name: Mike Schlottman
Contact:

[MERGED] Slow windows file level restore with a large ammoun

Post by mschlott » Jul 24, 2013 1:43 pm

Are there any known issues with windows file level restore being slow when recovering hundreds of thousands of files? I ran into this once before and after working with support it looked like it may be related to having an orphaned user with permissions to the files. In this case there were no orpaned users. The restore speed is about 1MB/S. I can restore the entire VM very quickly from the same backup so I know it is not a problem with network or disk.

foggy
Veeam Software
Posts: 18251
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow file level restore and problem with permission

Post by foggy » Jul 24, 2013 2:08 pm

Mike, yes, this looks like a known issue that will be addressed in v7, please review this topic above for details. Actually, this was announced by Anton in his weekly digest less than a month ago:
Speaking about networkless file level recovery (FLR) for Windows, the latest weekly report from R&D brough some excellent news that I am eager to share. We've had quite a few complaints on the forums that networkless restore of a large number of small files can be very slow. Our developers took a closer look at the issue, identified primary bottleneck, and redesigned the way how multiple small files are restored in v7. According to the most recent build notes, this change resulted in up to 40 times restore performance increase! Before somebody asks, Linux FLR had similar issues restoring a bunch of small files over the network, but those were caused by a different problem, and this was addressed in v7 as well (up to 50 times file restore performance increase). Pretty amazing stuff!

mschlott
Enthusiast
Posts: 84
Liked: 2 times
Joined: Jan 20, 2011 5:45 pm
Full Name: Mike Schlottman
Contact:

Re: Very slow file level restore and problem with permission

Post by mschlott » Jul 24, 2013 2:12 pm

This is good news. Without this improvement, I've got people telling me I need to dump Veeam for netbackup. Will V7 restores run at good speeds using v6 backups or only with backups done with V7?

Right now my only workaround is to do an instant recovery and robocopy the files to the original location. Not exactly the one click restore I need to allow operators to restore files when I am on vacation.

Thanks for the quick reply.

mike

foggy
Veeam Software
Posts: 18251
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow file level restore and problem with permission

Post by foggy » Jul 24, 2013 2:21 pm

mschlott wrote:Will V7 restores run at good speeds using v6 backups or only with backups done with V7?
Does not depend on the version the backup was created with.

Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Very slow file level restore and problem with permission

Post by Fiskepudding » Aug 21, 2013 12:47 pm

Have done some more testing with V7. Same Inetpub folder, where the windows restore failed (slowed down to x KB/s and practily stopped).
This is now much better.

I have a stable sped of about 2 MB/s, so there are major improvements. My guess however is that it would be faster to instant restore the entire VM and retrieve the files.
But if time is not pressing, at least the "Restore guest fiels (windows)" actually works.

6336 folders
59138 files
1,5 GB
took 16 minutes to restore.

mschlott
Enthusiast
Posts: 84
Liked: 2 times
Joined: Jan 20, 2011 5:45 pm
Full Name: Mike Schlottman
Contact:

Re: Very slow file level restore and problem with permission

Post by mschlott » Aug 21, 2013 1:03 pm

This is not good news to me. A restore speed of 2MB/s is not acceptable when recovering 100G of data. What this means to me is that this type of recover is no longer an operator level action. It will involve escalating the recover to someone with more experience.

Is this considered to be the final resolution or is there on going work being done on this issue? Will this be logged as a bug, or is this to be accepted as the way Veeam does large restores?

Gostev
SVP, Product Management
Posts: 24787
Liked: 3520 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow file level restore and problem with permission

Post by Gostev » Aug 21, 2013 2:39 pm

Yes, this is the final resolution. We are talking up to 50x times improvement from previous version here, by the way.

If you ask me, 5000 files a minute is excellent performance, especially considering you are doing the restore from compressed and deduped image level backup file. I can hardly get the same speed copying files on my netbook. And 5000 files a minute should be plenty for most file restore scenarios.

By the way, you cannot really assume you will get 2MB/s on 100GB of data, as the speed will depend on the amount of files primarily. You will get much faster speed restoring fewer of larger files. Because honestly, 60000 files in 1.5 GB are really small files. Now, if you really are in need of restoring 4 million of files (extrapolating Espen's kind of data to 100GB) on a frequent basis, that's a very interesting server workload and disaster scenario there...

mschlott
Enthusiast
Posts: 84
Liked: 2 times
Joined: Jan 20, 2011 5:45 pm
Full Name: Mike Schlottman
Contact:

Re: Very slow file level restore and problem with permission

Post by mschlott » Aug 21, 2013 2:49 pm

Thanks for the quick reply Gostev. I'll have to give this a try with my data. In my case the average file size is about 1/2 MB, so perhaps my performance will be much greater than 2MB/S. Once I get V7 configured I'll run a test restore to test my performance.

Fiskepudding
Expert
Posts: 213
Liked: 26 times
Joined: Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen
Contact:

Re: Very slow file level restore and problem with permission

Post by Fiskepudding » Aug 21, 2013 7:45 pm 1 person likes this post

I think that restore of so many small files in 16 minutes is not bad.

Did test restore of c:\Windows (2003) and got 10 MB/s
16,5MB
Folders: 5903
Files: 43021

TIME: 29min 15 sec.

So things have improved in V7 no doubt.

natrimac
Influencer
Posts: 16
Liked: 1 time
Joined: May 13, 2013 2:37 pm
Full Name: Will Pulsifer
Contact:

Re: Very slow file level restore and problem with permission

Post by natrimac » Apr 29, 2014 8:40 pm

Is this still and open issue? i have the same problem with the latest patched version of v7 . i cant get over 2 mb no matter how many files i try to restore. I need to have NTFS rights preserved so other OS is a BAD option.

foggy
Veeam Software
Posts: 18251
Liked: 1558 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow file level restore and problem with permission

Post by foggy » Apr 30, 2014 6:07 am

Will, as it is stated in Anton's post above, everything depends on the number of small files you're trying to restore. Are you stating that the speed for the small number of files is also that slow? Btw, could you please describe your setup briefly, especially regarding the location of Veeam B&R management server and repository. Thanks.

masonit
Service Provider
Posts: 218
Liked: 15 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit » Sep 18, 2014 2:50 pm

Is there a general issue in Veeam when you restore alot of small files? We just did a restore of 170 000 files ~ total 4 GB with Guest files Windows over lan (7.0.0.839). Average speed aroung 300 - 400 KB/s. No issue when restoring bigger files to the same server (25 - 30 MB/s). I know that big files restore better than small ones but should we not expect more?

\Masonit

Gostev
SVP, Product Management
Posts: 24787
Liked: 3520 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow file level restore and problem with permission

Post by Gostev » Sep 18, 2014 9:22 pm

It's a general issue with and without Veeam. Try to perform a regular copy of these files over the network (not from Veeam backup), and you will see that it will not be much faster.

masonit
Service Provider
Posts: 218
Liked: 15 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Re: Very slow file level restore and problem with permission

Post by masonit » Sep 24, 2014 6:43 am

Tried copy exact same files from production vm to another server. Then there is much better performance (9 MB/s). Also tried using "copy to" during restore but the perfomance then is same as with normal restore (300 - 400 KB/s)

\Masonit

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 34 guests