Very slow file level restore and problem with permissions

Availability for the Always-On Enterprise

Re: Very slow file level restore and problem with permission

Veeam Logoby AlexWhit » Mon May 27, 2013 3:17 pm

that will be why :( i will just have to put up with the slow restore. saying that it is back up to 7MB/s now
AlexWhit
Enthusiast
 
Posts: 36
Liked: never
Joined: Thu Jul 21, 2011 10:10 am
Full Name: Alex Whittaker

[MERGED] Slow windows file level restore with a large ammoun

Veeam Logoby mschlott » Wed Jul 24, 2013 1:43 pm

Are there any known issues with windows file level restore being slow when recovering hundreds of thousands of files? I ran into this once before and after working with support it looked like it may be related to having an orphaned user with permissions to the files. In this case there were no orpaned users. The restore speed is about 1MB/S. I can restore the entire VM very quickly from the same backup so I know it is not a problem with network or disk.
mschlott
Enthusiast
 
Posts: 83
Liked: 2 times
Joined: Thu Jan 20, 2011 5:45 pm
Full Name: Mike Schlottman

Re: Very slow file level restore and problem with permission

Veeam Logoby foggy » Wed Jul 24, 2013 2:08 pm

Mike, yes, this looks like a known issue that will be addressed in v7, please review this topic above for details. Actually, this was announced by Anton in his weekly digest less than a month ago:

Speaking about networkless file level recovery (FLR) for Windows, the latest weekly report from R&D brough some excellent news that I am eager to share. We've had quite a few complaints on the forums that networkless restore of a large number of small files can be very slow. Our developers took a closer look at the issue, identified primary bottleneck, and redesigned the way how multiple small files are restored in v7. According to the most recent build notes, this change resulted in up to 40 times restore performance increase! Before somebody asks, Linux FLR had similar issues restoring a bunch of small files over the network, but those were caused by a different problem, and this was addressed in v7 as well (up to 50 times file restore performance increase). Pretty amazing stuff!
foggy
Veeam Software
 
Posts: 14904
Liked: 1096 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Very slow file level restore and problem with permission

Veeam Logoby mschlott » Wed Jul 24, 2013 2:12 pm

This is good news. Without this improvement, I've got people telling me I need to dump Veeam for netbackup. Will V7 restores run at good speeds using v6 backups or only with backups done with V7?

Right now my only workaround is to do an instant recovery and robocopy the files to the original location. Not exactly the one click restore I need to allow operators to restore files when I am on vacation.

Thanks for the quick reply.

mike
mschlott
Enthusiast
 
Posts: 83
Liked: 2 times
Joined: Thu Jan 20, 2011 5:45 pm
Full Name: Mike Schlottman

Re: Very slow file level restore and problem with permission

Veeam Logoby foggy » Wed Jul 24, 2013 2:21 pm

mschlott wrote:Will V7 restores run at good speeds using v6 backups or only with backups done with V7?

Does not depend on the version the backup was created with.
foggy
Veeam Software
 
Posts: 14904
Liked: 1096 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Very slow file level restore and problem with permission

Veeam Logoby Fiskepudding » Wed Aug 21, 2013 12:47 pm

Have done some more testing with V7. Same Inetpub folder, where the windows restore failed (slowed down to x KB/s and practily stopped).
This is now much better.

I have a stable sped of about 2 MB/s, so there are major improvements. My guess however is that it would be faster to instant restore the entire VM and retrieve the files.
But if time is not pressing, at least the "Restore guest fiels (windows)" actually works.

6336 folders
59138 files
1,5 GB
took 16 minutes to restore.
Fiskepudding
Expert
 
Posts: 213
Liked: 26 times
Joined: Wed Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen

Re: Very slow file level restore and problem with permission

Veeam Logoby mschlott » Wed Aug 21, 2013 1:03 pm

This is not good news to me. A restore speed of 2MB/s is not acceptable when recovering 100G of data. What this means to me is that this type of recover is no longer an operator level action. It will involve escalating the recover to someone with more experience.

Is this considered to be the final resolution or is there on going work being done on this issue? Will this be logged as a bug, or is this to be accepted as the way Veeam does large restores?
mschlott
Enthusiast
 
Posts: 83
Liked: 2 times
Joined: Thu Jan 20, 2011 5:45 pm
Full Name: Mike Schlottman

Re: Very slow file level restore and problem with permission

Veeam Logoby Gostev » Wed Aug 21, 2013 2:39 pm

Yes, this is the final resolution. We are talking up to 50x times improvement from previous version here, by the way.

If you ask me, 5000 files a minute is excellent performance, especially considering you are doing the restore from compressed and deduped image level backup file. I can hardly get the same speed copying files on my netbook. And 5000 files a minute should be plenty for most file restore scenarios.

By the way, you cannot really assume you will get 2MB/s on 100GB of data, as the speed will depend on the amount of files primarily. You will get much faster speed restoring fewer of larger files. Because honestly, 60000 files in 1.5 GB are really small files. Now, if you really are in need of restoring 4 million of files (extrapolating Espen's kind of data to 100GB) on a frequent basis, that's a very interesting server workload and disaster scenario there...
Gostev
Veeam Software
 
Posts: 21444
Liked: 2362 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Very slow file level restore and problem with permission

Veeam Logoby mschlott » Wed Aug 21, 2013 2:49 pm

Thanks for the quick reply Gostev. I'll have to give this a try with my data. In my case the average file size is about 1/2 MB, so perhaps my performance will be much greater than 2MB/S. Once I get V7 configured I'll run a test restore to test my performance.
mschlott
Enthusiast
 
Posts: 83
Liked: 2 times
Joined: Thu Jan 20, 2011 5:45 pm
Full Name: Mike Schlottman

Re: Very slow file level restore and problem with permission

Veeam Logoby Fiskepudding » Wed Aug 21, 2013 7:45 pm 1 person likes this post

I think that restore of so many small files in 16 minutes is not bad.

Did test restore of c:\Windows (2003) and got 10 MB/s
16,5MB
Folders: 5903
Files: 43021

TIME: 29min 15 sec.

So things have improved in V7 no doubt.
Fiskepudding
Expert
 
Posts: 213
Liked: 26 times
Joined: Wed Feb 01, 2012 7:24 am
Full Name: Espen Dykesteen

Re: Very slow file level restore and problem with permission

Veeam Logoby natrimac » Tue Apr 29, 2014 8:40 pm

Is this still and open issue? i have the same problem with the latest patched version of v7 . i cant get over 2 mb no matter how many files i try to restore. I need to have NTFS rights preserved so other OS is a BAD option.
natrimac
Influencer
 
Posts: 16
Liked: 1 time
Joined: Mon May 13, 2013 2:37 pm
Full Name: Will Pulsifer

Re: Very slow file level restore and problem with permission

Veeam Logoby foggy » Wed Apr 30, 2014 6:07 am

Will, as it is stated in Anton's post above, everything depends on the number of small files you're trying to restore. Are you stating that the speed for the small number of files is also that slow? Btw, could you please describe your setup briefly, especially regarding the location of Veeam B&R management server and repository. Thanks.
foggy
Veeam Software
 
Posts: 14904
Liked: 1096 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Very slow file level restore and problem with permission

Veeam Logoby masonit » Thu Sep 18, 2014 2:50 pm

Is there a general issue in Veeam when you restore alot of small files? We just did a restore of 170 000 files ~ total 4 GB with Guest files Windows over lan (7.0.0.839). Average speed aroung 300 - 400 KB/s. No issue when restoring bigger files to the same server (25 - 30 MB/s). I know that big files restore better than small ones but should we not expect more?

\Masonit
masonit
Service Provider
 
Posts: 141
Liked: 10 times
Joined: Tue Oct 09, 2012 2:30 pm
Full Name: Magnus Andersson

Re: Very slow file level restore and problem with permission

Veeam Logoby Gostev » Thu Sep 18, 2014 9:22 pm

It's a general issue with and without Veeam. Try to perform a regular copy of these files over the network (not from Veeam backup), and you will see that it will not be much faster.
Gostev
Veeam Software
 
Posts: 21444
Liked: 2362 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Very slow file level restore and problem with permission

Veeam Logoby masonit » Wed Sep 24, 2014 6:43 am

Tried copy exact same files from production vm to another server. Then there is much better performance (9 MB/s). Also tried using "copy to" during restore but the perfomance then is same as with normal restore (300 - 400 KB/s)

\Masonit
masonit
Service Provider
 
Posts: 141
Liked: 10 times
Joined: Tue Oct 09, 2012 2:30 pm
Full Name: Magnus Andersson

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: benks, Bing [Bot], Google [Bot] and 32 guests