Comprehensive data protection for all workloads
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Very slow file level restore and problem with permission

Post by tsightler »

I'm going to do a blog post on this but suffice it to say that the Veeam file browser, while it has improved significantly over the years, still has a ways to go to close the performance gap with tools like robocopy and xxcopy. Fortunately you can still use those tools with Veeam by access the C:\VeeamFLR folder and copying files from there instead. I expect we'll continue to enhance the performance of the file browser over time, but it's pretty hard to be these tools for now and robocopy has been a native part of Windows since 2008R2.
davidkillingsworth
Enthusiast
Posts: 56
Liked: 10 times
Joined: Nov 21, 2011 5:41 am
Location: Hong Kong
Contact:

Re: Very slow file level restore and problem with permission

Post by davidkillingsworth »

Hi,

I am using Veeam 8 (patched) and I'm trying to restore guest files from a linux box and store them on the same Windows server on an attached NAS drive where the Veeam backup jobs are configured.

It ran for 3 hours and nothing happened. I then canceled, restarted the FLR restore and copied just a few folders and it works, but it's very very slow. There are hundreds of subfolders in hundreds of subfolders.

It's just unusable as a solution.

Is there anything I can do to speed this up. I'll try to FTP them from the linux box to Windows and see if that helps.
gktechnology
Novice
Posts: 3
Liked: never
Joined: Apr 28, 2015 8:55 pm
Full Name: George Bradley
Contact:

[MERGED] File level restore slow

Post by gktechnology »

Just had to restore 71GB of data to a Windows Server VM from a backup repository. This was in a single folder and consisted of around 120K subfolders & files. Speed was showing at 2MB/s and it took nearly 10.5 hours to complete. Just let it run after hours and it was successful but it doesn't sound right to me. I can download from our online backup service provider quicker than this!

Any suggestions as to where I might start looking for a bottleneck?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Very slow file level restore and problem with permission

Post by Shestakov »

Hello George,
As you can see in the thread, it`s a known issue when copied a lot of small files.
What version of VBR are you at?
Thanks.
gktechnology
Novice
Posts: 3
Liked: never
Joined: Apr 28, 2015 8:55 pm
Full Name: George Bradley
Contact:

Re: Very slow file level restore and problem with permission

Post by gktechnology »

7.0.0.841 if I remember correctly.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Very slow file level restore and problem with permission

Post by Shestakov »

Thanks for the reply George. Try to update to the newest version. There were a some improvements in FLR performance tehre.
Thanks!
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Very slow file level restore and problem with permission

Post by lando_uk »

Hi, just to revive this thread.

Using file level (copy) task I'm currently restoring 25GB files (47,000 files, 5300 folders) and I'm getting 2 MB/s. So its going to take about 4hrs.

This is to recover a ransomware attack. We use both Veeam and Commvault, Commvault is much quicker restoring so this isn't a OS specific thing, its a Veeam performance thing.

Looking at the data flow, the restore accesses the vbk and vib on the backup repository, it then gets sent to the B&R master server that the file explorer is running from, and then it gets sent back to the guest server. Wouldn't it be better if the files went straight from the repository/proxy to the guest, rather than going around the houses?

I was thinking maybe its the slow RAID6 repository with forward incrimentals, but reading the forums it seems 2 MB/s is the norm for these kinds of restores, and even if I had more IO performance it wouldn't matter.

m
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow file level restore and problem with permission

Post by Gostev »

lando_uk wrote:Looking at the data flow, the restore accesses the vbk and vib on the backup repository, it then gets sent to the B&R master server that the file explorer is running from, and then it gets sent back to the guest server. Wouldn't it be better if the files went straight from the repository/proxy to the guest, rather than going around the houses?
This will be an option as a part of ROBO enhancements in v9. Please stay tuned to v9 feature announcements for more information, in fact these enhancements will be the part of next announcement I believe.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Very slow file level restore and problem with permission

Post by dellock6 »

You remember correctly Anton, just wait one week ;)
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Very slow file level restore and problem with permission

Post by lando_uk »

You know, what we also need is a permissions only restore option on the File Explorer - was this feature we have used a lot with Commvault, when people have accidently destroyed permissions structures, but not files.
Erhardt
Influencer
Posts: 18
Liked: never
Joined: Aug 23, 2013 2:25 pm
Full Name: Erhardt
Contact:

Re: Very slow file level restore and problem with permission

Post by Erhardt »

Had the same issue today (Veeam 8 U3), recovering 38GB and tons of folders and subfolders.
First the recovery were very slow (200kb/s), then it increased to 3-4 MB/s.

Next time i will try the "VeeamFLR" folder. I look forward to Veeam 9 :P
infused
Service Provider
Posts: 178
Liked: 13 times
Joined: Apr 20, 2013 9:25 am
Full Name: Hayden Kirk
Contact:

Re: Very slow file level restore and problem with permission

Post by infused »

Bumping as I had to do this today.

60,000 files in 40gb of data.
200kb/s

This is on version 9. In the end, I just did a complete vm recovery which took 5mins or so at 250mb/s.

So is there an issue still?
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow file level restore and problem with permission

Post by Gostev »

Need to investigate the specific case, for example the issue can be due to restore going over VIX instead of network.
Other than that, I've already seen people reporting 5-10 times faster FLR with v9 comparing to v8 here on the forum.
infused
Service Provider
Posts: 178
Liked: 13 times
Joined: Apr 20, 2013 9:25 am
Full Name: Hayden Kirk
Contact:

Re: Very slow file level restore and problem with permission

Post by infused »

This was VIX as it's impossible to do network.

What I did want to ask though is how I can use the other os with segregated networks?

For example, I tried doing this. The VM started, data was mounted, I could ping the device on the network it was deployed to (separate distributed switches), but because Veeam couldn't talk to it, it timed out.

Is there a way to get this to work, or am I doing it wrong?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow file level restore and problem with permission

Post by foggy »

You can configure routing to allow access from Veeam B&R console to FLR appliance.
edv@rubner.com
Influencer
Posts: 17
Liked: 5 times
Joined: Jun 26, 2015 1:35 pm
Full Name: Rubner EDV
Contact:

Re: Very slow file level restore and problem with permission

Post by edv@rubner.com »

Hello everybody,

I have the following problem:
If I try to perform a restore to a share with the option "Copy To…", then I reach a speed of 25 mb / sec
If I try to perform a restore with the option "Restore -> Overwrite", then I reach a speed of 250 kb / sec

The restore I perform always on the same server
My user has Domain Admin rights

Veeam 9.0 with the last update

Does somebody has any idea?

Thanks and Greetings
Michael
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow file level restore and problem with permission

Post by foggy »

This most likely comes down to the fact that if using the Copy To option, the backup is mounted to the remote console, not the backup server or repository, like in case of restore to original location.
renelies
Lurker
Posts: 1
Liked: never
Joined: Jan 11, 2017 6:37 am
Full Name: Rene van Lieshout
Contact:

Re: Very slow file level restore and problem with permission

Post by renelies »

Hi,

There actually is a relative easy solution to speed up restore (in cases where it really takes hours to start cause of the thousands of files you want to copy or restore). I'll try to take it step by step (an we're still on 8, so it might be a bit different)

1. Setup SFTP access to your Multi-OS FLR (https://www.veeam.com/kb2062)
2. Get IP of FLR (also on https://www.veeam.com/kb2062)
3. Download a statically compiled rsync (https://www.symantec.com/connect/downlo ... -rsync-307)
4. SCP it over to your FLR
5. Use rsync :)

It would be easier if Veeam would include rsync in the Busybox installation by default though.
WRS2200
Enthusiast
Posts: 28
Liked: 3 times
Joined: Aug 06, 2015 8:21 pm
Full Name: Weston Strom
Contact:

Re: Very slow file level restore and problem with permission

Post by WRS2200 »

Thanks for the post. This helps a similar issue I have been having.
bertdhont
Service Provider
Posts: 45
Liked: 5 times
Joined: Nov 08, 2013 2:53 pm
Full Name: Bert D'hont
Contact:

Re: Very slow file level restore and problem with permission

Post by bertdhont »

We also had some issues with restore speed using File Level Restore.
As a workaround, I did the following:
  1. 1. Start Instant VM Recovery of the VM, but do not power on.
  • 2. Edit setting of the VM where you want to restore the files,
    add the disk of the restored VM (in Veeam NFS Datastore)
  • 3. Now the backupdata is available in the VM, and you can restore with robocopy.
The big advantage is that restore is done locally, so there is no network traffic.
Also, if Veeam server is in another VLAN than the VM where you want to restore to, traffic doesn't need to be routed...

I wonder if it isn't possible for Veeam to implement this in the Veeam GUI as an extra option when using File level restore...
hyvokar
Veteran
Posts: 406
Liked: 29 times
Joined: Nov 21, 2014 10:05 pm
Contact:

[MERGED] Slow restore speed

Post by hyvokar »

Hi!

I need to restore about 10GB worth of data to our web server.
The data consists ~1.5million files. The restore speed I see is 100-200KB/s, and the restore will take for ever.

The data is being restored from older 12x 10k sas raid5 to not-so-old 12x 10k sas raid6 via 1Gb ethernet.
We have physical backup server with 2x xeon E5507 processors and 48GB mem. The backup storage is connected via FC.

Is this kind of performance expected?

I recall asking this before, but would it be possible for veeam to rar guest windows files, so I could take the rar to the web server and unrar there, preserving file attributes.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE
hyvokar
Veteran
Posts: 406
Liked: 29 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Slow restore speed

Post by hyvokar »

Restore took 22 hours.

Got these error messages:

17.10.2017 6:56:03 Warning Total size on disk: 0,0 B (calculating)
17.10.2017 6:56:03 Warning Files to restore: 0 (calculating)
17.10.2017 6:56:03 Warning Folders to restore: 1 (calculating)
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE
lando_uk
Veteran
Posts: 371
Liked: 32 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Slow restore speed

Post by lando_uk »

Yes, file restore sucks, you wont get much response on here as everyone knows how bad it is. It's the only reason we still have to keep Commvault for our file servers.
marc.rousseau
Enthusiast
Posts: 26
Liked: 11 times
Joined: May 01, 2014 9:48 am
Full Name: marc.rousseau
Contact:

Re: Very slow file level restore and problem with permission

Post by marc.rousseau »

Yes there's a real issue with FLR even with Veeam 9.5U3.

I did a simple test : generate 50000 dummy files of 64k then do a backup.

Start a FLR job, try a "copy to..." to a local drive on Veeam server : 6MB/sec
During this FLR session, simply go to C:\VeeamFLR (where backup is mounted), copy the folder to restore, paste it in the same local drive : 30MB/sec

Someone at Veeam, maybe an advice, a clue ?
Tarqy
Service Provider
Posts: 42
Liked: 5 times
Joined: Aug 08, 2014 1:51 pm
Full Name: Barry Knox
Contact:

Re: Very slow file level restore and problem with permission

Post by Tarqy » 1 person likes this post

Is this being looked at by Veeam?

As above an FLR restore of 41000 files transfers at 250 KB/s (this is for both overwrite and copy to)

Windows file copy from FLR mount point to the remote server and its transfers at between 15-30MB/s (from FLR mount point to local backup server is even quicker!)
VMCE
hyvokar
Veteran
Posts: 406
Liked: 29 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Very slow file level restore and problem with permission

Post by hyvokar »

Tarqy wrote:Is this being looked at by Veeam?
:D
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow file level restore and problem with permission

Post by Gostev »

If you have support cases open for these particular issues - then most certainly yes, and it could be even fixed already! Otherwise - unlikely.

Please remember to always include case ID with any problem reports, otherwise they are not actionable by me or my team.

Generally speaking, being the one who puts together What's New documentation, I recall significant FLR performance enhancements implemented in almost every release, addressing specific issued determined based on the previously opened support cases.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow file level restore and problem with permission

Post by Gostev »

Tarqy wrote:As above an FLR restore of 41000 files transfers at 250 KB/s (this is for both overwrite and copy to).
Windows file copy from FLR mount point to the remote server and its transfers at between 15-30MB/s
These particular numbers sound very much like restore thought ESXi host over VIX vs. restore over network (SMB).
cbrasga
Influencer
Posts: 16
Liked: 2 times
Joined: Apr 27, 2013 2:09 am
Full Name: Cazi Brasga
Contact:

Re: Very slow file level restore and problem with permission

Post by cbrasga »

Just performed a File Level Restore (Overwrite) of 25,000 small files (1GB total) via SAN Snapshot for a VM. It was only copying at a speed of 100-200KB was taking hours. Way too slow.

Please fix the native FLR restore speed for large number of files.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Very slow file level restore and problem with permission

Post by foggy »

Hi Cazi, please open a case with our technical support so our engineers could take a look at your environment and identify the actual reasons of why the restore took that long - otherwise we just cannot know what to fix. Thanks.
Post Reply

Who is online

Users browsing this forum: FelixW, Semrush [Bot] and 166 guests