Very slow file level restore and problem with permissions

[tsightler]

I'm going to do a blog post on this but suffice it to say that the Veeam file browser, while it has improved significantly over the years, still has a ways to go to close the performance gap with tools like robocopy and xxcopy. Fortunately you can still use those tools with Veeam by access the C:\VeeamFLR folder and copying files from there instead. I expect we'll continue to enhance the performance of the file browser over time, but it's pretty hard to be these tools for now and robocopy has been a native part of Windows since 2008R2.

[davidkillingsworth]

Hi,

I am using Veeam 8 (patched) and I'm trying to restore guest files from a linux box and store them on the same Windows server on an attached NAS drive where the Veeam backup jobs are configured.

It ran for 3 hours and nothing happened. I then canceled, restarted the FLR restore and copied just a few folders and it works, but it's very very slow. There are hundreds of subfolders in hundreds of subfolders.

It's just unusable as a solution.

Is there anything I can do to speed this up. I'll try to FTP them from the linux box to Windows and see if that helps.

[gktechnology]

Just had to restore 71GB of data to a Windows Server VM from a backup repository. This was in a single folder and consisted of around 120K subfolders & files. Speed was showing at 2MB/s and it took nearly 10.5 hours to complete. Just let it run after hours and it was successful but it doesn't sound right to me. I can download from our online backup service provider quicker than this!

Any suggestions as to where I might start looking for a bottleneck?

[Shestakov]

Hello George,
As you can see in the thread, it`s a known issue when copied a lot of small files.
What version of VBR are you at?
Thanks.

[gktechnology]

7.0.0.841 if I remember correctly.

[Shestakov]

Thanks for the reply George. Try to update to the newest version. There were a some improvements in FLR performance tehre.
Thanks!

[lando_uk]

Hi, just to revive this thread.

Using file level (copy) task I'm currently restoring 25GB files (47,000 files, 5300 folders) and I'm getting 2 MB/s. So its going to take about 4hrs.

This is to recover a ransomware attack. We use both Veeam and Commvault, Commvault is much quicker restoring so this isn't a OS specific thing, its a Veeam performance thing.

Looking at the data flow, the restore accesses the vbk and vib on the backup repository, it then gets sent to the B&R master server that the file explorer is running from, and then it gets sent back to the guest server. Wouldn't it be better if the files went straight from the repository/proxy to the guest, rather than going around the houses?

I was thinking maybe its the slow RAID6 repository with forward incrimentals, but reading the forums it seems 2 MB/s is the norm for these kinds of restores, and even if I had more IO performance it wouldn't matter.

m

[Gostev]

Looking at the data flow, the restore accesses the vbk and vib on the backup repository, it then gets sent to the B&R master server that the file explorer is running from, and then it gets sent back to the guest server. Wouldn't it be better if the files went straight from the repository/proxy to the guest, rather than going around the houses?

This will be an option as a part of ROBO enhancements in v9. Please stay tuned to v9 feature announcements for more information, in fact these enhancements will be the part of next announcement I believe.

[dellock6]

You remember correctly Anton, just wait one week

[lando_uk]

You know, what we also need is a permissions only restore option on the File Explorer - was this feature we have used a lot with Commvault, when people have accidently destroyed permissions structures, but not files.

[Erhardt]

Had the same issue today (Veeam 8 U3), recovering 38GB and tons of folders and subfolders.
First the recovery were very slow (200kb/s), then it increased to 3-4 MB/s.

Next time i will try the "VeeamFLR" folder. I look forward to Veeam 9

[infused]

Bumping as I had to do this today.

60,000 files in 40gb of data.
200kb/s

This is on version 9. In the end, I just did a complete vm recovery which took 5mins or so at 250mb/s.

So is there an issue still?

[Gostev]

Need to investigate the specific case, for example the issue can be due to restore going over VIX instead of network.
Other than that, I've already seen people reporting 5-10 times faster FLR with v9 comparing to v8 here on the forum.

[infused]

This was VIX as it's impossible to do network.

What I did want to ask though is how I can use the other os with segregated networks?

For example, I tried doing this. The VM started, data was mounted, I could ping the device on the network it was deployed to (separate distributed switches), but because Veeam couldn't talk to it, it timed out.

Is there a way to get this to work, or am I doing it wrong?

[foggy]

You can configure routing to allow access from Veeam B&R console to FLR appliance.

[edv@rubner.com]

Hello everybody,

I have the following problem:
If I try to perform a restore to a share with the option "Copy To…", then I reach a speed of 25 mb / sec
If I try to perform a restore with the option "Restore -> Overwrite", then I reach a speed of 250 kb / sec

The restore I perform always on the same server
My user has Domain Admin rights

Veeam 9.0 with the last update

Does somebody has any idea?

Thanks and Greetings
Michael

[foggy]

This most likely comes down to the fact that if using the Copy To option, the backup is mounted to the remote console, not the backup server or repository, like in case of restore to original location.

[renelies]

Hi,

There actually is a relative easy solution to speed up restore (in cases where it really takes hours to start cause of the thousands of files you want to copy or restore). I'll try to take it step by step (an we're still on 8, so it might be a bit different)

1. Setup SFTP access to your Multi-OS FLR (https://www.veeam.com/kb2062)
2. Get IP of FLR (also on https://www.veeam.com/kb2062)
3. Download a statically compiled rsync (https://www.symantec.com/connect/downlo ... -rsync-307)
4. SCP it over to your FLR
5. Use rsync

It would be easier if Veeam would include rsync in the Busybox installation by default though.

[WRS2200]

Thanks for the post. This helps a similar issue I have been having.

[bertdhont]

We also had some issues with restore speed using File Level Restore.
As a workaround, I did the following:

1. 1. Start Instant VM Recovery of the VM, but do not power on.

• 2. Edit setting of the VM where you want to restore the files,
  add the disk of the restored VM (in Veeam NFS Datastore)

• 3. Now the backupdata is available in the VM, and you can restore with robocopy.

The big advantage is that restore is done locally, so there is no network traffic.
Also, if Veeam server is in another VLAN than the VM where you want to restore to, traffic doesn't need to be routed...

I wonder if it isn't possible for Veeam to implement this in the Veeam GUI as an extra option when using File level restore...

[hyvokar]

Hi!

I need to restore about 10GB worth of data to our web server.
The data consists ~1.5million files. The restore speed I see is 100-200KB/s, and the restore will take for ever.

The data is being restored from older 12x 10k sas raid5 to not-so-old 12x 10k sas raid6 via 1Gb ethernet.
We have physical backup server with 2x xeon E5507 processors and 48GB mem. The backup storage is connected via FC.

Is this kind of performance expected?

I recall asking this before, but would it be possible for veeam to rar guest windows files, so I could take the rar to the web server and unrar there, preserving file attributes.

[hyvokar]

Restore took 22 hours.

Got these error messages:

17.10.2017 6:56:03 Warning Total size on disk: 0,0 B (calculating)
17.10.2017 6:56:03 Warning Files to restore: 0 (calculating)
17.10.2017 6:56:03 Warning Folders to restore: 1 (calculating)

[lando_uk]

Yes, file restore sucks, you wont get much response on here as everyone knows how bad it is. It's the only reason we still have to keep Commvault for our file servers.

[marc.rousseau]

Yes there's a real issue with FLR even with Veeam 9.5U3.

I did a simple test : generate 50000 dummy files of 64k then do a backup.

Start a FLR job, try a "copy to..." to a local drive on Veeam server : 6MB/sec
During this FLR session, simply go to C:\VeeamFLR (where backup is mounted), copy the folder to restore, paste it in the same local drive : 30MB/sec

Someone at Veeam, maybe an advice, a clue ?

[Tarqy]

Is this being looked at by Veeam?

As above an FLR restore of 41000 files transfers at 250 KB/s (this is for both overwrite and copy to)

Windows file copy from FLR mount point to the remote server and its transfers at between 15-30MB/s (from FLR mount point to local backup server is even quicker!)

[hyvokar]

Is this being looked at by Veeam?

[Gostev]

If you have support cases open for these particular issues - then most certainly yes, and it could be even fixed already! Otherwise - unlikely.

Please remember to always include case ID with any problem reports, otherwise they are not actionable by me or my team.

Generally speaking, being the one who puts together What's New documentation, I recall significant FLR performance enhancements implemented in almost every release, addressing specific issued determined based on the previously opened support cases.

[Gostev]

As above an FLR restore of 41000 files transfers at 250 KB/s (this is for both overwrite and copy to).
Windows file copy from FLR mount point to the remote server and its transfers at between 15-30MB/s

These particular numbers sound very much like restore thought ESXi host over VIX vs. restore over network (SMB).

[cbrasga]

Just performed a File Level Restore (Overwrite) of 25,000 small files (1GB total) via SAN Snapshot for a VM. It was only copying at a speed of 100-200KB was taking hours. Way too slow.

Please fix the native FLR restore speed for large number of files.

[foggy]

Hi Cazi, please open a case with our technical support so our engineers could take a look at your environment and identify the actual reasons of why the restore took that long - otherwise we just cannot know what to fix. Thanks.