Veeam Backup Browser Very Slow

[wa15]

Hello all,

We have been seeing very poor performance when trying to restore guest Windows files. After completing the initial restore wizard, the backup browser starts for file level recovery and that's where it takes a minimum of 15 minutes to actually display the data (and mount the data I am assuming) and occasionally it times out with the message "Connection to Veeam Backup Service is not available. The read operation failed, see inner exception. A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond"

This is when trying to browse/restore files from a 900GB backup; and same results occur with a backup copy across couple different VM's. The CPU/Network/Memory usage on the main Veeam server all are fine. Any idea what could be causing this and what other things I should check?

[Vitaliy S.]

Hi Farshid,

What is your repository server type? Have you tried running "Other OS" FLR wizard? Do you see the same behavior?

Thank you!

[wa15]

Thanks for the response.

The repository server is a Windows server directly attached to the Veeam server running 2012R2. And the VM's being backed up are all 2012 or 2012 R2. I will try selecting "Other OS" and see what the behavior is.

[Vitaliy S.]

BTW, do you store your backups on a dedupe volume?

[wa15]

The backups are not stored on dedupe'd volumes, but some of the 2012 VM's that being backed up are deduped.

[Vitaliy S.]

Got it, let us know what performance you have if you restore via Other OS wizard. Additionally, I would suggest sending the debug logs of native FLR wizard to our technical team for investigation.

[wa15]

I think I realized what the issue is; I am trying to do file level restores from Deduped 2012 volumes, and even though the main Veeam server is 2012 R2, it doesn't have the dedup feature installed. I think that's the issue, here?

[foggy]

Correct, to be able to restore files from deduplicated volumes, backup server must be installed on Windows Server 2012 or 2012 R2 with Data Deduplication feature enabled in the OS settings (this is mentioned in the release notes document).

[wa15]

Thanks foggy.

Unfortunately I am still seeing the backup browser time out when trying to do FLR. I had a question though; when doing a FLR from the main Veeam Server located at the corporate office; the local backup at a remote site is selected. The sites are connected via a VPN tunnel. Is Veeam smart enough to know the backup is located across a WAN link and not try to download the entire backup across the WAN?

[Gostev]

Yes. Veeam will only download the required blocks of the virtual disk. The 15 minutes delay when opening Backup Browser is due to downloading virtual disk blocks containing MFT, so that we can display you the file structure. This particular step can be bypassed by using 1-Click FLR functionality of the Enteprise Manager (requires that guest indexing is enabled). Thanks!

[wa15]

Thanks Gostev,

I am seeing the same behavior with Backup Enterprise Manager unfortunately This is the error give in BEM "Remote object's stub was not found. Ref: [Type: [ Management.IFileRestoreManagementService], Uid"

I opened up a support case, hopefully they can figure it out.

thanks!

[Gostev]

I am seeing the same behavior with Backup Enterprise Manager unfortunately

By same behavior you mean it takes 15 minutes to display the file structure? This is simply impossible, as it uses local guest index file for this, and as such this process should be instant. I am really confused now.

[wa15]

I should have clarified. I can see the file structure just fine, but when I select a file and click restore or download; it takes about 10-15 minutes after the mounting backup of VM and then fails with the "Remote object's stub was not found. Ref: [Type: [ Management.IFileRestoreManagementService], Uid" error message. The ticket I have open is Case # 00520790

[Gostev]

Got it. This error is some completely different issue.

[wa15]

Interesting that this issue is only present when doing FLR through Veeam AND Enterprise Manager on VM's that are backed up remotely. FLR on backups stored locally works fine. If Enterprise Manager sees that a backup file is stored remotely on a box at a remote location that acts as repository, proxy, and WAN accelerator, how will it try to "mount" the backup? Because it looks like it gets hung when trying to mount the backup. It surely can't mount the backup on the home office over the WAN link, will it?

[Gostev]

For FLR, backups are "mounted" on the backup server where said backup is registered.

[wa15]

Hi Farshid,

What is your repository server type? Have you tried running "Other OS" FLR wizard? Do you see the same behavior?

Thank you!

And using the "Other OS" wizard seems to work, but it would be nice to be able to use the Enterprise Manager to do file level restores (:

[wa15]

@ Gostev,

Thank you. What is the best way to approach this? Do we need to have Veeam B&R onto each remote site's poxy/wan accelerator and register the backup there? It would be nice if Veeam would recognize that the backup is stored remotely (since it's coming from a remote Veeam Windows repository/proxy) and would just mount the backup there?

[Gostev]

You need to either install a backup server locally in each site, or use "Other OS" FLR.

[wa15]

Got it. Thanks Gostev.

Any plans to make it so that you have the option of choosing on which proxy the backup file gets mounted to so that we don't have to install B&R in each side? Let's say if there are 20 remote sites then you would need the entire B&R application to be able to integrate it with the Backup Enterprise Manager instead of having just one B&R server.

[Vitaliy S.]

Backups are mounted to the backup server, not proxy, but thanks for the feedback.

[Gostev]

Any plans to make it so that you have the option of choosing on which proxy the backup file gets mounted to so that we don't have to install B&R in each side? Let's say if there are 20 remote sites then you would need the entire B&R application to be able to integrate it with the Backup Enterprise Manager instead of having just one B&R server.

This has been requested a few times by now, so I am positive we will end up implementing this in the future versions.

[wa15]

Ok good to know! Thanks!

[wa15]

@ Gostev,

I don't mean to beat a dead thread, but any idea when the ability to mount a backup located at a remote repository onto a proxy at the remote site will be added to Veeam? Not expecting an exact month obviously but just if it's high enough on the priority list that it'll be sometime this year? Our team is finding this to be quite a bit of a pain and I can't quite picture how this is going to work on a larger scale. Having 20-30 backups servers (one for each remote site) just sounds painful! Though I understand they can be managed via the BEM.

[wa15]

Also, others that have a multi-site setup with Veeam and use the Backup Enterprise Manager to do FLR's, if you could chime in on your setup, that'll be appreciated. Do you have the entire Veeam B&R suite installed at each remote site and the local backup jobs to that site registered to it?

[Vitaliy S.]

Hi Farshid,

All requested features are prioritized on the number of factors, for example, on the amount of requests and the presence of workarounds. I completely agree with you that this would be a beneficial feature, so this functionality will be on our radar when planning feature set for new releases.

Thanks!

[jake1110]

This is HUGE for me. I mean ridiculously big issue. I just found this thread and was about to post about this myself.

We have 250+ remote sites that back up locally at each site. All of those sites have their own local storage for files. We index those servers so they're available to Enterprise Manager to restore from. I was under the impression that using Enterprise Manager, being indexed already, wouldn't require us to pull back all of that data and would restore from it's remote repository. We give our support staff access to Enterprise Manager, not the Veeam B&R server GUI as we don't want them using the "Other OS" option for every little file restore. That takes far too long. It would be very beneficial if the backup file could attach to either the repository, or proxy server to facilitate restores more efficiently. I know this is the plan for tape backups already, since many people don't want to attach their tape drives to their primary Veeam B&R server.

I know I can install Veeam Servers at every site and manage them all using EM, but then that's several hundred more instances to install, keep patched, install SQL on, ect.

This is very frustrating for me and further exacerbates our scalability issues with Veeam. I really do like Veeam, but it's central management of many remote sites has not been nearly as smooth as our previous solution. This is a pretty big deal to me and rather disappointing that the product always pulls that much data back, especially on slower connections(T1).

[Gostev]

This is one of those features that devs kept postponing due to complexity and availability of workarounds, but I am sure this will be implemented sooner rather than later, because there is a lot of pressure for this feature from our sales engineers as well.

[jake1110]

Thanks for the response and acknowledgement Gostev.

I'm testing out increasing the timeout value for FLR, using the key "remotingtimeout" under HKLM\Software\Veeam\Veeam Backup and Replication.

Slow is better than nothing for me. Most of our restores are relatively small, so hopefully this will help out.

[wa15]

@ Jake, just wondering if increasing the timeout value for the FLR worked for you?