Restore from remote site behaviour

[kennyt2000]

Hi Guys,

I have a question around the procedure for performing file restores when you have a central backup server managing backup jobs in multiple sites.

e.g
Site 1 - Veeam backup server,
SIte 2 - Backup Proxy physical server, job is performed and stored locally
SIte 3 - Backup Proxy physical server, job is performed and stored locally
SIte 4 - Backup Proxy physical server, job is performed and stored locally

Just to note we are running V6 with the latest patches.

Now we performed some fie restores and saw the following behaviours and wondered if it was correct or if we are doing something wrong

Site 2 needed files restoring so we log onto the server in site 1 which has the veeam backup software installed.
Start a file restore job and wait, and wait. I started to investogate why it was taking so long and what i could see is a VIB file was being transfered up to the server in site 1 from site 2. The file was being stored in a temp folder under the persons user account that was running the job. The VIB file was over 27GB so it took a fair amount of time to finish.

This was also the same if running a file restore job from enterprise manager.

So my question is why was the file not presented from site 2, and is it expected the files would be moved to site 1 for any restore jobs?

If not do you gus know where i need to modify this behaviour?

Thanks in advance

[Gostev]

Are you picking the correct proxy in the Restore wizard?

[kennyt2000]

Hi Gostev,

We do not get a choice for chosing a proxy when we open the restore wizard.

Step By Step
Restore Options, we select Guest Files (Windows)
Choose the virtual machine
Select the Restore Point
Enter the resore reason
Click Finish to close the wizard and start the task.

Also with the one click restore in the enterprise manager we are not seeing any choice of proxy either.

Thanks

Kenny

[Gostev]

I did not realize you were talking about Guest Files (Windows) restore, I thought this is about full VM restore (which is when the data transfer size becomes an issue). Windows FLR works by mounting the backed up disk to the backup server, so the behavior you are seeing is fully expected.

[kennyt2000]

really?

so if you have a server with 2TB of data on a disk it will want to upload the disk across the wan to the backup server to be mounted. This seems to be slightly backward and i would of expected the proxy to mount the disk and then allow you to access it.

Is there any way around this as multiple consoles / full installs of the backup server do not seem to be supported?

Thanks

Kenny

[kennyt2000]

can multiple backup servers interact with the same proxy in a remote site?
almost turning my current setup in reverse with a backup proxy in the DR/Remote site and full installs in all of the local sites?

this seems to be a fundemental flaw somehow.

[NightBird]

Same problem here.

Why don't the remote backup proxy mount the disk for FLR restore ?

[Gostev]

so if you have a server with 2TB of data on a disk it will want to upload the disk across the wan to the backup server to be mounted.

No. The only data transferred over WAN are compressed blocks belonging to the restored file.

can multiple backup servers interact with the same proxy in a remote site?

They can, but they will not be aware of tasks assigned to proxy by other backup servers, so the intelligent load balancing results may be not so "intelligent" in the end, resulting in overloaded backup proxies with too many concurrent tasks assigned by different backup servers.

Why don't the remote backup proxy mount the disk for FLR restore?

Historical reasons - FLR UI had always worked on the actual backup server. We never had backup proxies until end of last year

[kennyt2000]

No. The only data transferred over WAN are compressed blocks belonging to the restored file.

This isnt the case, we wanted to restore 30MB of data and it transfered a 27GB vib file over. Obviously a massive drain on resources for such a small task

They can, but they will not be aware of tasks assigned to proxy by other backup servers, so the intelligent load balancing results may be not so "intelligent" in the end, resulting in overloaded backup proxies with too many concurrent tasks assigned by different backup servers.

I dont think we have any other choice at this moment in time.

Historical reasons - FLR UI had always worked on the actual backup server. We never had backup proxies until end of last year

Is this being looked at in the near furture before i go and rip out our veeam infrastructure and start again with the new layout?

[richlane79]

We have also noticed this behaviour, and would like to see this addressed in a new version. It severely limits the restore ability on remote sites, especially with small WAN connections.
Don't get me wrong, the centralised \ proxy architecture in V6 is great. This would make it better!

[Gostev]

This isnt the case, we wanted to restore 30MB of data and it transfered a 27GB vib file over. Obviously a massive drain on resources for such a small task.

This most definitely should not be the case. And VIB transfer makes especially no sense, because what is being mounted is disk state at the point in time, not some given incremental backup file. While disk state is constituted from blocks of different backup files - so some requested blocks will be taken from VBK, and those updated since last full backup - from different VIBs.

The only data transferred should be a few MFT block (so that you can actually browse the file system with the Backup Browser), and the blocks belonging to the restored file. I would recommend that you open a support case and have them investigate your deployment and why are you seeing this.

Just an idea - could it be some 3rd party application trying to read/scan newly mounted volume? Think antivirus. That would explain why so many disk blocks are being requested and have to traverse the WAN.

Thanks!

[stevil]

I'm currently testing File level recovery via remote proxy on VBR 6.1.
When trying to navigate through a directory tree on our remote 'central' server, the response time is very very slow. Looking at the network usage the WAN link is being heavily utilised with every click of the mouse on each folder/subtree making restores very time consuming/difficult.

Is this a known behavior when using remote proxies or is it recommended to do file level recovery at the local site, necessitating a local full veeam install?

Cheers

[stevil]

Thanks for redirecting my post here.

This seems like a real limitation of the central server / proxy model which is yet to be addressed after 5 months+ Before I roll this out across EAME I'll ask the question again, is this looking likely to be fixed before the year is out? I really don't want to keep a full VBR server at each site just for the occasional restore :/

[Vitaliy S.]

Hi Steve,

You can use Other OS restore wizard to restore Windows guest files if you have a remote repository located over the WAN link. In this case VM virtual disks will be mounted to the appliance VM in the remote site and all restore traffic will not go over the WAN link.

Thanks!

[Starman]

Hello again gentlemen.

I'm working on an issue that came up today when I needed to fire up a restore job and check for a folder using file level recovery.

This server is in a satellite office being backed up by Veeam in my data center. It has a nice fat 100 mb coax based connection and is generally very fast. But since its 300 gigs of total size, I just use the backup proxy to a local data store.

I have the backup proxy installed on the local server and Veeam backs it up right onto storage in that site. All local and very fast. I'm still planning one using backup copy to them move that to another office later (that's just a bit off topic).

Anyhow, I fired up Veeam in my data center and tried to pull up the backups at that site with file level restore and the restore job then hung for over 12 hours after I had picked the restore point and such (I have them set to 365 and have dailies incremental for at least 2 months now since we went to 7.0)

So it hung for 12 hours (over night) until I stopped it. My only gues is that its pulling the restore info back over this pipe somehow. Is this how a restore would work using a backup proxy set up?

[Gostev]

Hello Todd, FLR has to happen through backup server, so your best bet right now is to install a copy of Veeam in the remote site, import required backup and perform FLR locally. Hope this helps.

[Starman]

Hello Todd, FLR has to happen through backup server, so your best bet right now is to install a copy of Veeam in the remote site, import required backup and perform FLR locally. Hope this helps.

Thanks Gustev, I'll set a back up copy with WAN accelerator to move them local (I need to as it is) and then just pull backups from there (this is a rare issue as it is, typical local users with sloppy mouses deleting folders months ago and not realizing it). I've just delayed since I'm still working out the backup copies in the main data center to Amazon.

Wish I could have gone to the Veeam party this year at VMworld but it was going on during my breakout session (and we had a dinner afterwards).

Cheers!

[frankive]

I am facing the same issue as other in this thread.
I have a central Veeam server in our datacenter and has lan to lan to our customer. Our typical customer run a standalone hyper-v server with a couple of vms. i have installed the proxy on the hyper-v server and setup the job through the central backup server in our datacenter. The backup goes to a local usb-drive and we use backup copy job to copy the files to our site. All good so far and nice speed.
But if i f.ex want to restore a .txt file which is 10Kb I go to the restore wizard (Windows guest). It willl then transfer the A LOT of data (100GB+) to try to mount the FLR to our central datastore.. Not very efficient. A solution of can be to restore from the backup copy job files but for very large files that would be a problem in the opposite direction again..
I can install the Veeam Console on a computer on the remote site and import the backups manual and rescan the repository but Veeam wants some RAM to make this happens and of course would need a windows server license since we don't want to have to much software on the separate VMs for some of our customer.

Is this behavior gonna change so FLR can be mounted on a proxy in the future or is this not something on the roadmap?
What about a license free LinuxVM which just could run the Veeam Backup Console on the remote site which could perform the FLR on the remote site? Then we don't need to have a windows license for this. I see that you run linux on the proxy in the sure backup/virtual lab so perhaps you have someone who could manage to get this working? Wine?

[tsightler]

I don't know what the future holds and I too hope for improvements in the future but for now there are several workarounds for this behavior:

1. For some cases you can just use the "Other OS" restore. The "Other OS" restore uses a small Linux appliance and you can pick the host where it runs so you can pick the ESX host at the remote site. The "Other OS" restore can restore files from most Windows volumes but in the current version does have issues with "special" characters which makes it less usable for non-US customers.

2. Another option is to setup a SureBackup lab and application group that contains the VMs with the files you need to restore (basically leveraging our U-AIR technology). You can then manually start the SureBackup job which will boot the VM with the files into a Surebackup lab. You can then use Windows Explorer to copy the files.

3. You can also leverage Instant Restore to restore individual files by choosing to not power on the VM after restore, then attaching the disks from the instant restored VM to another VM and copying the files.

4. Another method of using Instant Restore for this is to performance the instant restore but choose not to power on the VM after the restore, then edit the VM to remove/disconnect the network adapter, then power on the instant restore VM and login. Attach a temporary VMDK to this VM, copy the files to it, then detach, stop the instant restore, attach the VM where the files need to be restored, and copy them over.

Option #1 is probably the fastest, but has the known issues with special characters and doesn't preserve pemissions
Option #2 is probably the most complex to initially configure, but makes the task fairly easy when the time comes to use it and lets you use native file copy tools that can preserve permissions and attributes if desired
Options #3 and #4 feel like a lot of steps, but are actually pretty easy once you do it a few times and are typically the fastest when you need to restart a large amount of small files or a few very large files.

Of course none are perfect, which is why I call them workarounds.

[captainflannel]

Is there any updates or plans for a solution? It would be nice to be able to choose a backup proxy or host machine to perform the windows FLR. Seems like a huge limitation.

[foggy]

Please see the adjacent thread discussing the same request.