Comprehensive data protection for all workloads
Post Reply
kennyt2000
Influencer
Posts: 12
Liked: 2 times
Joined: Jun 25, 2009 3:05 pm
Full Name: Kenny Thornton
Contact:

Restore from remote site behaviour

Post by kennyt2000 »

Hi Guys,

I have a question around the procedure for performing file restores when you have a central backup server managing backup jobs in multiple sites.

e.g
Site 1 - Veeam backup server,
SIte 2 - Backup Proxy physical server, job is performed and stored locally
SIte 3 - Backup Proxy physical server, job is performed and stored locally
SIte 4 - Backup Proxy physical server, job is performed and stored locally

Just to note we are running V6 with the latest patches.

Now we performed some fie restores and saw the following behaviours and wondered if it was correct or if we are doing something wrong

Site 2 needed files restoring so we log onto the server in site 1 which has the veeam backup software installed.
Start a file restore job and wait, and wait. I started to investogate why it was taking so long and what i could see is a VIB file was being transfered up to the server in site 1 from site 2. The file was being stored in a temp folder under the persons user account that was running the job. The VIB file was over 27GB so it took a fair amount of time to finish.

This was also the same if running a file restore job from enterprise manager.

So my question is why was the file not presented from site 2, and is it expected the files would be moved to site 1 for any restore jobs?

If not do you gus know where i need to modify this behaviour?

Thanks in advance
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore from remote site behaviour

Post by Gostev »

Are you picking the correct proxy in the Restore wizard?
kennyt2000
Influencer
Posts: 12
Liked: 2 times
Joined: Jun 25, 2009 3:05 pm
Full Name: Kenny Thornton
Contact:

Re: Restore from remote site behaviour

Post by kennyt2000 »

Hi Gostev,

We do not get a choice for chosing a proxy when we open the restore wizard.

Step By Step
Restore Options, we select Guest Files (Windows)
Choose the virtual machine
Select the Restore Point
Enter the resore reason
Click Finish to close the wizard and start the task.

Also with the one click restore in the enterprise manager we are not seeing any choice of proxy either.

Thanks

Kenny
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore from remote site behaviour

Post by Gostev »

I did not realize you were talking about Guest Files (Windows) restore, I thought this is about full VM restore (which is when the data transfer size becomes an issue). Windows FLR works by mounting the backed up disk to the backup server, so the behavior you are seeing is fully expected.
kennyt2000
Influencer
Posts: 12
Liked: 2 times
Joined: Jun 25, 2009 3:05 pm
Full Name: Kenny Thornton
Contact:

Re: Restore from remote site behaviour

Post by kennyt2000 »

really?

so if you have a server with 2TB of data on a disk it will want to upload the disk across the wan to the backup server to be mounted. This seems to be slightly backward and i would of expected the proxy to mount the disk and then allow you to access it.

Is there any way around this as multiple consoles / full installs of the backup server do not seem to be supported?

Thanks

Kenny
kennyt2000
Influencer
Posts: 12
Liked: 2 times
Joined: Jun 25, 2009 3:05 pm
Full Name: Kenny Thornton
Contact:

Re: Restore from remote site behaviour

Post by kennyt2000 »

can multiple backup servers interact with the same proxy in a remote site?
almost turning my current setup in reverse with a backup proxy in the DR/Remote site and full installs in all of the local sites?

this seems to be a fundemental flaw somehow.
NightBird
Expert
Posts: 242
Liked: 57 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Restore from remote site behaviour

Post by NightBird »

Same problem here.

Why don't the remote backup proxy mount the disk for FLR restore ?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore from remote site behaviour

Post by Gostev » 1 person likes this post

kennyt2000 wrote:so if you have a server with 2TB of data on a disk it will want to upload the disk across the wan to the backup server to be mounted.
No. The only data transferred over WAN are compressed blocks belonging to the restored file.
kennyt2000 wrote:can multiple backup servers interact with the same proxy in a remote site?
They can, but they will not be aware of tasks assigned to proxy by other backup servers, so the intelligent load balancing results may be not so "intelligent" in the end, resulting in overloaded backup proxies with too many concurrent tasks assigned by different backup servers.
kennyt2000 wrote:Why don't the remote backup proxy mount the disk for FLR restore?
Historical reasons - FLR UI had always worked on the actual backup server. We never had backup proxies until end of last year :wink:
kennyt2000
Influencer
Posts: 12
Liked: 2 times
Joined: Jun 25, 2009 3:05 pm
Full Name: Kenny Thornton
Contact:

Re: Restore from remote site behaviour

Post by kennyt2000 »

Gostev wrote:No. The only data transferred over WAN are compressed blocks belonging to the restored file.
This isnt the case, we wanted to restore 30MB of data and it transfered a 27GB vib file over. Obviously a massive drain on resources for such a small task
Gostev wrote:They can, but they will not be aware of tasks assigned to proxy by other backup servers, so the intelligent load balancing results may be not so "intelligent" in the end, resulting in overloaded backup proxies with too many concurrent tasks assigned by different backup servers.
I dont think we have any other choice at this moment in time.
Gostev wrote:Historical reasons - FLR UI had always worked on the actual backup server. We never had backup proxies until end of last year :wink:
Is this being looked at in the near furture before i go and rip out our veeam infrastructure and start again with the new layout?
richlane79
Novice
Posts: 3
Liked: never
Joined: May 10, 2012 10:17 am
Contact:

Re: Restore from remote site behaviour

Post by richlane79 »

We have also noticed this behaviour, and would like to see this addressed in a new version. It severely limits the restore ability on remote sites, especially with small WAN connections.
Don't get me wrong, the centralised \ proxy architecture in V6 is great. This would make it better!
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore from remote site behaviour

Post by Gostev »

kennyt2000 wrote:This isnt the case, we wanted to restore 30MB of data and it transfered a 27GB vib file over. Obviously a massive drain on resources for such a small task.
This most definitely should not be the case. And VIB transfer makes especially no sense, because what is being mounted is disk state at the point in time, not some given incremental backup file. While disk state is constituted from blocks of different backup files - so some requested blocks will be taken from VBK, and those updated since last full backup - from different VIBs.

The only data transferred should be a few MFT block (so that you can actually browse the file system with the Backup Browser), and the blocks belonging to the restored file. I would recommend that you open a support case and have them investigate your deployment and why are you seeing this.

Just an idea - could it be some 3rd party application trying to read/scan newly mounted volume? Think antivirus. That would explain why so many disk blocks are being requested and have to traverse the WAN.

Thanks!
stevil
Expert
Posts: 113
Liked: 2 times
Joined: Sep 02, 2010 2:23 pm
Full Name: Steve B
Location: Manchester, UK
Contact:

[MERGED] File level recovery via remote proxy very slow

Post by stevil »

I'm currently testing File level recovery via remote proxy on VBR 6.1.
When trying to navigate through a directory tree on our remote 'central' server, the response time is very very slow. Looking at the network usage the WAN link is being heavily utilised with every click of the mouse on each folder/subtree making restores very time consuming/difficult.

Is this a known behavior when using remote proxies or is it recommended to do file level recovery at the local site, necessitating a local full veeam install?

Cheers :)
stevil
Expert
Posts: 113
Liked: 2 times
Joined: Sep 02, 2010 2:23 pm
Full Name: Steve B
Location: Manchester, UK
Contact:

Re: Restore from remote site behaviour

Post by stevil »

Thanks for redirecting my post here.

This seems like a real limitation of the central server / proxy model which is yet to be addressed after 5 months+ Before I roll this out across EAME I'll ask the question again, is this looking likely to be fixed before the year is out? I really don't want to keep a full VBR server at each site just for the occasional restore :/
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Restore from remote site behaviour

Post by Vitaliy S. »

Hi Steve,

You can use Other OS restore wizard to restore Windows guest files if you have a remote repository located over the WAN link. In this case VM virtual disks will be mounted to the appliance VM in the remote site and all restore traffic will not go over the WAN link.

Thanks!
Starman
Enthusiast
Posts: 44
Liked: 10 times
Joined: Sep 27, 2011 5:11 pm
Full Name: Todd Leavitt
Contact:

[MERGED] Restore flow question

Post by Starman »

Hello again gentlemen.

I'm working on an issue that came up today when I needed to fire up a restore job and check for a folder using file level recovery.

This server is in a satellite office being backed up by Veeam in my data center. It has a nice fat 100 mb coax based connection and is generally very fast. But since its 300 gigs of total size, I just use the backup proxy to a local data store.

I have the backup proxy installed on the local server and Veeam backs it up right onto storage in that site. All local and very fast. I'm still planning one using backup copy to them move that to another office later (that's just a bit off topic).

Anyhow, I fired up Veeam in my data center and tried to pull up the backups at that site with file level restore and the restore job then hung for over 12 hours after I had picked the restore point and such (I have them set to 365 and have dailies incremental for at least 2 months now since we went to 7.0)

So it hung for 12 hours (over night) until I stopped it. My only gues is that its pulling the restore info back over this pipe somehow. Is this how a restore would work using a backup proxy set up?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore flow question

Post by Gostev »

Hello Todd, FLR has to happen through backup server, so your best bet right now is to install a copy of Veeam in the remote site, import required backup and perform FLR locally. Hope this helps.
Starman
Enthusiast
Posts: 44
Liked: 10 times
Joined: Sep 27, 2011 5:11 pm
Full Name: Todd Leavitt
Contact:

Re: Restore flow question

Post by Starman »

Gostev wrote:Hello Todd, FLR has to happen through backup server, so your best bet right now is to install a copy of Veeam in the remote site, import required backup and perform FLR locally. Hope this helps.
Thanks Gustev, I'll set a back up copy with WAN accelerator to move them local (I need to as it is) and then just pull backups from there (this is a rare issue as it is, typical local users with sloppy mouses deleting folders months ago and not realizing it). I've just delayed since I'm still working out the backup copies in the main data center to Amazon.

Wish I could have gone to the Veeam party this year at VMworld but it was going on during my breakout session (and we had a dinner afterwards).

Cheers!
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: Restore from remote site behaviour

Post by frankive » 1 person likes this post

I am facing the same issue as other in this thread.
I have a central Veeam server in our datacenter and has lan to lan to our customer. Our typical customer run a standalone hyper-v server with a couple of vms. i have installed the proxy on the hyper-v server and setup the job through the central backup server in our datacenter. The backup goes to a local usb-drive and we use backup copy job to copy the files to our site. All good so far and nice speed.
But if i f.ex want to restore a .txt file which is 10Kb I go to the restore wizard (Windows guest). It willl then transfer the A LOT of data (100GB+) to try to mount the FLR to our central datastore.. Not very efficient. A solution of can be to restore from the backup copy job files but for very large files that would be a problem in the opposite direction again..
I can install the Veeam Console on a computer on the remote site and import the backups manual and rescan the repository but Veeam wants some RAM to make this happens and of course would need a windows server license since we don't want to have to much software on the separate VMs for some of our customer.

Is this behavior gonna change so FLR can be mounted on a proxy in the future or is this not something on the roadmap?
What about a license free LinuxVM which just could run the Veeam Backup Console on the remote site which could perform the FLR on the remote site? Then we don't need to have a windows license for this. I see that you run linux on the proxy in the sure backup/virtual lab so perhaps you have someone who could manage to get this working? Wine? :)
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Restore from remote site behaviour

Post by tsightler »

I don't know what the future holds and I too hope for improvements in the future but for now there are several workarounds for this behavior:

1. For some cases you can just use the "Other OS" restore. The "Other OS" restore uses a small Linux appliance and you can pick the host where it runs so you can pick the ESX host at the remote site. The "Other OS" restore can restore files from most Windows volumes but in the current version does have issues with "special" characters which makes it less usable for non-US customers.

2. Another option is to setup a SureBackup lab and application group that contains the VMs with the files you need to restore (basically leveraging our U-AIR technology). You can then manually start the SureBackup job which will boot the VM with the files into a Surebackup lab. You can then use Windows Explorer to copy the files.

3. You can also leverage Instant Restore to restore individual files by choosing to not power on the VM after restore, then attaching the disks from the instant restored VM to another VM and copying the files.

4. Another method of using Instant Restore for this is to performance the instant restore but choose not to power on the VM after the restore, then edit the VM to remove/disconnect the network adapter, then power on the instant restore VM and login. Attach a temporary VMDK to this VM, copy the files to it, then detach, stop the instant restore, attach the VM where the files need to be restored, and copy them over.

Option #1 is probably the fastest, but has the known issues with special characters and doesn't preserve pemissions
Option #2 is probably the most complex to initially configure, but makes the task fairly easy when the time comes to use it and lets you use native file copy tools that can preserve permissions and attributes if desired
Options #3 and #4 feel like a lot of steps, but are actually pretty easy once you do it a few times and are typically the fastest when you need to restart a large amount of small files or a few very large files.

Of course none are perfect, which is why I call them workarounds.
captainflannel
Enthusiast
Posts: 31
Liked: 5 times
Joined: Feb 27, 2012 8:53 pm
Contact:

Re: Restore from remote site behaviour

Post by captainflannel »

Is there any updates or plans for a solution? It would be nice to be able to choose a backup proxy or host machine to perform the windows FLR. Seems like a huge limitation.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Restore from remote site behaviour

Post by foggy »

Please see the adjacent thread discussing the same request.
Post Reply

Who is online

Users browsing this forum: Amazon [Bot], Bing [Bot], ENBS, Ivan239, MarioZ and 168 guests