Host-based backup of VMware vSphere VMs.
Post Reply
dlowe
Influencer
Posts: 19
Liked: 2 times
Joined: Mar 16, 2012 3:20 pm
Full Name: Darryl Lowe
Contact:

Veeam Feature Enhancement Request - File Level Restores

Post by dlowe »

We have a multi-tenant environment whereby there is no direct network connection between the Veeam server and the customer's VM's, so we do 'application aware' backups via VIX / hotadd, which essentially does the backup over 8Gb Fibre Channel disk channel. Recently we had to restore a customer's 12GB File Share folder (20,000 files) back to their production environment, which has become more common these days with things like the Crypto virus... When attempting to use the File Level Restore wizard 'Guest files (Windows)' it took over 8.5 hours to only restore 42% of the files, so we had to use an alternate method to restore the files instead of the simplicity of the File Level Restore wizard (basically we did an instant recovery of the customer's file share server VM, added a temporary disk to the recovered server, copied the files from the restored server to the newly added disk, detached the disk from the recovered VM, attached the disk to the production VM, and copied the files back to their original location).

In order to find out what is causing the extreme slow recovery of the files, I opened a ticket and worked with Veeam Technical support. Please refer to Veeam Support Case # 00937649 for details.
What we found out is that the File Level Restore wizard was not necessarily designed specifically for multi-tenant environments with no direct network connection betweeen the Veeam server & the production VM server, so these extremely slow recovery speeds are likely as per the design.

*So, my feature enhancement suggestion would be for Veeam to add it to their roadmap for a future release, to have an additional option in the 'File Level Restore' wizard, that allows for the restore of files back to the original VM directly over the SAN (Fiber Channel or whatever the SAN fabric is), via the 'Hotadd / VIX' technology, similar to how is currently used for 'application aware' backups in this type of environment/scenario.

Thanks,
Darryl
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam Feature Enhancement Request - File Level Restores

Post by foggy »

Darryl, FLR via VIX is already possible (and was used during your restore operation, since there's no connection to the original VM). May I ask you whether the Veeam B&R server is located remotely to the production environment? Please note that for FLR, backups are mounted on the backup server itself, so data is transferred back and forth in this case (not optimal but historically established, and we're working on improvements in this area). You can either use Other OS wizard that works differently or setup a separate Veeam B&R console in each site, specifically for such restores. Thanks.
dlowe
Influencer
Posts: 19
Liked: 2 times
Joined: Mar 16, 2012 3:20 pm
Full Name: Darryl Lowe
Contact:

Re: Veeam Feature Enhancement Request - File Level Restores

Post by dlowe »

Yeah I knew VIX had to be involved with the restore, but it's the hotadd part that I believe is not happening & the file restores are going over the VMware management network (VIX). It would be nice if there was an option to have it utilize something like a reverse hotadd and restore via attached disk. If there was some way/option/checkbox in the File Restore Wizard to temporarily have the restored disk attach or mount directly to the production VM (instead of the Veeam B&R server), do the restore direct from disk (Fiber Channel) instead of via network, and have it detach once completed, it would be multiple times faster.

The Veeam B&R server is located locally and connected to the same networking as the production environment. Note that restoring to a folder on the Veeam server was much faster, but still would be too slow if we had to restore more than 12GB of data. However I think the VMware VIX management network is the bottleneck when doing file restores, on top of the multiple connection/communication points used which may also be slowing it down when the files are smaller but numerous...

When I worked with Veeam Support we tried 'Other OS', but it also goes over the network instead of direct disk and so it wasn't super fast either. Setting up separate Veeam B&R consoles in each customer environment would be too much complexity, effort and additional VMware resources used. Hence my request for the hotadd based file restore feature enhancement request :)
Maybe instead of hotadd (if that is not possible to do into a production VM), what if the option/checkbox spun up a temporary appliance on the same ESXi host that the production VM is on, attached the recovered disk to this appliance, set the network on the same port group with VMXNET3, and recovered basically within the same fast network between the appliance & production file server VM without ever leaving the ESXi host..!? It would still require hotadd to another VM besides the Veeam server to accomplish this...
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Veeam Feature Enhancement Request - File Level Restores

Post by chrisdearden »

That sounds like the other OS FLR , but with more dynamic networking (and placement of the recovery helper )

Do you run DHCP on your production network ? You'd need a way of allocating a free IP on the same subnet as the restore target , as well as opening ports to the backup server for a control channel

I dont know if hot adding restored disk would cause issues with duplicate disk ID's in windows - hence requiring the restore helper VM.

The VIX API isn't good for transferring a lot of data at the best of times.

Also if we instant restore a disk , its presented via the Vpower NFS mount , so there is always going to be some network involved, unless you did a full restore of the disk via hotadd , which would seem a lot , just to recover some files.
dlowe
Influencer
Posts: 19
Liked: 2 times
Joined: Mar 16, 2012 3:20 pm
Full Name: Darryl Lowe
Contact:

Re: Veeam Feature Enhancement Request - File Level Restores

Post by dlowe »

chrisdearden wrote:Also if we instant restore a disk , its presented via the Vpower NFS mount , so there is always going to be some network involved, unless you did a full restore of the disk via hotadd , which would seem a lot , just to recover some files.
Recovering files was no simple task recently whereby 12GB would have taken over 24hours via the File Level Restore wizard. The hotadd based file recovery is the feature possibility I am most interested in seeing, as it would for the most part be a direct disk restore. If a customer has a file sharing server with 200-300GB or more of small/medium files and it gets infected by the crypto virus (just as one example of something we've seen recently), and say half the files are gone; If we were to try such a recovery right now in our environment, especially since we have a multi-tenant environment with only VIX network access between the Veeam server & the production VMs, it would take over a week to recover the files.

It would be very convenient & simplistic to be able to recover via File Restore selections (including permissions), instead of having to recover the entire disk or VM, and then copy the data over manually via disk attach/detach, etc. Some form of hotadd based File Level Restore option would provide a SuperFast File Level Restore functionality that no other competing vendor currently has bragging rights on... :wink:
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Veeam Feature Enhancement Request - File Level Restores

Post by chrisdearden »

So if we used a windiws based for helper in that situation, permissions would be preserved and problem solved?
Post Reply

Who is online

Users browsing this forum: No registered users and 79 guests