Availability for the Always-On Enterprise
Veeam Software
Posts: 23131
Liked: 2922 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Instant VM Recovery

Post by Gostev » Nov 11, 2013 4:03 pm

Q: What is Instant VM Recovery?
A: Instant VM Recovery allows you to instantly recover any VM into your production environment by running it directly from backup file. Best analogy is that this gives you "spare tyre" so that you can get to a service shop. You cannot go at full speed, but you are still going instead of being stuck in woods. To complete the restore, you can use native hypervisor capabilities to migrate the recovered VM to production storage without any impact on users (this is like changing spare tyre to a real one as you go). Alternatively, you can move VM to production storage during off-hours with short downtime using Veeam Quick Migration (this is like pulling over to a service shop to change tire). Note that Instant VM Recovery supports bulk operations (multiple VMs at once).

Q: As a percentage, what's the difference in performance when running a VM that has been replicated versus one that’s running from a backup file?
A: It depends on many factors (backup files location, storage speed of Veeam Backup server, number of concurrently running instantly recovered VMs). Generally, for all low I/O server (such as DC, DNS, DHCP, WWW, AV, PRINT) the performance difference will be hardly noticeable. Impact on high I/O servers will be much more noticeable, because vPower engine throughput is limited.

Q: Our hypervisor license does not include migration capabilities, or migration does not work. What are my options to complete the restore?
A: Simply perform failover during the next maintenance window. To do that, use Quick Migration functionality in Veeam. Unlike with hypevisor-based migration, this approaches require short downtime. However, this is still beneficial as this allows you to convert unplanned downtime (which is what cost businesses money) into planned downtime during your regular maintenance windows.

Q: Will Storage VMotion or Quick Migration carry over the actual, latest VM disks state (including delta from backup state)?
A: Yes, this happens automatically.

Q: What happens if Veeam Backup server fails when you have instantly recovered VMs running?
A: Just what happens when your production storage fails - nothing pretty. Although the chance of consequent failures of 2 different storage devices is pretty unlikely to say the least. It is like getting a hole in your newly placed tire...

Q: With the instant VM recovery feature present, why would you replicate?
A: 2 big reasons: not to have dependencies on vPower engine to run the VM, plus full disk I/O performance in case of disaster (important for large-scale disasters).

Q: Will Instant VM Recovery work with RDM that have been backed up?
A: Yes. RDM in virtual mode is backed up as VMDK and are available directly in the backup file. RDM in physical mode is skipped during backup, however there is nothing preventing instantly recovered VMs from connecting and using it (if it is not impacted by the disaster, of course).

Manual Recovery Verification

Q: I have Veeam B&R Standard Edition that is advertized to have "manual" SureBackup recovery verification. What is the process?
A: To perform manual recovery verification, you should use Instant VM Recovery feature. For example, for simple VM boot up test, just go through the Instant VM Recovery wizard and power on VM, but do not select a checkbox to connect VM to a network.

Q: I performed boot up test as described above, and while VM booted fine, most applications could not start?
A: This is expected because there is no network connectivity, so application cannot establish connection to domain controller, DNS servers, and other services it is dependent on. If you want to test application recovery, create isolated network and edit instantly recovered VM network configuration before powering it on. Perform this for all required VMs which run dependent applications (such as DC and DNS), placing them all on the same isolated network, and start them in correct order (for example, DNS > DC > Application).

Q: This sounds very similar to what we are already doing during our monthly/quarterly/annual DR tests. What is the catch?
A: The catch is that with Veeam, you are able to run VMs directly from backup files, without spending many hours of extracting all those VMs from backup to production storage. Even finding free disk space alone (to extract all required backups) often becomes a challenge. So DR test that would previously take a whole weekend can now be completed in less than 30 mins.

Q: This manual process sounds too complex to perform for every backup, every VM, every time as your marketing materials state?
Q: That is right, which is why our Enterprise Edition provides fully automated recovery verification that performs all of the tasks described above automatically, including running required test scripts against each VM. It even creates and manages isolated test environment automatically for you. This allows you to perform DR test every single day, with review of email report showing recovery verification job results being the only manual activity in the whole process.

Veeam Software
Posts: 23131
Liked: 2922 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Virtual Lab

Post by Gostev » Nov 11, 2013 4:07 pm

Q: Do I need to create virtual lab for Instant VM Recovery?
A: No. Instant VM Recovery feature is available even in the Standard Edition, which does not provide ability to create virtual labs.

Q: What exactly this "virtual lab" is?
A: By virtual lab we mean automatically managed, fully isolated environment where VMs can be run directly from backup file to facilitate features such as universal application item recovery (U-AIR), SureBackup (recovery verification) and On-Demand Sandbox. Virtual lab uses isolated virtual networks that mirror production networks, and uses proxy appliance for routing between production and isolated networks, and between isolated networks. Each virtual lab places all temporarty VMs in the designated folder and resource pool. You can use resource pool to control resource usage of virtual lab VMs.

Q: How does the proxy appliance work?
A: The proxy appliance allows to route traffic between computers in the production network, and temporary VMs running from backup in the isolated network. Think about proxy appliance as your home router, which routes traffic between your home network and internet.

Q: Do you change temporary VMs IP addresses to prevent IP conflicts for VMs which are already running in production?
A: In fact, all temporary VMs in the isolated network have exact same IP addresses as in production network. IP address conflicts are simply not possible, as different VLANs are used for production and isolated networks.

Q: How is it possible to access temporary VMs in the isolated network from production network, if VMs in both networks have the same IP addresses?
A: Each temporary VM is assigned so called "masquerade address" from selected masquerade network (part of virtual lab settings). Routing table on Veeam Backup server is automatically updated, and proxy appliance IP address in the production network is assigned as gateway for masquerade network. Acting as gateway, the proxy appliance performs address translation and substitutes masquerade IP address with real IP address in the isolated network. Although this sounds pretty complex, all happens transparently for you as a user.

Q: What if i want all computers on the network to be able to access those temporary VMs running in the virtual lab?
A: You should assign the proxy appliance static IP address in the virtual lab settings, and update your production router settings to forward all request destined into masquerade network (as configured in virtual lab settings) to the proxy appliance IP address. Alternatively, if you only need to access select VMs in the isolated network, you can use virtual lab's Static Address Mapping feature and point specific IP addresses in the production network to selected IP addresses in the isolated network. Proxy appliance will grab specified production IP addresses for its production network interface, and will take care of routing automatically.

Q: Is it possible to enable internet access from within the virtual lab?
A: Yes, you will see the corresponding settings on the Proxy step of the Virtual Lab wizard.

Application Groups

Q: What is Application Group?
A: Application Group is our way to handling application dependencies for any VMs running in the isolated environment. Simplest example is Microsoft Exchange server - if you power it on in the isolated environment which does not have DNS server and Domain Controller present, mailbox store will not start.

Q: Can you give an example of what is typical Application Group looks like for small Windows shop.
A: Any application group should contain at least DNS server for name resolution, and directory server for authentication. In Windows world and smaller environments, both services are typically provided by Domain Controller, so application groups may look like these (put DNS before DC if DNS server is separate):
Exchange: DC > Exchange
FTP Server: DC > File Server > IIS
SharePoint: DC > SQL > SharePoint

Q: Can you give an example of what typical Application Group looks like for small Linux shop with no directory services used.
A: Any application group should contain at least DNS server for name resolution. Application groups may look like these:
CMS: DNS > MySQL > Apache w/CMS code
CRM: DNS > Oracle > CRM Server

Q: I have pretty static and small environment with just a few VMs. How should I configure application groups?
A: Simply put all of your VMs in a few application groups keeping in mind the required boot order. You can create one group per application, or you can have more than one application per group, or even all of them in a single group.

Q: I have a large and dynamic environment with many VMs created and deleted daily. Micro-managing application groups is hardly possible.
A: We have thought of that. In this case, you should setup your application groups to contain essential infrastructure services only (for example, DNS and DC is something almost every application in Windows shop depends on). Now, SureBackup jobs provide you with capability to "link" this application group and one or more of your backup jobs together. With such setup, SureBackup first starts the application group VMs and leaves them running for the duration of job, and then proceeds to powering on VMs from linked backup jobs one by one for verification. As a result, as you are adding or removing VMs from environment, they will be automatically added to backup jobs (granted you have backup job setup on container basis), which in turn will make them processed as a part of SureBackup job without requiring you to edit its settings.

On-Demand Sandbox

Q: How do I setup a sandbox?
A: Create new application group and stuff it with VMs you want to be available in sandbox. Create new SureBackup job, select the newly created application group, and select "Keep VMs running" checkbox. Now, simply run the SureBackup job. As soon all VM in the job start, your sandbox is ready! You can open VM console for all VMs running in the sandbox normally, and do whatever you need to do! Or, simply connect to applications running in sandbox with native management tools you are using to manage the same application in your production environment.

Q: When I start SureBackup job, it always runs using latest backups - but I need to go 1 week ago?
A: To start SureBackup to restore point other than latest backup, right-click the job, select "Start To" in the short-cut menu, and select the desired date and time to start the job to.

Q: I can ping and access sandbox VMs running in the isolated environment by masquearade IP address from backup server, but not from any other computer?
A: This is because routing table was updated automatically on Veeam Backup server as a part of SureBackup job. If you execute route print you will see one of routes pointing to proxy appliance IP address in production network. So, to make other computers able to access sandbox VMs, you should either update their local routing table (note that Universal AIR wizard does this automatically), or configure router in your environment accordingly to make this work for all computers at once (however, it might be easier to use Static Mapping feature of the virtual lab instead).

User-Directed Recovery

Q: How do I enable users to restore their own items from latest backup?
A: Essentially, you need to create ever-running SureBackup job with application in question, and provide convenient DNS name to it.
1. Create new application group with VMs running required application and its dependencies.
2. Create new SureBackup job with "Keep application group running" option selected, and schedule it to run after each backup.
3. Open settings for Virtual Lab used by SureBackup job, and edit static IP mapping feature to map isolated IP address of required VM to some unused IP address in the production network.
4. Update production DNS and assign simple DNS name to the IP address chosen earlier, for example "exchange-yesterday".
With that in place, any user in the production environment will be able to access the server running from latest backup in the isolated environment by this DNS name. Any VMs published in such a way will be running until the next backup job is started. The next run of backup job will stop any linked SureBackup jobs, and perform backup. Then, SureBackup job will start using the newly created latest backup file according to its schedule, and again runs for the whole day until the next backup. Effectively, you will have a copy of application running from last night's backup always available to users behind easy to remember DNS name - enabling them to logon to familiar user interface and recover any data they need. This system will require no maintenance whatsoever.

Q: What are some use cases for this?
A: Just a few ideas, I am sure you will find many more uses - please let me know!
1. User-driven application item recovery from selected applications featuring web UI. Publish Exchange from last night backup to let users logon to OWA and extract required mail items or other data from their mailboxes. Publish SharePoint server to let users recovered deleted documents. Publish CRM server to let users to recover accidentally deleted or corrupted customer records. Remember, since it is a copy of production system, all permissions are still in place – the user will only be able to retrieve items he or she had access to.
2. Development shops. Publish yesterday's copy of Oracle database, and let your developers test new code, lookup previous state of any values, compare database schema, and so on. Every day, the server behind selected DNS name will run the latest copy of Oracle from last night's backup, making it extremely simple and convenient to access, with the system requiring zero maintenance. Any changes made to the database will be discarded once the SureBackup job running the VM is stopped.

Veeam Software
Posts: 23131
Liked: 2922 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland

U-AIR (Universal Application Item Recovery)

Post by Gostev » Nov 11, 2013 4:13 pm

Q: Can you explain the problem this functionality is addressing?
A: Please refer to the following blog post for more details: Separation of Permissions in Backup and Recovery

Q: I am the only admin in a small environment. Do I have to use the requests workflow in order to be able to restore application items?
A: No, obviously we do not want you to spend time creating and approving lab requests for yourself. As backup administrator, you can simply right-click any VM in the running SureBackup job session, and start application-item recovery procedure immediately. Just make sure AIR wizards are installed on your Veeam Backup server.

Q: How it works from end user perspective?
A: User perspective:
1. Install AIR wizard depending on the application you need to recover from.
2. Right-click the Virtual Lab Manager tray icon to create new virtual lab request.
3. Specify description, estimated time needed, required VM and restore point, and submit request.
4. Wait until the request is approved by backup administrator, and the lab is started.
5. Continue on in the AIR wizard and perform the restore.
6. If you need more time, just extend virtual lab lease.
7. When done, dismiss the lab (if you forget, it will expire automatically).

Q: How it works from backup administrator pespective?
A: Administrator perspective:
1. Receive email notification about new lab request.
2. Open Enterprise Manager, make desicion to approve or deny access to requested data to the user.
3. Go through request approval wizard. If necessary, adjust request settings (such as lab lease time).
4. Manage active labs if needed. For example, stop lab used by developers to let somebody else perform emergency restore using the same virtual lab.
5. No need to babysit lab, as they will automatically expire after requested time passes.

Q: How it works under the hood?
A: All operations described below are fully automated and "hidden" from users:
1. AIR wizard generates lab request and passes it over to Virtual Lab Manager (VLM), which and sends it over to Enterprise Manager.
2. Request is approved by admin, who selects SureBackup job to use as a part of approval process.
3. Enterprise Manager will automatically locate Veeam Backup server for selected SureBackup, and will have it run the SureBackup job for required VM only.
4. Once all dependent, and the selected VM are running and ready (ready means recovery verification wass successful), Veeam Backup server notifies Enterprise Manager.
5. Enterprise Manager notifies requesting VLM install, and provides network parameters for lab (proxy appliance IP address and masquearade IP address of requested VM).
6. VLM updates routing table on local machine, and notifies user the everything is ready with popup notification.
7. User is now able to proceeds through AIR wizard to perform application item recovery.
8. Lab automatically expires automatically as requested time passes (unless user extends the lease, or per user request to dismiss the lab).

Q: Do I have to install AIR wizards on Veeam Backup server?
A: No, AIR wizards can be installed on any computer. For example, you can install them on developers' workstations.

Q: Am I required to use Veeam AIR wizards to recover individual items? Unfortunately wizard X lacks feature Y which I really need.
A: No, if you prefer you can simply use native tools to get individual application items from applications running off backup files in isolated virtual lab environment. For example, Microsoft Exchange or SharePoint through OWA (see User-Directed Recovery below), and Microsoft SQL through Management Studio, Oracle through Oracle SQL Developer (video in Oracle session), MySQL through MySQL Workbench (video in Session 3), etc.

Universal AIR Wizard

Q: How this "universal" recovery is supposed to work?
A: Once the lab is prepared, Universal AIR wizard will provide masquerade IP address of requested VM in the isolated environment, and update routing on current computer automatically to enable transparent access. You can then use native management tools to extract required items from the application, and put them back to production server. For example, you can use free Oracle SQL Developer to perform item-level recovery from Oracle database, or Microsoft SQL Management Studio to perform item-level recovery from Microsoft SQL database, or MySQL Workbench to perform item-level recovery from MySQL database, etc. - any application!

Q: I had never dealt with databases before. Is there a demo that can help me better understand the concept, so I can explain this to my colleagues?
A: Session #3 video shows example of item-level restore from MySQL server backing my very own WordPress blog using free MySQL Workbench tool starting at 31:00.

Microsoft Active Directory AIR Wizard

Q: What is the list of capabilities of Active Directory AIR wizard?
A: Restore both entire objects, or individual attribute values of existing objects back to production Active Directory.

Microsoft Exchange AIR Wizard

Q: Why do I need this if you have Veeam Explorer for Microsoft Exchange?
A: Unlike Veeam Explorer for Microsoft Exchange, this wizard supports Exchange versions prior to 2010. We are intended to discontinue this wizard in the next major release, as we expect most people to migrate to at least Microsoft Exchange 2010 due Microsoft Exchange 2007 reaching end of support from Microsoft.

Q: What is the list of capabilities of Exchange AIR wizard?
A: Restore individual mailbox items (email, appointments, contacts) back into production mailbox, to Outlook PST file, or to individual files.

Q: Can I restore from other user's mailbox?
Q: Just like with other Exchange item recovery products, you need to make sure your account has sufficient permission to access mailbox it is injecting the restored data to. By default, domain adminstrators do NOT have permissions to other users' mailboxes. Full Mailbox Access permission on all mailboxes you want to be able to restore data for must be granted to account you are using to perform Exchange AIR.

Microsoft SQL AIR Wizard

Q: What is the list of capabilities of SQL Server AIR wizard?
A: Recover SQL databases, tables, records, schema etc. directly to the original SQL server, or to a SQL script.

Veeam Software
Posts: 23131
Liked: 2922 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland


Post by Gostev » Nov 11, 2013 4:14 pm

Bottleneck Analysis

Q: Job statistics tells me I have a bottleneck, and I cannot seem to get rid of this no matter what I do. What am I doing wrong?
A: You are doing nothing wrong. Even most powerful backup infrastructure will have a bottleneck - just like any bottle, no matter of size, has the bottleneck. We merely show you the "weakest" (thinnest) link of the chain - what to consider upgrading next to be able to improve processing performance. However, if you are happy with you jobs performance and backup window, you should not do anything about it (just consider this info as an FYI).

Q: What do the 4 load numbers I get in per-VM statistics mean?
A: These numbers show percent of time the given data processing stage was busy versus waiting for other stages to provide or accept the data. Do not expect the numbers across all processing stages to add up to 100%, as busy time of each processing stage is measured separately and independently.

Q: What are the data processing stages?
A: No matter what job you are running, and how you have the product deployed, there are 4 main data processing stages that data passes in the specific order (think data processing conveyor). These are Source > Proxy > Network > Target, and each processing stage has a load monitoring counter associated with it.

"Source" is the source (production) storage disk reader component. The percent busy number for this component indicates percent of time that the source disk reader spent reading the data from the storage. For example, 99% busy means that the disk reader spent all of the time reading the data, because the following stages are always ready to accept more data for processing. This means that source data retrieval speed is the bottleneck for the whole data processing conveyor. As opposed to that, 1% busy means that source disk reader only spent 1% of time actually reading the data (because required data blocks were retrieved very fast), and did nothing the rest of the time, just waiting for the following stages to be able to accept more data for processing (which means that the bottleneck is elsewhere in the data processing conveyor).

"Proxy" is the backup proxy server (source backup proxy in case of replication). Proxy performs on-the-fly deduplication and compression of data received from the source component, which can be quite resource intensive operation on hundreds MB/s data streams. The percent busy number for proxy component shows the proxy CPU load. For example, if proxy shows 99% busy, it means that the proxy CPU is overloaded, and is likely presenting a bottleneck on the whole data processing conveyor.

"Network" is the network queue writer component (with network being Ethernet, or a storage network). This component gets processed data from the proxy component, and sends it over a network to the target component. The percent busy number for the network component shows percent of time that network writer component was busy writing the data into the network stack queue. For example, 99% busy means that the network writer component spends most of the time pushing pending data into the network, because there is always some previous data still waiting to be sent over to the target. This means that your network throughtput is insufficient, and is presenting a bottleneck on the whole data processing conveyor.

"Target" is the target (backup/replica storage) disk writer component. Percent busy number for the target component shows percent of time that the target disk writer component spent writing the data to the storage. For example, if target shows 99% busy, it means that the target disk writer component spent most of its time performing I/O to backup files. This means your target storage speed is presenting a bottleneck for the whole data processing conveyor, because all the pending I/O operations cannot complete fast enough, and due to that there is always some data waiting in the incoming queue of the network component that is waiting to be written to disk.

Q: Can I see load numbers in the real-time?
A: If you hover over the bottleneck value in the real-time statistics window, you will get a tooltip with the current values. However, because this data is real-time, it may be affected by intermittent issues, or temporary conditions (such as file system cache population in the beginning of the job). Averaged load data logged in the session log for each VM is more accurate and reliable.


Who is online

Users browsing this forum: Google [Bot] and 66 guests