Comprehensive data protection for all workloads
SVP, Product Management
Posts: 24092
Liked: 3279 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Instant VM Recovery

Post by Gostev » Nov 11, 2013 4:03 pm

Q: What is Instant VM Recovery?
A: Instant VM Recovery allows you to instantly recover any VM into your production environment by running it directly from backup file. Best analogy is that this gives you "spare tyre" so that you can get to a service shop. You cannot go at full speed, but you are still going instead of being stuck in woods. To complete the restore, you can use native hypervisor capabilities to migrate the recovered VM to production storage without any impact on users (this is like changing spare tyre to a real one as you go). Alternatively, you can move VM to production storage during off-hours with short downtime using Veeam Quick Migration (this is like pulling over to a service shop to change tire). Note that Instant VM Recovery supports bulk operations (multiple VMs at once).

Q: As a percentage, what's the difference in performance when running a VM that has been replicated versus one that’s running from a backup file?
A: It depends on many factors (backup files location, storage speed of Veeam Backup server, number of concurrently running instantly recovered VMs). Generally, for all low I/O server (such as DC, DNS, DHCP, WWW, AV, PRINT) the performance difference will be hardly noticeable. Impact on high I/O servers will be much more noticeable, because vPower engine throughput is limited.

Q: Our hypervisor license does not include migration capabilities, or migration does not work. What are my options to complete the restore?
A: Simply perform failover during the next maintenance window. To do that, use Quick Migration functionality in Veeam. Unlike with hypevisor-based migration, this approaches require short downtime. However, this is still beneficial as this allows you to convert unplanned downtime (which is what cost businesses money) into planned downtime during your regular maintenance windows.

Q: Will Storage VMotion or Quick Migration carry over the actual, latest VM disks state (including delta from backup state)?
A: Yes, this happens automatically.

Q: What happens if Veeam Backup server fails when you have instantly recovered VMs running?
A: Just what happens when your production storage fails - nothing pretty. Keep in mind that no data is lost as long as the backup server comes up again later with no data corruption. Although the chance of consequent failures of 2 different storage devices is pretty unlikely to say the least. It is like getting a hole in your newly placed tire...

Q: What happens if there is no space left in the vPower NFS write cache location?
A: The VM will probably crash. Do not change anything (especially do not stop IVMR) and create a support call. Support might be able to help if you can free up space.

Q: With the instant VM recovery feature present, why would you replicate?
A: 2 big reasons: not to have dependencies on vPower engine to run the VM, plus full disk I/O performance in case of disaster (important for large-scale disasters).

Q: Will Instant VM Recovery work with RDM that have been backed up?
A: Yes. RDM in virtual mode is backed up as VMDK and are available directly in the backup file. RDM in physical mode is skipped during backup, however there is nothing preventing instantly recovered VMs from connecting and using it (if it is not impacted by the disaster, of course).

Manual Recovery Verification

Q: I have Veeam B&R Standard Edition. How can I do "manual" SureBackup recovery verification. What is the process?
A: To perform manual recovery verification, you should use Instant VM Recovery feature. For example, for simple VM boot up test, just go through the Instant VM Recovery wizard and power on VM, but do not select a checkbox to connect VM to a network.

Q: I performed boot up test as described above, and while VM booted fine, most applications could not start?
A: This is expected because there is no network connectivity, so application cannot establish connection to domain controller, DNS servers, and other services it is dependent on. If you want to test application recovery, create isolated network and edit instantly recovered VM network configuration before powering it on. Perform this for all required VMs which run dependent applications (such as DC and DNS), placing them all on the same isolated network, and start them in correct order (for example, DNS > DC > Application).

Q: This sounds very similar to what we are already doing during our monthly/quarterly/annual DR tests. What is the catch?
A: The catch is that with Veeam, you are able to run VMs directly from backup files, without spending many hours of extracting all those VMs from backup to production storage. Even finding free disk space alone (to extract all required backups) often becomes a challenge. So restore test that would previously take a whole weekend can now be completed in less than 30 mins.

Q: This manual process sounds too complex to perform for every backup, every VM, every time as your marketing materials state?
Q: That is right, which is why our Enterprise Edition provides fully automated recovery verification that performs all of the tasks described above automatically, including running required test scripts against each VM. It even creates and manages isolated test environment automatically for you. This allows you to perform DR test every single day, with review of email report showing recovery verification job results being the only manual activity in the whole process.

SVP, Product Management
Posts: 24092
Liked: 3279 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Virtual Lab

Post by Gostev » Nov 11, 2013 4:07 pm

Q: Do I need to create virtual lab for Instant VM Recovery?
A: No. Instant VM Recovery feature is available even in the Standard Edition, which does not provide ability to create virtual labs.

Q: What exactly this "virtual lab" is?
A: By virtual lab we mean automatically managed, fully isolated environment where VMs can be run directly from backup file to facilitate features such as universal application item recovery (U-AIR), SureBackup (recovery verification) and On-Demand Sandbox. Virtual lab uses isolated virtual networks that mirror production networks, and uses proxy appliance for routing between production and isolated networks, and between isolated networks. Each virtual lab places all temporarty VMs in the designated folder and resource pool. You can use resource pool to control resource usage of virtual lab VMs.

Q: How does the proxy appliance work?
A: The proxy appliance allows to route traffic between computers in the production network, and temporary VMs running from backup in the isolated network. Think about proxy appliance as your home router, which routes traffic between your home network and internet.

Q: Do you change temporary VMs IP addresses to prevent IP conflicts for VMs which are already running in production?
A: In fact, all temporary VMs in the isolated network have exact same IP addresses as in production network. IP address conflicts are simply not possible, as different VLANs are used for production and isolated networks.

Q: How is it possible to access temporary VMs in the isolated network from production network, if VMs in both networks have the same IP addresses?
A: Each temporary VM is assigned so called "masquerade address" from selected masquerade network (part of virtual lab settings). Routing table on Veeam Backup server is automatically updated, and proxy appliance IP address in the production network is assigned as gateway for masquerade network. Acting as gateway, the proxy appliance performs address translation and substitutes masquerade IP address with real IP address in the isolated network. Although this sounds pretty complex, all happens transparently for you as a user.

Q: What if i want all computers on the network to be able to access those temporary VMs running in the virtual lab?
A: You should assign the proxy appliance static IP address in the virtual lab settings, and update your production router settings to forward all request destined into masquerade network (as configured in virtual lab settings) to the proxy appliance IP address. Alternatively, if you only need to access select VMs in the isolated network, you can use virtual lab's Static Address Mapping feature and point specific IP addresses in the production network to selected IP addresses in the isolated network. Proxy appliance will grab specified production IP addresses for its production network interface, and will take care of routing automatically.

Q: Is it possible to enable internet access from within the virtual lab?
A: Yes, you will see the corresponding settings on the Proxy step of the Virtual Lab wizard.

Application Groups

Q: What is Application Group?
A: Application Group is our way to handling application dependencies for any VMs running in the isolated environment. Simplest example is Microsoft Exchange server - if you power it on in the isolated environment which does not have DNS server and Domain Controller present, mailbox store will not start.

Q: Can you give an example of what is typical Application Group looks like for small Windows shop.
A: Any application group should contain at least DNS server for name resolution, and directory server for authentication. In Windows world and smaller environments, both services are typically provided by Domain Controller, so application groups may look like these (put DNS before DC if DNS server is separate):
Exchange: DC > Exchange
FTP Server: DC > File Server > IIS
SharePoint: DC > SQL > SharePoint

Q: Can you give an example of what typical Application Group looks like for small Linux shop with no directory services used.
A: Any application group should contain at least DNS server for name resolution. Application groups may look like these:
CMS: DNS > MySQL > Apache w/CMS code
CRM: DNS > Oracle > CRM Server

Q: I have pretty static and small environment with just a few VMs. How should I configure application groups?
A: Simply put all of your VMs in a few application groups keeping in mind the required boot order. You can create one group per application, or you can have more than one application per group, or even all of them in a single group.

Q: I have a large and dynamic environment with many VMs created and deleted daily. Micro-managing application groups is hardly possible.
A: We have thought of that. In this case, you should setup your application groups to contain essential infrastructure services only (for example, DNS and DC is something almost every application in Windows shop depends on). Now, SureBackup jobs provide you with capability to "link" this application group and one or more of your backup jobs together. With such setup, SureBackup first starts the application group VMs and leaves them running for the duration of job, and then proceeds to powering on VMs from linked backup jobs one by one for verification. As a result, as you are adding or removing VMs from environment, they will be automatically added to backup jobs (granted you have backup job setup on container basis), which in turn will make them processed as a part of SureBackup job without requiring you to edit its settings.

On-Demand Sandbox

Q: How do I setup a sandbox?
A: Create new application group and stuff it with VMs you want to be available in sandbox. Create new SureBackup job, select the newly created application group, and select "Keep VMs running" checkbox. Now, simply run the SureBackup job. As soon all VM in the job start, your sandbox is ready! You can open VM console for all VMs running in the sandbox normally, and do whatever you need to do! Or, simply connect to applications running in sandbox with native management tools you are using to manage the same application in your production environment.

Q: When I start SureBackup job, it always runs using latest backups - but I need to go 1 week ago?
A: To start SureBackup to restore point other than latest backup, right-click the job, select "Start To" in the short-cut menu, and select the desired date and time to start the job to.

Q: I can ping and access sandbox VMs running in the isolated environment by masquearade IP address from backup server, but not from any other computer?
A: This is because routing table was updated automatically on Veeam Backup server as a part of SureBackup job. If you execute route print you will see one of routes pointing to proxy appliance IP address in production network. So, to make other computers able to access sandbox VMs, you should either update their local routing table (note that Universal AIR wizard does this automatically), or configure router in your environment accordingly to make this work for all computers at once (however, it might be easier to use Static Mapping feature of the virtual lab instead).

Application-Owner or Developer-Directed Recovery

Q: How do I enable developers or engineers to restore data from an application that cannot be restored with file-level restore or the Veeam Explorers?
A: Essentially, you need to create ever-running SureBackup job with application in question, and provide convenient DNS name to it.
1. Create new application group with VMs running required application and its dependencies.
2. Create new SureBackup job with "Keep application group running" option selected, and schedule it to run after each backup.
3. Open settings for Virtual Lab used by SureBackup job, and edit static IP mapping feature to map isolated IP address of required VM to some unused IP address in the production network.
4. Update production DNS and assign simple DNS name to the IP address chosen earlier, for example "exchange-yesterday".
With that in place, any user in the production environment will be able to access the server running from latest backup in the isolated environment by this DNS name. Any VMs published in such a way will be running until the next backup job is started. The next run of backup job will stop any linked SureBackup jobs, and perform backup. Then, SureBackup job will start using the newly created latest backup file according to its schedule, and again runs for the whole day until the next backup. Effectively, you will have a copy of application running from last night's backup always available to users behind easy to remember DNS name - enabling them to logon to familiar user interface and recover any data they need. This system will require no maintenance whatsoever.

Q: What are some use cases for this?
A: Just a few ideas, I am sure you will find many more uses - please let me know!
1. After unintended changes on production PostgreSQL you want to check out the differences between the latest backup and production.
2. Development shops. Publish yesterday's copy of MySQL database, and let your developers test new code, lookup previous state of any values, compare database schema, and so on. Every day, the server behind selected DNS name will run the latest copy of MySQL from last night's backup, making it extremely simple and convenient to access, with the system requiring zero maintenance. Any changes made to the database will be discarded once the SureBackup job running the VM is stopped.

SVP, Product Management
Posts: 24092
Liked: 3279 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Data Labs Self Service And U-AIR Wizard (Universal Application Item recovery)

Post by Gostev » Nov 11, 2013 4:13 pm

Q: How does it work from developer or application owner perspective?
A: User perspective:
1. Install AIR wizard on user workstation (located on VBR ISO).
2. Right-click the Virtual Lab Manager tray icon to create new virtual lab request.
3. Specify description, estimated time needed, required VM and restore point, and submit request.
4. Wait until the request is approved by backup administrator, and the lab is started.
5. Continue in the AIR wizard and do whatever you like (test update, do a restore that is not possible with Veeam Explorers or file restore etc.).
6. If you need more time, just extend virtual lab lease.
7. When done, dismiss the lab (if you forget, it will expire automatically).

Q: How it works from backup administrator perspective?
A: Administrator perspective:
1. Receive email notification about new lab request.
2. Open Enterprise Manager, make decision to approve or deny the lab request.
3. Go through request approval wizard. If necessary, adjust request settings (such as lab lease time).
4. Manage active labs if needed. For example, stop lab used by developers to let somebody else perform emergency restore using the same virtual lab.
5. No need to babysit lab, as they will automatically expire after requested time passes.

Q: How it works under the hood?
A: All operations described below are fully automated and "hidden" from users:
1. AIR wizard generates lab (Virtual Lab must exist already) request and passes it over to Virtual Lab Manager (VLM), which and sends it over to Enterprise Manager.
2. Request is approved by admin, who selects SureBackup job to use as a part of approval process.
3. Enterprise Manager will automatically locate Veeam Backup server for selected SureBackup and will have it run the SureBackup job for required VM only.
4. Once all dependent, and the selected VM are running and ready (ready means recovery verification was successful), Veeam Backup server notifies Enterprise Manager.
5. Enterprise Manager notifies requesting VLM install and provides network parameters for lab (proxy appliance IP address and masquerade IP address of requested VM).
6. VLM updates routing table on local machine and notifies user the everything is ready with popup notification.
7. User is now able to proceeds through AIR wizard to perform application item recovery.
8. Lab automatically expires automatically as requested time passes (unless user extends the lease, or per user request to dismiss the lab).

Q: How this "universal recovery" is supposed to work?
A: Once the lab is prepared, Universal AIR wizard will provide masquerade IP address of requested VM in the isolated environment, and update routing on current computer automatically to enable transparent access. You can then use native management tools to extract required items from the application and put them back to production server. For example, you can use free Oracle SQL Developer to perform item-level recovery from Oracle database, or Microsoft SQL Management Studio to perform item-level recovery from Microsoft SQL database, or MySQL Workbench to perform item-level recovery from MySQL database, etc. - any application!

Q: I had never dealt with databases before. Is there a demo that can help me better understand the concept, so I can explain this to my colleagues?
A: This video shows (beginning from 28:10min) an example where the presenter connects the SQL Management Studio to the production SQL database and in parallel to the Data Lab / Universal Restore SQL Database. (The video is quite old, but the concept is still the same today)

Q: Do I have to install the AIR wizard on Veeam Backup server?
A: No, the AIR wizard is usually only needed on the workstations of the people who do the requests. For example, you can install them on developers' workstations.

SVP, Product Management
Posts: 24092
Liked: 3279 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland


Post by Gostev » Nov 11, 2013 4:14 pm

Bottleneck Analysis

Q: Job statistics tells me I have a bottleneck, and I cannot seem to get rid of this no matter what I do. What am I doing wrong?
A: You are doing nothing wrong. Even most powerful backup infrastructure will have a bottleneck - just like any bottle, no matter of size, has the bottleneck. We merely show you the "weakest" (thinnest) link of the chain - what to consider upgrading next to be able to improve processing performance. However, if you are happy with you jobs performance and backup window, you should not do anything about it (just consider this info as an FYI).

Q: What do the 4 load numbers I get in per-VM statistics mean?
A: These numbers show percent of time the given data processing stage was busy versus waiting for other stages to provide or accept the data. Do not expect the numbers across all processing stages to add up to 100%, as busy time of each processing stage is measured separately and independently.

Q: What are the data processing stages?
A: No matter what job you are running, and how you have the product deployed, there are 4 main data processing stages that data passes in the specific order (think data processing conveyor). These are Source > Proxy > Network > Target, and each processing stage has a load monitoring counter associated with it.

"Source" is the source (production) storage disk reader component. The percent busy number for this component indicates percent of time that the source disk reader spent reading the data from the storage. For example, 99% busy means that the disk reader spent all of the time reading the data, because the following stages are always ready to accept more data for processing. This means that source data retrieval speed is the bottleneck for the whole data processing conveyor. As opposed to that, 1% busy means that source disk reader only spent 1% of time actually reading the data (because required data blocks were retrieved very fast), and did nothing the rest of the time, just waiting for the following stages to be able to accept more data for processing (which means that the bottleneck is elsewhere in the data processing conveyor).

"Proxy" is the backup proxy server (source backup proxy in case of replication). Proxy performs on-the-fly deduplication and compression of data received from the source component, which can be quite resource intensive operation on hundreds MB/s data streams. The percent busy number for proxy component shows the proxy CPU load. For example, if proxy shows 99% busy, it means that the proxy CPU is overloaded, and is likely presenting a bottleneck on the whole data processing conveyor.

"Network" is the network queue writer component. This component gets processed data from the proxy component, and sends it over a network to the target component (e.g. the repository). The percent busy number for the network component shows percent of time that network writer component was busy writing the data into the network stack queue. For example, 99% busy means that the network writer component spends most of the time pushing pending data into the network, because there is always some previous data still waiting to be sent over to the target. This means that your network throughtput is insufficient, and is presenting a bottleneck on the whole data processing conveyor.

"Target" is the target (backup/replica storage) disk writer component. Percent busy number for the target component shows percent of time that the target disk writer component spent writing the data to the storage. For example, if target shows 99% busy, it means that the target disk writer component spent most of its time performing I/O to backup files. This means your target storage speed is presenting a bottleneck for the whole data processing conveyor. All the pending I/O operations cannot complete fast enough (the storage fabric could also be the limit), and due to that there is always some data waiting in the incoming queue of the network component that is waiting to be written to disk.

Q: Can I see load numbers in the real-time?
A: If you hover over the bottleneck value in the real-time statistics window, you will get a tooltip with the current values. However, because this data is real-time, it may be affected by intermittent issues, or temporary conditions (such as file system cache population in the beginning of the job). Averaged load data logged in the session log for each VM is more accurate and reliable.

Veeam Software
Posts: 3028
Liked: 369 times
Joined: Sep 01, 2014 11:46 am
Location: Austria


Post by HannesK » Jan 11, 2019 10:10 am


Q: Which tape autoloaders / libraries does Veeam support?
A: Veeam supports LTO compatible devices (physical and virtual). You can find a list of libraries and autoloaders which were reported to work in the tape forum

Q: Is path failover supported?
A: Veeam Backup & Replication supports path failover for tape devices with multiple drives that manage multiple paths over multiple SANs.

Q: Which device driver should I use?
A: Veeam Backup & Replication supports custom vendors drivers as well as generic windows drivers. However, it is recommended to install vendors drivers in non-exclusive mode.

Q: Is WORM media supported
A: Yes, since version 9.5 update 4.

Q: Can Veeam share one library with 3rd party tape solutions?
A: If the library supports partitioning, yes. By partitioning the library Veeam can only see its part of the library and same for 3rd party tape solutions.

Q: Can I install another backup product that also writes to the same tape (library) on the Veeam server?
A: No, this is not possible.


Who is online

Users browsing this forum: Bing [Bot] and 26 guests