Using IVMR (Instant Virtual Machine Recovery) for testing

[kjstech]

I'm using IVMR on machines to test certain application upgrade paths and configurations without affecting production.
I have a SureBackup VM Proxy appliance that I manually turn on.
I run Instant VM Recovery but do not connect the network.
Once it is in place (I usually name the machine with -lab designation and put it in its own testing folder and resource pool), I manually edit the NIC and change it to my lab-ServerNetwork which does not have any uplinks and its on VLAN300. It has a connection to SureBackup VM Proxy so if I would need Internet access, I just set a proxy server in the VM.

Questions:
1. When I do any testing, a prereq is always a Domain Controller. Why is it when I power up my DC, I can never log into it right away. I get "No logon servers are available to satisfy your request". After some time the VM reboots itself and then I can log in without an issue.

2. Can these "Lab" VM's remain running overnight? Yesterday I tore them all down before I went home because I didn't want any potential issues with last nights backup. I wasn't sure if it could read or backup something that may be "in use".

3. Performance is abysmal. Disk latencies to the vPower NFS datastore that Veeam presents to the ESXi host this is all running on range from 62-5865 ms, sometimes even in the 5 digits (when VM's are initially powered on). I even have the changes redirected to another NFS filesystem on our production SAN. Everything is 10gb. Veeam VM has a 10gb interface that can talk to Exagrid backup repository on its 10gb interface, and the IVMR test machines are all running on the same host as Veeam, so that connection should all be "internal". Would it be faster to just restore VM's with different names, and edit their network adapter connection to our existing NFS production storage? I realize "up front" it may take longer to move the data, but once there we can save a ton of time by having good performance in the machine. I am always using the last night's backup so it should be in Exagrid's "Landing Zone".

I know about SureBackup, I initially was trying to use that, however there's no control of the session. It only powers on one VM at a time and even if that VM is powered on you have to wait an extraordinarily long timeout (30-60 minutes depending on the setting) for the next VM to even start. If you do not put such a high timeout for "power on" the wizard will think the VM did not power on and tear down your whole session. So I'd rather have manual verification and testing and using IVMR seems to be the solution if we can get the performance improved a bit.

[foggy]

1. This is fully expected and actually how automatic DC recovery works in Veeam B&R. You need to wait until the VM reboots on its own.
2. Backup job will fail if it needs to run into the file that is locked by Instant Recovery session. So it depends on the backup method used, forward incremental will be ok, while reverse incremental will fail.
3. So what backup method is it? In case of forward incremental, the full backup file the latest restore point is based on can already be deduplicated. How many VMs are you IR'ing concurrently?

I know about SureBackup, I initially was trying to use that, however there's no control of the session. It only powers on one VM at a time and even if that VM is powered on you have to wait an extraordinarily long timeout (30-60 minutes depending on the setting) for the next VM to even start.

You can use linked backup jobs instead of application groups and specify the maximum number of VMs that can be started at the same time.

[kjstech]

Ok I left an IVMR lab up yesterday and last night backups ran without an issue. For our Exchange server, I just restored it under a new name. It was way to slow. It took about 2 1/2 hours to restore the 1.4 TB machine to our NFS test SAN, but at least the performance is better. In the big picture it would have taken me even longer to do testing running it from IVMR because of the poor performance.

The backup type we use is Incremental (recommended), with Create synthetic full backups periodically (Saturday).
There are 3 VM's right now running in Instant Recovery. When I'm done with my tests I will go in and unpublish them and Veeam will clean them out.
I have 2 VM's I just did a restore on (Exchange to test the CU12 upgrade, and vcenter to test going from 5.0 to 6.0u2). They are all in their own Surebackup lab resource pool and folder, all attached to a lab-ServerNetwork which has no uplinks, but Veeam_SureBackup_Lab_Proxy has a nic in this network (and others) so if we need to quickly download something on one of these test machines we add the lab proxy server in IE's connection settings, then take it out when done.

IVMR is running from the most recent restore (well now one that is a day old), and we use Exagrid appliance which by default stores everything in its landing zone in an already deduped state. Exagrid is on a 10gbe network subnet, and Veeam also has a nic in this same subnet, as do all ESXi hosts and Production storage (VNX5200 NFS). We have test storage on 1gbps NFS (EMC NX4) that all ESXi hosts have access to, and thats where I restore machines to for testing, whereas IVMR machines store the changed data to one of our filesystems on our 10gbe NFS EMC5200. The 10gbe network has jumbo frames turned on, and the nics on both veeam proxies are verified to have 9000 mtu and ping with do not fragment and high mtu size verifies connectivity. It seems network wise the connection is there, just SureBackup and IVMR performance is terrible.

[foggy]

For Instant Recovery, faster disks are more important than the network, since it does completely random I/O. The amount of I/O the restored VM guest OS generates also directly affects IR performance and Exchange is pretty highly-transactional application.

[kjstech]

Ok but if its reading the latest backup, it should be in Exagrid's landing zone, so wouldn't that be pretty efficient? Not only that but I direct the changes to a filesystem on production EMC VNX5200 NFS share, which is typically average latency in the single digits and less than 5 ms.

[foggy]

Not only the latest increment, but the entire chain up to the latest full backup should be in the landing zone.