Using veeam and linked clones to build out lab environments

[mschlott]

At my employer, we keep 3 environments, Production, QA and Development. They are for the most part copies of each other, that are periodically updated by exporting and importing datbases, copying files and running scripts to update databases to allow developers greater access in the other environments. We have challanges in that we may have ongoing UAT in our QA environment, that may require no changes to QA data for months at a time. This prevents QA testing of other applications that may require more recent data copied from production. Building out a new environment for a single project would take many weeks of man hours for planning, building and modifying system to make sure that it does not interfier with the production environment.

So I was playing with vmware view and it reinforced to me the power of linked clones. Knowing that I already have the ability to have multiple versions of replicas of my production environment by using Veeam Backup and Replication, I was inspired to come up with a solution to having multiple environments without buying a bunch of new storage and have the ability to have multiple versions of data.

I first set up a veeam replication job for my production servers, that I will run manually only when a refresh of data is needed. I keep a maximum of 4 copies of the replica.

I came across this page, http://vmutils.blogspot.com/2011/06/scr ... seudo.html , which helped with the scripting of creating a set of linked clones. The scripts on this site worked nicely for creating an isolated network and creating linked clones of each VM as long as there is only one network. I had to make some modifications for my site, because we have a DMZ and INSIDE network, with a firewall in between. I am using a free version of Vyatta for my firewall.

So now I am able to build out a full isolated copy of production, consisting of 30 or more servers in a very short period of time. To connect to this isolated environment, I configured a single Windows server to have 2 interfaces. One on the isolated network, and one on my real network. Using this, developers can RDP to this machine and use all of their normal tools to access the environment.

I have come to a brick wall that prevents me from rolling this out as a replacement to our current Dev and QA environments. Outside integrations, and UAT. They both have simmilar problems. UAT is easier to explain, so I'll start there. We have dozens of internal users who must sign off on some of the changes we make. Typically we send them the URL to our QA site and say test here. QA has no links to production, and if it ever does, we fix that in QA.

Using an isolated linked clone of production we could not just send out a test URL because the network is isolated. We would either have to tell the UAT testers to RDP to a machine that has access to the QA servers, or set up a proxy server that has access to the enviroment and have the user change their browser proxy setting to use this environment. If we have them use RDP, we loose the benefit of UAT that includes various desktops and browsers. If we have the UAT testers change their proxy settings, we risk that a group of non-technical end users accidentally updates production data.

Another idea I had to get around UAT which would also help with third party integrations, is to set up an apache reverse proxy server that responds to QA URLs and sends requests back to the isolated production servers. The problem with this, is that it depends on there being no hard coded URLs in any of the code that could not be caught by a reverse proxy. This works fine for code we have controll over but we have some third party apps that this will not work with.

This is not a Veeam question, but rather an example of how you can further use veeam to build out a lab with a request for some ideas to overcome the problems I have yet to overcome.

Is anyone else using Veeam replicas in this way? Have you gone to the next level and made your lab available on public networks? Am I missing a really cool feature of surebackup and vlabs that would solve this?

All feedback welcome.

mike

[tsightler]

Typically, the biggest problem for stuff like this is the overlapping namespace/IP network (i.e. servers in the "lab" have the same names and IP addresses as their production counterparts). I'm assuming that's true for your setup?

For applications that are almost exlusively web based, reverse proxies are probably the best/easiest answer. You can leverage the URI re-write rules to rewrite all URL information, for example adding a "sub-domain" to all queries responses and redirects. This isn't as difficult as it seems since this is already a common requirement when you have to provide public Internet access to internal services, so the tools are already out there and generally well documented. As you mentioned, Apache is a good options as their mod_rewrite is especially powerful and can almost certainly handle any situation that comes along. Yes, it may take some time to get a "perfect" set of rules to handle every possible redirect, but it will get closer and closer with time, and the initial rules to cover 98% of cases are pretty simple. The biggest issue is if you have any product that redirects to a specific IP address, this can be somewhat of a challenge.

[padhraicmaguire]

Hi Mike,

I know this is a very old thread but I'll give it a go that you might still see this. Would you be able to share the scripts you used to create the instant clones in the isolated environments. The source scripts you refer to in your post are no longer available for download. I'm looking at the InstantClone modules and the VMfork fling but instant-clone requires the machine powered on for the operation to work. I am using Veeam Replication to replicate VM's from Site A to Site B. I would like to create multiple isolated environments for the Development teams to use for testing code deployments. The replica's are powered off in the target site which i think is the same starting point you have.

Thanks.
Padhraic