Major issue with VSS and lab environment

[FrancWest]

Hi,

I've just discovered a major issue and I don't know if this is intended behavior or a bug. Here's the case: we have a lab environment that is an exact clone of our production environment, IP addresses are also the same. It's of course located in a different isolated VLAN so that it doesn't cause any issues with our production lan. The Veeam backup server is located in our production LAN. Now the problem: when enabling application aware backups for the backup job of our lab environment it appears that Veeam is instructing our production VM's to process using VSS. This can for exmaple be seen in the eventlogs of our production Exchange servers. I suspect that Veeam first tries to send the VSS commands over the network using TCP/IP to the VM's. It gets a response using TCP/IP so it continues. However, the response it gets is from a production VM and not the one in the lab that's meant to be backupped and instructs the production VM to VSS and not the lab machine. The backup itself is made correctly, it backs-up the lab VM's.

Since it can connect using TCP/IP it doesn't use VIX. This is of course a major issue.

I've already contacted support (case #02314541). They suggested to reverse the order of using RPC and VIX, so that it first uses VIX and then RPC. However, this isn't a complete solution. Since it uses VIX for all VM's and even then, if it fails to use VIX for some reason it again falls back to RPC to connect which re-introduces the issue. Until now there's no way to disable RPC. Support suggested using a firewall on the production VM's to prevent the backup proxy to connect using RPC, but this isn't a solution either. We use hardware firewalls instead of client based firewalls. Also, preventing RPC might cause other issues since Windows is using RPC by itself heavily.

Anyone here have suggestions how to solve this? It would be nice to have a setting so that you can disable the use of RPC on a per job basis to prevent this kind of very unwanted behavior.

Franc.

[foggy]

Hi Franc, have you tried to specify a guest interaction proxy residing in the DR location for the job pointing to the lab VMs?

[FrancWest]

Hi,

I think that's not going to work. The page you refer to mentions that if the backup server fails to connect to the proxy it will be the proxy itself and then the issue is still there again since the backup server is in the same subnet as the production VM's.

Also, it looks like the machine acting as the guest interaction must be accessible by the backup server using TCP/IP. It has to be added as a managed server and that wizard asks for an IP address or DNS name. Neither will work because the lab isn't accessible from the production network and thus from the backup server over TCP/IP.

Franc.

[foggy]

Basically, you need to open just a couple of ports between the backup server and the guest interaction proxy, isn't that an option?

[FrancWest]

I'll talk to our network security guy, but I think he's not going to allow this since it must be completely separated from production.

Isn't it an option to let us specify on a per job basis that Veeam has to use VIX explicitly so that RPC isn't used at all?

[foggy]

On the other side, what is the reason behind backing up the lab VMs instead of production ones (how are they kept in sync, btw)? Or - why don't you put the backup server responsible for that right in the lab?

[FrancWest]

The lab is used for the implementation of new software. For exmaple now we have a large project going for 3 months to replace our accounting software. This lab is a copy of our production environment. So keeping them in sync isn't necessary. Every once in a while we refresh our lab by making new clones of the relevant production VM's. Placing a Veeam backup server in the lab isn't possible, since the lab doesn't have a vCenter server and also doesn't have access to our external storage to store the backups.

So it's not instead, but in addition to. Of course we backup production too.

[FrancWest]

Other issue: when backing up the SQL server in our lab (which is a clone of the production VM) Veeam reports during the backup job 'Waiting for completion of other tasks (VM is SQL server with configuration database - not a parallel task)'. This is not correct, actually, it doesn't contain the Veeam database at all because this clone was made before we installed and configured Veeam in our environment).
Same issue as earlier in this thread: because the servername and IP address is the same, Veeam thinks it's the production SQL server (apparently without checking if the database actually exists on this VM).

[foggy]

Placing a Veeam backup server in the lab isn't possible, since the lab doesn't have a vCenter server and also doesn't have access to our external storage to store the backups.

You can back it up via direct host connection (in case the host is available on the lab network). I'm out of other options with this kind of setup (though I understand the reasoning behind).

[FrancWest]

unfortunately the host isn't accessible from the lab network. I now have opened the ports on the lab-firewall and forwarded them to the interaction proxy inside the lab. Unfortunately the veeam server is still using itself as the proxy although I've explicitly told it only to use that guest interaction proxy for the lab backup job. Support is looking into it.

[foggy]

Thanks for the update. Let's see what our engineers can say after checking your configuration.

[FrancWest]

According to support the guest interaction proxy also needs to be able to setup a connection to the backup server. However, since they both have ip addresses in the same subnet, but in a different vlan, that’s not possible. So I guess we are out of options and there is no way to make an application consistent backup of our lab.

[FrancWest]

I've got it working more or less. I needed to add a second NIC to both server (backup server and guest interaction proxy in the lab) and put them on a separate network. Then the backup server is able to communicate directly with the interaction proxy in the lab and vice versa. Application consistent backup seems to work fine now, but Veeam is still confused about this config. It still thinks the SQL server in the lab (which has the same hostname and ip address as it's production counterpart) hold the Veeam database and therefor it will be processed last. But this isn't the case, it doesn't contain the Veeam database.

Our lab setup isn't that uncommon, in fact it's setup the same way as the Virtual Lab function of the Sure Backup job works withing Veeam.

[foggy]

Nice solution, thanks for sharing. Regarding the SQL Server, this is because the VM in the lab has the same uuid.bios parameter as the VM in production. Veeam B&R tracks its database location using this parameter, so to avoid this behavior you can simply change it in the lab VM's .vmx file (keep in mind that other software applications can also use it for their needs though).