Surebackup and multiple IP nets in one production network

[andreash]

Hi,

we are running several IP networks inside a single production Network on ESX5.5, and our Surebackup Jobs are failing since the upgrade to V9.5.

My production setup is like this (I know this is not best practice, but sometimes things get handed over to you and you have to live with it):
- one production network on three ESX 5.5 hosts, called "VM Production"
- multiple IP networks inside this single LAN
-- 192.168.1.0/24 with default GW 192.168.1.254 - this is where most servers live
-- 192.168.2.0/24 with default GW 192.168.2.254 - a management network
-- 192.168.3.0/24 with default GW 192.168.3.254 - servers for a separate project
- all these Default GWs are virtual IPs on a single Network port on the physical firewall device. No VLAN tagging in place.

All ~50 VMs on this Cluster are backed up using VEEAM 9.5, and replicated to a fourth (legacy) ESX 5.5 host with two datastores.
So i have two replication jobs, one for each datastore on the replication host.

Up until VEEAM 9 we had a weekly Surebackup Job set up that would test all VMs on the replication host. The Virtual Appliance was configured for Advanced Single Host Networking, with one single Isolated Network, and multiple vNICs in Network Settings:
- vNIC1: Masquerade IP 192.168.255.x, Appliance IP 192.168.1.254
- vNIC2: Masquerade IP 192.168.254.x, Appliance IP 192.168.2.254
- vNIC3: Masquerade IP 192.168.253.x, Appliance IP 192.168.3.254
This was working fine until the upgrade to V9.5. The appliance failed to start and I redeployed it, and afterwards servers in the first network could be verified successfully, but servers in 192.168.2 und .3 would still fail. I logged into the appliance and couldn't find any reference to these networks, so i decided to rebuild the appliance.
Unfortunately it would not allow me to create multiple vNICs for the same Isolated Network: "vNIC with such isolated Network connected already exists".

Support told me this setup was not supported and I should file a feature request (Case ID is 02029961). As this was working in the past I'd rather think this is a bug report

Any idea how to set this up in V9.5? Separating the single Network into multiple vLANs is not an option on the physical network, I'm afraid.

Regards,
Andreas

[Andreas Neufert]

Will check... Correct case number is: 02021609

[Andreas Neufert]

Asked team support team to investigate why this was changed.

[andreash]

Hi Andreas,
thank you for picking this up! Is this setup really so unusual?

I have tried to come up with a Workaround but have not yet been able to come up with something useful.

Regards,
Andreas

[Andreas Neufert]

I used this setup before and it was mentioned as solution for customers with above 9 Networks as well.

No feedback yet will ask again.

[Peter Draeger]

We have the same problem - esx6.0 and veeam 9.5. On version 9.0 everything was fine.
Existing shurebackups starts with "Error: An error occurred during host configuration."

This is what i tried so far:
I deleted all ShureBackups / Virtual Labs and tried to create them again. If I add more than one isolated network, all networks get the VLAN ID of the last network. It is not possible to add networks with different VLAN IDs. With Version 9.0, each network could be assigned its own VLAN ID.

If I create a vNIC, I get the error message: "Masquerade network address has been detected with warning. Masquerade network address <ip> is already in use by the virtual lab <vlabname>. Would you like to use these settings?


I will open a ticket

[Andreas Neufert]

This is something different and it is OK. THis is just a warning only and you can say yes.

[svenh]

thank you for picking this up! Is this setup really so unusual?

I have tried to come up with a Workaround but have not yet been able to come up with something useful.

While not exactly a secure solution (as you already know) I can assure you: this is not that unusual.

I myself have some (legacy) networks where there are two IP nets in one physical network. Mostly a public IPv4 range and a private RFC1918 one for inter-server communication and management.
("Legacy" because those are mostly networks from an older design, newer deployments use correctly separated networks via VLANs.)

[Andreas Neufert]

What I can tell you by now is that some of the customers had problems with this kind of unsupported setup and as well we had some cases were customers defined it by mistake that way. We investigate at the moment if there is a option to enable this configuration somehow without to run into the same problems again. This will need some time.

[svenh]

What I can tell you by now is that some of the customers had problems with this kind of unsupported setup and as well we had some cases were customers defined it by mistake that way. We investigate at the moment if there is a option to enable this configuration somehow without to run into the same problems again. This will need some time.

Thank you. I guess best bet would be an "Advanced Setting" in the "vNIC Connection Settings" dialog, allowing to add additional IP addresses.

[andreash]

Hi Andreas,
thanks for staying on top of this.

Unfortunately I haven't been able to come up with a workaround yet. While it is possible to skip the ping test entirely the Surebackup Job already fails when mapping the network.
I think this area could use some more granularity. For example I have tried to assign a second virtual IP from an independent Networks to a single Interface, and it appears Surebackup picks the virtual IP over the base IP, failing the ping test.
It would already be helpful to be able to exclude some servers from a Linked Job from Surebackup, so I could spread one Job over multiple Surebackup Jobs.

Regards,
Andreas

[atrotter]

You're not alone. I too just experienced this directly after upgrading from 9.0 to 9.5 Update 1. I used to have two subnets defined on the single isolated network and it worked fine in 9.0. Now, the SureBackup jobs fail at the ping test. The logs show the ping test utilizing the incorrect mask IP--that is, it uses the mask IP of the other subnet rather than the one that it was assigned to. I have a ticket open with Veeam, currently escalated to Tier 2. Hopefully they can provide some insight into the issue.

[andreash]

I have received the following Feedback from Tier 2 Support:
"Unfortunately, I have to state this feature ("allowing multiple vNICs with different masquerade/appliance IP addresses in Virtual Lab constructor for one isolated network") had been mistakenly allowed in Veeam version 8 and 9.
As of now this setup is not supported and is prohibited in the new version."

Btw, there's also another change in the surebackup logic: (Windows) Network interfaces with an alias IP get recognized with the alias IP, not the Primary one. This breaks another ping test in my case.

My workaround looks like this:
1. for every IP network, create a separate Virtual Lab with the Isolated Network mapped to this IP Network (i.e. I have "VLAB 192.168.1.0", VLAN "192.168.2.0" and so on)
2. for every IP Network, create an Application Group containing all VMs inside this Network with source Backup
3. for every IP Network, create a second Application Group containing all VMs inside this Network with source Replication
4. for every IP Network, create a Surebackup Job with the corresponding VLAB and Application Group with source Backup
4. for every IP Network, create a second Surebackup Job with the corresponding VLAB and Application Group with source Replication
So basically I am no longer testing "Linked Jobs" but Application Groups which means no parallel processing. This obviously also only works if there are no dependecies between those networks. We only run simple ping test currently, so that's acceptable for now.

Any changes in the VM-Infrastructure (i.e. new Servers) need to be manually added to the above setup, i can no longer simply add them to the replication job.

So, for me I no longer have two Surebackup Jobs (one for Backup and Replication each), but I will have 6 of them, plus the added overhead for maintenance.

Please, VEEAM Team, think about this and get us back the previous functionality!

Regards
Andreas

[andreash]

Major drawback #1: All VMs in an Application Group stay online, so I need a lot of Ressources or a lot of Jobs.
Major drawback #2: Processing stops if a VM brings up an error (doesn't boot in time because of Update Installation for example)

[JasonRoss]

I had this same issue and resolved it. I upgraded from 9 to 9.5, needed to modify a virtual lab, but was getting the error message "vNIC with such isolated Network connected already exists". I tried everything... I still wasn't able to complete the final creation stage with multiple isolated networks. Hope this helps somebody.

It is a simple mistake that I would ALMOST call a design flaw. - From the "Isolated Network" section after you click the Add button for additional networks, the middle "Isolated network" dropbox keeps the same name as the original one. Hence the VLAN ID changing for all of the networks.

Edit the name to match the production network. For example...

Production: xx Network
Isolated network: Virtual Lab xx Network
VLAN ID: xx

After this, assign a vNic for each isolated network with the appropriate network settings. Each masquerade address will need to be unique in the lab.

[TheMonk]

So came across this problem after upgrading to 9.5 and my colleague upgrading our vshpere environment to 6.5. Instantly I figured it was his fault but before outing him publicly I thought id google. Found this topic and I like others can concur that before 9.5 all was working.

Id tried everything and then found JasonRoss' fix and yep its worked. I think hes spot on with regards to it being a design flaw from veeam.

Anyways if you are experiencing this issue then follow what Jason has said and it will resolve

[Andreas Neufert]

Thanks for the feedback. The original problem was something different.

The Isolated network naming is the same since many versions and yes, you need to change them manually.

[TheMonk]

ive never had to change it manually its always changed itself.... looking at my existing VLabs that all worked and now don't they are infact correct.....however as soon as I go to edit the isolated network the one added is no longer there so can only default to the first one. As soon as I re-input the name of the isolated network and click ok the lab sets up and my SB jobs work again as they did in 9.0

[foggy]

In fact, the behavior was indeed changed in v9.5. As mentioned above, adding multiple vNICs with different network settings for a single isolated network was mistakenly allowed in previous versions (in fact such configuration worked in some specific cases only) and was intentionally disabled in v9.5.

[juergen.eisenbarth]

Hello,

the description from Jason works only with different VLAN IDs. With several different ip subnets and one VLAN ID, i can not create a Network mapping, because the Name of production Network in VMWare is the same.

My Workaround is one Virtual Lab for each IP subnet.

Regards
jürgen

[jharrell]

I had this same issue and resolved it. I upgraded from 9 to 9.5, needed to modify a virtual lab, but was getting the error message "vNIC with such isolated Network connected already exists". I tried everything... I still wasn't able to complete the final creation stage with multiple isolated networks. Hope this helps somebody.

It is a simple mistake that I would ALMOST call a design flaw. - From the "Isolated Network" section after you click the Add button for additional networks, the middle "Isolated network" dropbox keeps the same name as the original one. Hence the VLAN ID changing for all of the networks.

Edit the name to match the production network. For example...

Production: xx Network
Isolated network: Virtual Lab xx Network
VLAN ID: xx

After this, assign a vNic for each isolated network with the appropriate network settings. Each masquerade address will need to be unique in the lab.

I owe you a steak. I was stuck on this for a couple of hours. I didn't notice the isolated networks names not being updated when adding the multiple networks and couldn't figure out why I was not able to add multiple vNICS. Thank you sir!

P.S. I would call this a "bug"

[JKoons]

I know this discussion is a little old, but I am running into the same issues and Veeam support is saying this isn't possible to workaround.

One of our large vLANs has VMs configured with different default gateways, so I need to be able to setup networking in my Virtual Lab with another Isolated Network (which is already setup once) and then add another vNIC under Network Settings with my other gateway needed. Has anyone found a way to work around this issue?

[Andreas Neufert]

A bit strange setup. How do you route things with multiple Gateways/Default routes? Do you have local routing entries within the servers to achieve this?

Anyway if you need an additional router between lab networks, you can setup a router/firewall system with the second default gateway IP addresses to get the routing done.

As well there is the option that you define all the subnets (not only one) within the Virtual lab wizard and enable routing checkbox. This should do it as well.

I guess I am maybe not really able to follow what you want to do. Can you maybe do a quick drawing of the product situation and attache here?

[NickKulkarni]

Thanks for the help Jason

upgraded to v11a and it killed my surebackup jobs

I have a similar situation in that three of my Windows Server vmware VMs are multihomed because they double as Backup Proxies so have two seperate NIC on two seperate subnets one for production the other for backup.

Try as I might I couldn't see what was wrong with the error messages that the vNIC was assigned to a network that already existed.

This seems to be the answer

JasonRoss » Mar 08, 2017 1:14 am

It is a simple mistake that I would ALMOST call a design flaw. - From the "Isolated Network" section after you click the Add button for additional networks, the middle "Isolated network" dropbox keeps the same name as the original one. Hence the VLAN ID changing for all of the networks.

Edit the name to match the production network. For example...

Production: xx Network
Isolated network: Virtual Lab xx Network
VLAN ID: xx

After this, assign a vNic for each isolated network with the appropriate network settings. Each masquerade address will need to be unique in the lab.

[Frank_190]

This bit me again. I forgot about this post and opened a support case (Veeam Support - Case # 05636268). Is there any way Veeam can correct this issue? It is frustrating, to say the least.

[hegedus.gabor]

https://youtu.be/k47rs4-xCz8?t=429
Still not corrected the laziness of the programmer who thought how nice it is to use a dropdown menu prefilled with the same network name instead of a "new" button, where you can add a new network name corresponding to a "new" private vnic you want to create.
Instead, you have to create an explaining video and keep answering the same questions on forums, and support cases.

[Egor Yakovlev]

Hi all.

...after you click the Add button for additional networks, the middle "Isolated network" dropbox keeps the same name as the original one...

This behavior has been fixed in the upcoming v12. Do not forget to register for Veeam Backup & Replication V12 Launch Event!

/Thanks!

[jloube]

I just registered here to chime in on how silly this limitation is. Multiple networks running on the same VLAN is perfectly valid setup and it's super weird that Surebackup won't allow this.

2 separate virtual labs isn't really a valid workaround. What if I have a DB server on one network and a Web server on another, and I want to do a test that needs both?

[Mildur]

Hi Joel

2 separate virtual labs isn't really a valid workaround. What if I have a DB server on one network and a Web server on another, and I want to do a test that needs both?

That scenario works absolutely fine with Advanced Single-Host Virtual Labs and a single virtual lab.

https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Best,
Fabian

[jloube]

just to clarify, *one* VLAN, with *two* (or more) IP networks. There's no way to add network #2,3,4,etc to any virtual lab config as far as I can tell.

essentially, each vNIC requires a dedicated Isolated Network, and each Isolated Network requires a unique VLAN. And those are completely arbitrary limitations which make SureBackup completely incompatible with many perfectly valid network setups.