Virtual Lab advanced routing brainteaser

[joergr]

Alright, here comes a nice virtual lab routing brainteaser

Let´s say i backup two servers, servera (10.10.1.10/19) and serverb (10.10.1.11/19), the gateway of these in production is 10.10.0.201/19. then i create an app group covering these two machines. i then create a virtual lab, the production vnic of the proxy has the static configured ip 10.10.4.200/19. I create a surebackup job which only focuses on the app group (not additional connection to a backup job) and leave it to powered on after job completes. At proxy app conectivity i select to use the 10.255.x.x network and the 10.10.0.201 as appliance ip. i then define static mapings to both machines, machine 1 as production 10.10.1.10 masquerade as 10.10.1.100, and also the machine 2 production 10.10.1.11 masquerade as 10.10.1.101.

Now my intention is to boot the two machines in the isolated lab but be able to access them both from the production network via the 10.10.1.100 and 10.10.1.101.

The good news is: This works wonderful. WHEN my production machine is located in the 10.10.0.0/19 network. It can ping and access the masquerade ip - very well, very slick, very good.

But if i am coming from another production network, let´s say 172.16.0.0/16, the machine inside this network is not able to ping the masquerade ip. Now: The 10.10.0.201/19 has a working route to the 172.16.0.0/16 and vice versa. So my guess is the proxy appliance will get my ping request but is not able to transfer it back to my 172 machine.

Now it would be intersting how the proxy appliance transforms the ip routing logic, is it dynamically switching from the 10.10.1.101 to the 10.10.1.11 in the isolated network OR does it internally use the initial masquerade 10.255.1.11 to access it? Thus i could imaginge the internal logic tries to reply on the 10.255.x.x subnet and the translation logic on the proxy only listens to 10.255.x.x requests to transform (which would require me to add a route to the 10.255.x.x on my production router - but the manual says i don´t need this when doing static mappings). Or other way, the internal logic of the proxy app is not able to inject the standard gateway of the production network to the packets at the route back.

OK, this is maybe a very lame question (because i badly overlooked something very important - SHAME ON ME!!!!!) or it is a hardcore question to you guys;-)

Now which of both is it and do you have any ideas?

Best regards from germany,
Joerg

[Gostev]

Hi Joerg, I will talk to devs and ask for best way to troubleshoot this.

Answering your question, the switch is dynamic, appliance uses "real" (production) IP address to talk to VM that runs in the isolated environment. The notion of "masquerade address" does not exist on that other side of the fence, it only exists on production side.

[joergr]

Thanks Anton. Any suggestion or troubleshooting tip is very welcome
Best regards, Joerg

[Gostev]

Hi Joerg, already heard back from someone, and he suggested exactly the same thing I was thinking... try to ping 10.10.4.200 from any computer in 172.16.0.0 network.

[joergr]

Hi Anton, i think that worked. But not 100% sure, will verify tomorrow. What if my memory is right, if it worked, what would that mean?
Best regards, Joerg

[Gostev]

This should be working, so if it does, we will need to looks for some other reason.
Of course, make sure the proxy appliance is running before pinging (start SureBackup job).

[joergr]

Hi Anton,

you thought right, i can NOT ping the 10.10.4.200 from the production 172 network. I checked the virtual lab proxy settings and they say static ip of 10.10.4.200, 255.255.224.0 and 10.10.0.201 as gateway. Everything set OK. Could it be it somehow don´t uses the gateway?

Best regards,
Joerg

[Gostev]

Hi Joerg, I am not sure what you are saying here... proxy appliance is regular computer on your production network that you should be able to ping on static production IP address that you have provided in its settings. You can also open appliance console and verify that correct address was in fact used (appliance displays its network settings in its console). Please, verify the displayed settings and let me know. Thanks!

[joergr]

Hi Anton,

what i am saying is that maybe the appliance doesn´t use the configured standard gateway. That is because i can ping the 10.10.4.200 from my 10.10.0.0/19 production network. Will boot the appliance now and check for these setting showing there. Will be back in few mins.

Best regards,
Joerg

[joergr]

I am back
It doesn´t show the ip settings in the console. It only shows the eth´s are up but nothing more. Is there a way to login and do an ifconfig?
best regards, Joerg

[Gostev]

I will check with devs and email you... we have a public holiday so it may take some time to get an answer.

[joergr]

ok thanks.

[joergr]

Follow-up. I recreated the whole virtual lab just to be 100% sure because i strongly remembered yesterday the .200 was pingable from 172 network. And voila, after complete recreation i can ping the x.200 from within my 172 network again. But i can not ping the mapped .101 ip address from the 172 network. But I CAN ping it from the 10 network.

Now i continued my research and found out that when connecting to the isolated machine, you can not ping the fake gateway aka the proxy appliance (10.10.0.201) from it. Now this may be a feature (for example maybe the proxy appliance is not answering to icmp INSIDE the isolated network BY DESIGN). Or maybe it is a hint what might go wrong.

best regards,
Joerg

[Gostev]

Hi Joerg, another idea is to try to set internal IP address (on the isolated network side) of proxy appliance to address other than gateway IP address in the production network. May be this is what confuses the appliance when it gets packet destined to 172.16.x.x. It know that it has to send this off to gateway, but it has the gateway IP address on it own internal network adaptor. See the possible issue?

[joergr]

Hi Anton,

yes, that could it be, a good idea. I will try this. But i wonder if the client in the isolated network with a real ip of 10.10.1.10 and a masquerade to outside world of 10.10.1.100 it itsself has a gateway configured to 10.10.0.201. The problem here: As it is in an isolated network it can only reach out to the proxy appliance, not further. So let´s say i give the proxy appliance inside the isolated network another address than 10.10.0.201 (as you suggested), the isolated client won´t be able to access it´s own gateway. Thats my thoughts. But i will try it out

Best regards,
Joerg

[Gostev]

Yes, of course that is right, but for testing purposes you can simply adjust test VM settings manually... I am just trying to nail down the issue. By the way, appliance has tcpdump on it, if you know how to use it. Or we can just take a break for now, enjoy the weekend, and plan for webex with devs on Monday.

[Gostev]

Hi Joerg, another idea is to try to set internal IP address (on the isolated network side) of proxy appliance to address other than gateway IP address in the production network. May be this is what confuses the appliance when it gets packet destined to 172.16.x.x. It know that it has to send this off to gateway, but it has the gateway IP address on it own internal network adaptor. See the possible issue?

P.S. Just heard from actual appliance dev (finally), and he said this cannot be the issue. He wanted to see logs, but I think webex might be best indeed.

[joergr]

Hi Anton,

wow, that would be great (webex somewhere next week), but PLEASE no hurry at all, it is pure curiosity driving me, not urgency

best regards,
Joerg

[joergr]

Well, i think i found the root cause (but will check out in detail next week): It seems to me that the static mappings are applied to the proxy appliance without gateway. They should inherit the gateway from either the original vm or from the proxy appliance ittself but since the proxy appliance ittself could have an ip from another network that could be unstable so i guess the best catch would be to inherit the gateway from the original vm it actually masquerades.

Thus, the static mappings are accessible from within the network they reside in but not from foreign networks.

A workaround i found out when you are in the need to access the isolated machines from within a foreign network: Assign the proxy appliance an ip of the foreign network, then either add a new default route to the 10.255.0.0/19 to your main gateway or for testing purposes only to your local machine (which resides in the foreign 172 network), via route 10.255.0.0 mask 255.255.224.0 172.16.99.220 (that is my new ip of the proxy appliance).

Workaround works, the masquerade 10.255.x.x is now pingable from the 172 machine - but ofcourse static mapping would be much smarter because you won´t have to modify or touch any routes or local routes at all. Thus, i would be very happy to work this out together with your dev guys

But again: NO HURRY. This is not urgent and can wait for weeks. It is just for curiosity and research

Best regards,
Joerg

[larry]

My sure backup lab at my DR site has 3 networks attached to it. All 3 networks route between themselves fine. I have some static mapping that I can access from the local network. I want to access the static mapping from a remote network, which I cant get working. It seems that the proxy server doesn’t set a default route to the local router which is the proxy servers address to the isolated network. How do I set this up ?

[depps]

Was a permanent solution for this ever developed?

[Vitaliy S.]

Hi Daniel, can you please describe your network setup and tell us what issue you're facing, so I could try to answer your question? Thanks!

[depps]

I guess I have two questions, heres the first

3 Sites involved in this question

London LHQ, workstation VLAN 10.1.103.0
London DC1, production server VLAN 10.8.4.0 - veeam vlan 10.8.1.0
London DC2, production server VLAN 10.9.4.0 - veeam vlan 10.8.1.0
DC1 Veeam server patched into both VLAN's (4GB trunk with 802.1Q) - 10.8.1.100 and 10.8.4.100
DC2 Veeam server patched into both VLAN's (4GB trunk with 802.1Q) - 10.9.1.100 and 10.9.4.100

Default gateways 10.x.x.254

DC1 ServerA 10.8.4.11
DC1 ServerA static mapping 10.8.4.111
DC1 VLAB eth0 10.8.4.200
DC1 VLAB eth1 10.8.4.254


DC1-SB-01 Surebackup job that runs at DC1 includes ServerA as above

During this SB job I can ping/mstsc 10.8.4.111 from other VM's in the DC1 production server VLAN

I can not however ping .111 from DC2 or LHQ sites

.111 can not ping DC1 VLAB eth1 10.8.4.254

LHQ and DC2 can ping DC1 VLAB eth0 10.8.4.200

VLAB eth0 10.8.4.200 can ping all sites

Case # 00641779