Weird loss-of-connectivity on reconnect problem...

baekgaard

I have a remote system that I maintain. I do this normally via ssh, but occasionally I need to act as a local user on the net and therefore use OpenVPN. I have had this running for years in a bridge configuration (mainly as the server is not the gateway and I cannot easily reconfigure the main gateway for a proper routed setup). And just to give the complete picture, I have also had a tunneled configuration set up, but due to previously overlapping local IP client/server addresses space(!) it had some interesting double-remapping using iptable etc.

My problem is this: I can connect and everything works absolute fine. If I disconnect, everything is still fine. However, if I then reconnect within some yet-to-be-determined time, the vpn connection appears to be set up correctly from the log files and the resulting state -- but as soon as it is established, I completely loose the connectivity to the server. I cannot even connect to the server via ssh nor telnet to other services on the server, even from another machine here locally on my NAT'ed client network. The time is more than 120 seconds, more likely upwards of 15 min, maybe somehow randomly determined.

My local network appears to be fine with no "visible" defects elsewhere, and I can ping the router on the server network so "global" connectivity looks good.

I suspect that "something" happens on the server that causes it to either 1) loose connectivity completely to its network, or 2) gets confused as how to route packets/connect back to my network. An alternate possibility is that another network component (e.g. the layer 2 switch in front of the server or its ISP provided router) gets confused somehow, possibly due to ARP polution or something else unforeseen.

However, it could also just be a simple problem in my setup files :-)

Any ideas on how to debug this further or any clues as to where the problem may lie?

The server is Debian 6 based and the client is Ubuntu 12.10 based.

Server config:

port 1194
proto udp
dev tap0
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/server.crt
key /etc/openvpn/keys/server.key
dh /etc/openvpn/keys/dh1024.pem
ifconfig-pool-persist ipp.txt
server-bridge 10.0.0.2 255.255.255.0 10.0.0.192 10.0.0.250
client-to-client
keepalive 10 120
tls-auth /etc/openvpn/keys/ta.key 0
comp-lzo
max-clients 50
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
verb 3

Client config (normally running via NetworkManager, but also fails when re-invoked directly from a shell):

client
remote xxx.tls # real servername goes here obviously
ca /home/me/vpn/ca.crt
cert /home/me/vpn/per.crt
key /home/me/vpn/per.key
auth-user-pass
comp-lzo yes
dev tap
proto udp
tls-auth /home/me/vpn/ta.key 1
nobind
auth-nocache
script-security 2
persist-key 
persist-tun
user openvpn
group openvpn

-- Per.

maikcat

please post the output of:

ifconfig (on linux server)
brctl show (on linux server)
netstat -nr (on client)

Michael.

baekgaard

Sorry, we encountered an error while displaying this content. If you're a user, please try again later. If you're an administrator, take a look in your Flarum log files for more information.

baekgaard

OK, so I've been able to figure out what goes wrong. The "short" version is this:

When the connection is created a somewhat random MAC address is assigned to the tap0 interface on the client side. This is the MAC address that is also associated with the IP address of client on the server network. In one case, the internal IP address of the server is 10.0.0.2 and the client ip becomes 10.0.0.192. The server originally has a MAC address of 80:ee:73:2b:cf:8d and the client random one could be e.g. 12:be:ca:0b:ac:37.

Thus, on the server network, the ARP "layer" should associate 10.0.0.2 with 80:ee:73:2b:cf:8d and 10.0.0.192 with 12:be:ca:0b:ac:37.

However, sometimes (but not always, it looks) the clients MAC address gets used for the servers own IP (in an ARP request, or in a gratituous ARP reply or by self-learning) and thus becomes associated with the wrong MAC address on the router!

Before a failing connection is established, the router has the correct mapping 10.0.0.2 -> 80:ee:73:2b:cf:8d in its ARP cache. But once a failing connection is established, the router suddenly maps 10.0.0.2 -> 12:be:ca:0b:ac:37!

When I ping the router, it apparently just replies to the MAC address of the sender, which is why I can ping the router but not get connected to the net even though the ARP cache is incorrect.

When the router tries to send packets to the server on 10.0.0.2, it uses the wrong MAC address.

The server does indeed receive these (it is in promiscuous mode as required by the current bridge setup), but since the MAC is the one of the client, the VPN on layer 2 tries to forward these back to the client, and thus immediately sends a packet to the client's public IP instead of forwarding it upwards on the internal stack.

If I flush the ARP cache on the router, everything starts to work immediately.

So... the "root" cause seems to be that the ARP cache of the router gets poluted by an incorrect entry.

Somewhere a package is presumable sent out from the server claiming the IP address of the server is on the MAC address of the client; instead it should have sent out the IP address of the client being on the MAC address of same client. Or maybe some "self-learning" is taking place on the router on some other bogus package? I have tried tcpdump on the eth0 and the br0 interfaces, but don't see such a package. But it could still be happening somewhere below the capture interface and/or in the VPN code, I guess?

The question is now what causes this, and how to fix it...

-- Per.

baekgaard

Looks like I'm almost talking to myself here... :-)

But for the record, and in the hope of getting further hints/help to fix the problem, here is some further information, that at least explains what goes on and some questions:

As pr. above, the reason for the loss of connectivity is because the ARP cache on the router gets an incorrect entry whereby the IP address of the VPN server no longer points to the real MAC address of it, but instead starts pointing to the assigned "random" MAC address of the client on the VPN bridge.

I've been debugging this further, and with tcpdump/wireshark, what I see is this:

1) During the negotiation sequence, everything looks correct: The gateway forwards packets to the VPN server (10.0.0.2) from the VPN client, using the real MAC address (80:ee:73:2b:cf:8d).

2) At some point towards the end of the setup sequence, the VPN server sends out this multicast package:

0000  01 00 5e 00 00 16 be 59  28 d9 d0 85 08 00 46 c0   ..^....Y (.....F.
0010  00 28 00 00 40 00 01 02  f9 f7 0a 00 00 02 e0 00   .(..@... ........
0020  00 16 94 04 00 00 22 00  f9 02 00 00 00 01 04 00   ......". ........
0030  00 00 e0 00 00 fb

... which is a "Membership Report / Join group 224.0.0.251 for any sources".

This is a multicast (IP.dst is 224.0.0.22 and eth.dst is 01:00:5e:00:00:16/IPv4mcast_00:00:16 as expected). BUT -- here comes the culprit -- it is sent with the IP.src of the VPN server (10.0.0.2) and the eth.src of the VPN client (here 59:28:d9:d0:85:08).

I believe this is incorrect; it should either announce the presense of the VPN client using its IP (10.0.0.192) and MAC (59:28:d9:d0:85:08) OR the IP of the server (10.0.0.2) and its own MAC (80:ee:73:2b:cf:8d). It further has the router alert option set (options == 148).

For the record: All other packages sent from the VPN server has the correct MAC address.

3) In any case, this causes subsequent packages from the router destined for the VPN server IP to have the layer 2 MAC address (eth.dst) of the VPN client -- which in turns causes an incorrect routing inside the VPN server layers :-(

4) A manual "ip arp flush" on the router immediately causes a new ARP request from the router which fixes the problem, and the communication is restored and continues as it should.

So... the next question is then: Why does the VPN server send out an IGMP message announcing (amongst other things) that the VPN server IP address now is incorrectly associated with the layer 2/MAC address of the VPN client?

And how do I disable that (or fix the problem -- it probably should have announced the presense of the VPN client on the server network, 10.0.0.192 instead)?

Further help/input much appreciated!

-- Per.

maikcat

Looks like I'm almost talking to myself here... :-)

no you are not...

just to remind you though that we are not all developers here... 😉

Michael.

baekgaard

maikcat wrote
Looks like I'm almost talking to myself here... :-)
no you are not...

Thanks, Michael :-) Good to know I'm not alone -- and if nothing else, maybe someone someday will have a similar problem and figure out how to fix it.

Anyway, hopefully now a final follow-up: The root cause was a "weird" interaction with one of the settings that relates to the tunneling setup. In order to do the needed remapping in that setup, I had some ipchains tables set up. One of these involved a masquerading on outgoing packages. It was unforeseen (by me at least) that I would see IGMP packages going out from the bridge itself, but as a consequence of the masquerading, I had its source IP remapped -- with disasterous results in this case. Apparently it didn't affect other (normal) packages that were just routed via the tap interface to the network, as other connectivity was OK once the ARP entries were correct. And kind of weird that it didn't happen all the time but only sometimes.

Now I've deleted the remapping setup and the ipchains masquerading, and all is fine with proper IGMP packages going out.

Glad I noticed this just; next step would have been to dig into the kernel code :-)

Thanks for your time and input,

-- Per.

maikcat

i assume that these were causing problems..

iface tun0 inet manual
post-up iptables -t nat -A PREROUTING -i tun0 -d 10.3.16.0/24 -j NETMAP --to 10.0.0.0/24
post-up iptables -t nat -A POSTROUTING -o br0 -j MASQUERADE
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
pre-down iptables -t nat -D PREROUTING -i tun0 -d 10.3.16.0/24 -j NETMAP --to 10.0.0.0/24
pre-down iptables -t nat -D POSTROUTING -o br0 -j MASQUERADE
openvpn tun-server

glad for you to figure this one out yourself.

Michael.

baekgaard

Yes, Michael -- specifically the execution of

iptables -t nat -A POSTROUTING -o br0 -j MASQUERAD

on ip up was the culprit.

The bridge obviously works on layer 2 and iptables works (as the name implies) only on layer 3, so as expected, all other packages through the bridge (i.e. between the tap0 and the eth0 interfaces) were not affected by this mapping. When it was running without connectivity problems, VPN traffic through bridge seemingly worked well.

The problem was caused by the IGMP packages. Some parts of IGMP (i.e. IGMP snooping) is sort of a layer 2 optimization working on layer 3, so I guess it can be said that IGMP is operating somehow across the layers, even though it resides in layer 3/just above layer 3.

I hadn't foreseen that something related to the tap0 interface (the joining of a new IP node) would trigger IP/IGMP packages being sent out locally, passing the through the iptables from above the tap0 layer, and thus be masqueraded, having their IP source changed.

Mea culpa!

-- Per.