Bridge mode: some MACs not learned

vvk

I have openvpn server in bridge mode setup and some clients connected to it.

Server running on Ubuntu 12.04, openvpn 2.2.1, server config:

port 1194
proto tcp
dev tap0

ca   /etc/openvpn/keys/ca.crt
key  /etc/openvpn/keys/server.key # This file should be kept secret.
cert /etc/openvpn/keys/server.crt
dh   /etc/openvpn/keys/dh1024.pem

cipher AES-256-CBC

ifconfig-pool-persist ipp.txt
server-bridge 192.168.253.4 255.255.255.0 192.168.253.100 192.168.253.200
keepalive 10 120
user nobody
group nogroup
persist-key
persist-tun
status /var/log/openvpn-mikrotik-status.log
verb 3

Tap device for openvpn created from /etc/network/interfaces and it's included to br0 bridge, along with eth0.7 interface.

iface br0 inet static
        bridge_ports eth0.7
        bridge_fd 0
        bridge_stp off
        address 192.168.71.253
        netmask 255.255.254.0
        up openvpn --mktun --dev tap0
        up ip link set dev eth0.7 promisc on
        up ip link set dev tap0 promisc on
        up ip link set dev tap0 up
        up ip a a 192.168.74.1/24 dev br0
        up brctl addif br0 tap0
        down brctl delif br0 tap0
        down openvpn --rmtun --dev tap0

Clients are mikrotik RouterOS routers. At mikrotik side we have ovpn-out1 interface, which is openvpn connection to the server. This interface and some local eth inetrfaces included in bridge "vpn-bridge". So, basically we plug some devices onto mikrotik, and they are transparently gets L2 connectivity to server's eth0.7.

This setup is working and most of devices gets their transparent access to the vlan7. The problem is: some special devices are not accessible after plugging into mikrotik router. When mikrotik connects to the server, we see record "MULTI: Learn: " in log, which represents all mac addresses of the devices, connected to mikrotik, EXCEPT macs of "some special devices. For some reason their MACs staying unknown to openvpn, so no connectivity with them in the result.

....
Feb 21 14:17:51 hydrogen ovpn-mikrotik-gw[21061]: mikrotik-3218/192.168.74.21:35656 MULTI_sva: pool returned IPv4=192.168.253.105, IPv6=1::2100:0:e47f:0
Feb 21 14:17:51 hydrogen ovpn-mikrotik-gw[21061]: mikrotik-3218/192.168.74.21:35656 PUSH: Received control message: 'PUSH_REQUEST'
Feb 21 14:17:51 hydrogen ovpn-mikrotik-gw[21061]: mikrotik-3218/192.168.74.21:35656 send_push_reply(): safe_cap=960
Feb 21 14:17:51 hydrogen ovpn-mikrotik-gw[21061]: mikrotik-3218/192.168.74.21:35656 SENT CONTROL [mikrotik-3218]: 'PUSH_REPLY,route-gateway 192.168.253.4,ping 10,ping-restart 120,ifconfig 192.168.253.105 255.255
.255.0' (status=1)
Feb 21 14:17:51 hydrogen ovpn-mikrotik-gw[21061]: mikrotik-3218/192.168.74.21:35656 MULTI: Learn: d4:ca:6d:af:85:c5 -> mikrotik-3218/192.168.74.21:35656
Feb 21 14:17:52 hydrogen ovpn-mikrotik-gw[21061]: mikrotik-3218/192.168.74.21:35656 MULTI: Learn: 00:d0:89:0a:4b:79 -> mikrotik-3218/192.168.74.21:35656
Feb 21 14:18:06 hydrogen ovpn-mikrotik-gw[21061]: mikrotik-3218/192.168.74.21:35656 MULTI: Learn: 00:20:4a:ef:17:ed -> mikrotik-3218/192.168.74.21:35656
Feb 21 14:18:08 hydrogen ovpn-mikrotik-gw[21061]: mikrotik-3218/192.168.74.21:35656 MULTI: Learn: 00:05:fe:03:72:54 -> mikrotik-3218/192.168.74.21:35656

I've looked inside src/openvpn/multi.c trying to determine how openvpn learns mac addresses, but it looks way complicated for me.

From openvpn server side I can ping special device with no success, BUT while doing this, MAC address of the device becomes known to linux arp cache (but not to openvpn):

# ping 192.168.70.187
PING 192.168.70.187 (192.168.70.187) 56(84) bytes of data.
^C
--- 192.168.70.187 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2009ms
# arp 192.168.70.187
Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.70.187           ether   00:0c:c6:79:e4:e0   C                     br0

While doing this, I see ARP query and answer:

# tcpdump -n -i br0 -e host 192.168.70.187
...
14:53:58.279279 0a:a7:16:d4:ff:46 > 00:0c:c6:79:e4:e0, ethertype ARP (0x0806), length 42: Request who-has 192.168.70.187 tell 192.168.71.253, length 28
14:53:59.263396 0a:a7:16:d4:ff:46 > 00:0c:c6:79:e4:e0, ethertype IPv4 (0x0800), length 98: 192.168.71.253 > 192.168.70.187: ICMP echo request, id 28521, seq 24, length 64
14:53:59.279342 0a:a7:16:d4:ff:46 > 00:0c:c6:79:e4:e0, ethertype ARP (0x0806), length 42: Request who-has 192.168.70.187 tell 192.168.71.253, length 28
14:54:00.263431 0a:a7:16:d4:ff:46 > 00:0c:c6:79:e4:e0, ethertype IPv4 (0x0800), length 98: 192.168.71.253 > 192.168.70.187: ICMP echo request, id 28521, seq 25, length 64
14:54:00.279347 0a:a7:16:d4:ff:46 > 00:0c:c6:79:e4:e0, ethertype ARP (0x0806), length 42: Request who-has 192.168.70.187 tell 192.168.71.253, length 28
14:54:01.263384 0a:a7:16:d4:ff:46 > 00:0c:c6:79:e4:e0, ethertype IPv4 (0x0800), length 98: 192.168.71.253 > 192.168.70.187: ICMP echo request, id 28521, seq 26, length 64
14:54:02.264938 0a:a7:16:d4:ff:46 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.70.187 tell 192.168.71.253, length 28
14:54:02.284016 00:05:fe:03:72:60 > 0a:a7:16:d4:ff:46, ethertype ARP (0x0806), length 60: Reply 192.168.70.187 is-at 00:0c:c6:79:e4:e0, length 46
14:54:02.284026 0a:a7:16:d4:ff:46 > 00:0c:c6:79:e4:e0, ethertype IPv4 (0x0800), length 98: 192.168.71.253 > 192.168.70.187: ICMP echo request, id 28521, seq 27, length 64
14:54:03.273376 0a:a7:16:d4:ff:46 > 00:0c:c6:79:e4:e0, ethertype IPv4 (0x0800), length 98: 192.168.71.253 > 192.168.70.187: ICMP echo request, id 28521, seq 28, length 64

The only way I can restore network connectivity to this special device is reboot it. After reboot it works while openvpn tunnel is up. If it restarts for some reason, device became inaccesible again.

So, my question is: how does exactly openvpn learns MACs which belongs to client side, and probably advice about how to figure out why these special devices are so special with such an unexpected network behavior.

vvk

OK, looks like arp reply from these special devices comes from different MAC address than they have:

17:29:14.127099 0a:a7:16:d4:ff:46 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.70.187 tell 192.168.71.253, length 28
17:29:14.159350 00:05:fe:03:72:60 > 0a:a7:16:d4:ff:46, ethertype ARP (0x0806), length 60: Reply 192.168.70.187 is-at 00:0c:c6:79:e4:e0, length 46

See? 00:05:fe:03:72:60 != 00:0c:c6:79:e4:e0. Probably this is the cause - openvpn doesn't like such packets, thus not allowing to learn MACs. Probably I should patch openvpn to change that behavior.

Some note about these devices: they are Traficam network cameras. They uses device called Traficam Interface to access network because cameras doesn't have own ethernet port (yet they are ethernet devices) - they uses BPL interface. So Traficam Interface just provides network connectivity for them. This means when you trying to access Traficam, TI (traficam interface controller) do the work, and exactly this device performs ARP handling.

Unfortunately, I cannot fix traficams, I can only patch openvpn.

vvk

Here is full routing table, related entries are highlighted:

# ip ro
default via 77.242.110.177 dev eth1
10.8.0.0/24 dev tun1 proto kernel scope link src 10.8.0.1
10.17.2.0/24 via 10.222.0.5 dev tun2
10.202.153.0/24 dev eth0.8 proto kernel scope link src 10.202.153.254
10.222.0.0/15 via 10.222.0.5 dev tun2
10.222.0.5 dev tun2 proto kernel scope link src 10.222.0.7
10.225.1.0/24 via 192.168.254.2 dev tun0
77.242.110.176/29 dev eth1 proto kernel scope link src 77.242.110.178
109.233.228.20 via 192.168.254.2 dev tun0
172.16.0.0/16 via 10.222.0.5 dev tun2
172.16.3.1 via 10.202.153.1 dev eth0.8
172.18.36.0/24 dev eth0 proto kernel scope link src 172.18.36.253
172.21.0.0/16 via 10.222.0.5 dev tun2
192.168.0.0/16 via 10.222.0.5 dev tun2
192.168.62.0/24 dev eth0.13 proto kernel scope link src 192.168.62.1
192.168.63.0/24 dev eth0.11 proto kernel scope link src 192.168.63.253
192.168.64.0/24 dev eth0.10 proto kernel scope link src 192.168.64.253
192.168.65.128/25 dev eth0.9 proto kernel scope link src 192.168.65.253
192.168.66.0/24 dev eth0.3 proto kernel scope link src 192.168.66.9
192.168.67.0/24 dev eth0.4 proto kernel scope link src 192.168.67.253
192.168.68.0/24 dev eth0 proto kernel scope link src 192.168.68.253
192.168.69.0/24 dev eth0.6 proto kernel scope link src 192.168.69.253
192.168.70.0/23 dev br0 proto kernel scope link src 192.168.71.253
192.168.74.0/24 dev br0 proto kernel scope link src 192.168.74.1
192.168.254.2 dev tun0 proto kernel scope link src 192.168.254.1

As you see, 192.168.70.0/23 192.168.74.0/24 networks are both present on br0 now, this is due to transition scheme. After installation of all mikrotiks, 192.168.70.0/23 will stay at br0, while 192.168.74.0/24 will be moved to new vlan, which will provide connectivity between mikrotiks and openvpn (insecure network).

vvk

It seems that openvpn only analyzes ethernet frame addresses in packets, and not "sender hardware address" inside ARP messages. So, cameras MACs stays unknown to openvpn until they send packets from itself (e.g. source MAC in ethernet frame = camera MAC, and not TI MAC).

vvk

debbie10t wroteTechnically, ARP is handled by the operating system not OpenVPN. Under a correctly created bridge network ARP requests will be forwarded the entire bridge network (this is one of the performance issues associated with an OSI-L2 Bridge setup).

Openvpn handles "MAC - client" associations when it's working in bridged mode. See src/openvpn/multi.c multi_learn_addr() and where it is called. When openvpn learns such mapping, we see log entries like

Feb 25 07:12:58 hydrogen ovpn-mikrotik-gw[21061]: mikrotik-3233/192.168.74.36:36748 MULTI: Learn: 4c:72:b9:d1:ed:91 -> mikrotik-3233/192.168.74.36:36748

in syslog and entries like

4c:72:b9:d1:ed:91,mikrotik-3233,192.168.74.36:36755,Tue Feb 25 14:59:35 2014

in openvpn status log.

In my case linux bridge performs all needed ARP handling (MAC addresses successfully populates arp cache) but openvpn doesn't handles arp replies so it knows nothing about these MACs.

Where does openvpn takes "4c:72:b9:d1:ed:91" to learn? It analyzes ethernet frame header. As I already showed above, in my case ethernet frame MAC differs from arp message:

17:29:14.127099 0a:a7:16:d4:ff:46 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.70.187 tell 192.168.71.253, length 28
17:29:14.159350 00:05:fe:03:72:60 > 0a:a7:16:d4:ff:46, ethertype ARP (0x0806), length 60: Reply 192.168.70.187 is-at 00:0c:c6:79:e4:e0, length 46

In this particular example openvpn learns 00:05:fe:03:72:60 but not 00:0c:c6:79:e4:e0, which I need.

Correct connection scheme is:

:Local :                :            Remote            :
:Server: -- Internet -- :Client: == Ethernet == TI interface == BPL link == :Traficam Camera:
:Bridge:                :  PC  :                : *eth : 
: *eth :                : *eth :                :      :
: *tap :                : *tap :                :      :

In the case of #2, provided you have routing setup for your client network, you should be able to contact your camera without your camera having to announce itself first. Technically, you do not even require TAP/OSI-L2 for a standard IP camera.

Yes, routed setup should work, but it's not an option due to another reasons. Best solution for me is to patch openvpn to implement ARP parsing. Probably I should write to openvpn-devel list to contact devs to ask their opinion.

vvk

Sure, mikrotik running in bridge mode.

All plugged devices except traficams working well. Finally I've patched openvpn to add MAC extraction not only from ethernet header, but also from ARP packets and it totally solved my problem. I'm going to submit this patch to openvpn-devel.

vvk

I suggest not to close this thread for a while, I'll post final conclusion later.

vvk

Here is the relevant thread on openvpn-devel. Patch was ACKed, but wasn't applied.

http://sourceforge.net/p/openvpn/mailman/openvpn-devel/thread/1394186298-226674-1-git-send-email-vvk%40vvk.pp.ru/#msg32072629