Hi all,
I'm trying to achieve an OpenVPN setup with dynamic routing.
Currently, I have 5 sites using the private IP ranges 10.10.1-5.0/24.
I have installed and configured an OpenVPN server on each entry point, which has a public IP address.
The configuration is generated by ansible, so it is fairly similar across the 5 machines. Here is an example, for machine 2:
port 1194
proto udp
dev tap0
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/server.crt
key /etc/openvpn/keys/server.key
dh /etc/openvpn/keys/server-dh.pem
topology subnet
#server 10.245.2.0 255.255.255.0, but giving server IP addres 10.245.2.254
mode server
tls-server
push "topology subnet"
ifconfig 10.245.2.254 255.255.255.0
ifconfig-pool 10.245.2.1 10.245.2.253 255.255.255.0
push "route-gateway 10.245.2.254"
#end server expand
ifconfig-pool-persist ipp.txt
keepalive 10 120
tls-auth /etc/openvpn/keys/server-ta.key 0
cipher AES-256-CBC
persist-key
status openvpn-status.log
verb 9
mute 2
As you can see, 10.245/16 is used as the OpenVPN point-to-point range, the way it is assigned is 10.245.[third byte of server IP address].[third byte of client IP address] (through the pool persist file).
Furthermore, here is a client configuration file, e.g. the file from server 1 to server 2:
client
dev tap2
proto udp
remote x.x.x.x 1194
resolv-retry infinite
nobind
persist-key
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/client.crt
key /etc/openvpn/keys/client.key
remote-cert-tls server
tls-auth /etc/openvpn/keys/ta-2.key 1
cipher AES-256-CBC
verb 9
mute 2
so that, whever an OpenVPN client tries to connect to the OpenVPN server N, it uses tapN (here: tap2 is used to connect to server2, while server 2 is on tap0).
The overall topology between the 5 servers is a full-mesh with a single direct connection between the 5 sites.
The way it is achieved is using a (sort of) greedy algorithm: connect to the other ones unless they are connected to me.
Effectively:
- server 1 does not have any client, and uses tap2-5 to connect to servers 2-5
- server 2 has server 1 as client on tap0, and uses tap3-5 to connect to servers 3-5
- ...
- server 5 only has clients connected to its tap0, and does not connect to anyone as a client.
This topology and ecosystem works well if I manually push the routes, e.g., on server 1
ip route add 10.10.2.0/24 via 10.245.2.254 dev tap2
ip route add 10.10.3.0/24 via 10.245.3.254 dev tap3
ip route add 10.10.4.0/24 via 10.245.4.254 dev tap4
ip route add 10.10.5.0/24 via 10.245.4.254 dev tap5
and the reverse routes to 10.10.1.0/24 on the tap0 of the other machines.
As stated earlier, I am really after dynamic routing as it would allow to easily add/remove sites; moreover it provides possible paths across another site if a direct connection came to fail.
So, I have a Quagga ospfd configuration running on each of the servers, alongside OpenVPN.
OSPF routers use the multicast addresses 224.0.0.5 and 224.0.0.6 to discover one another and exchange reachability information.
This is the main (only) reason why I am running TAP devices.
Here is the Quagga configuration for server 2
! -*- ospf -*-
log syslog
!
hostname ip-10-10-2-254
password zebra
router ospf
ospf router-id 10.10.2.254
redistribute static metric 100
network 10.10.2.240/28 area 0
interface eth0
ip ospf cost 100
interface tap0
ip ospf network point-to-multipoint
ip ospf cost 100
ip ospf area 0
interface tap3
ip ospf network point-to-point
ip ospf cost 250
ip ospf area 0
interface tap4
ip ospf network point-to-point
ip ospf cost 250
ip ospf area 0
interface tap5
ip ospf network point-to-point
ip ospf cost 250
ip ospf area 0
debug ospf packet hello
As you can see, each of the TAPs are configured as point-to-point interface, appart from the tap0 server device, which is configured as a point-to-multipoint interface.
Again, this configuration is generated by ansible and is similar to the other ones.
Good news: this configuration works, maybe 10% of the time 🙂
Bad news: it fails 90% of the time :/
What happens is that some OSPF routers do not appear to be able to talk to one another over the 224.0.0.5.
For example, using the quagga terminal on server 2:
Hello, this is Quagga (version 1.2.4).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
ip-10-10-2-254# show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface RXmtL RqstL DBsmL
10.10.1.254 1 Init/DROther 32.524s 10.245.2.1 tap0:10.245.2.254 0 0 0
10.10.5.254 1 Full/DROther 39.115s 10.245.5.254 tap5:10.245.5.2 0 0 0
ip-10-10-2-254# exit
As you can see, server 2 sees server 5 as an OSPF router and has succcessfully established an OSPF adjacency with it (state is FULL), it also sees server 1 as an OSPF router but fails to establish adjacency (state is Init, meaning that it sees some HELLO packets, but not anything further).
On the other side (server 1), we have
Hello, this is Quagga (version 1.2.4).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
ip-10-10-1-254# show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface RXmtL RqstL DBsmL
10.10.4.254 1 Full/DROther 38.730s 10.245.4.254 tap4:10.245.4.1 0 0 0
where server 1 only sees server 4, and has fully established adjacency, but none of the other 3.
Now, if I go on server 2, and tcpdump tap0, I see this
$ sudo tcpdump -i tap0 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:35:30.399645 IP 10.245.2.254 > 224.0.0.5: OSPFv2, Hello, length 48
19:35:36.412264 IP 10.245.2.1 > 224.0.0.5: OSPFv2, Hello, length 44
19:35:40.399850 IP 10.245.2.254 > 224.0.0.5: OSPFv2, Hello, length 48
19:35:46.412304 IP 10.245.2.1 > 224.0.0.5: OSPFv2, Hello, length 44
meaning that OSPF packets in both directions (from server 2, and to server 2 from client 1) arrive correctly in tap0.
Similarly, if I go on server 1, and tcpdump tap2, I see this
ubuntu@ip-10-10-1-254:~$ sudo tcpdump -i tap2 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap2, link-type EN10MB (Ethernet), capture size 262144 bytes
19:36:56.410380 IP 10.245.2.1 > 224.0.0.5: OSPFv2, Hello, length 44
19:37:00.410840 IP 10.245.2.254 > 224.0.0.5: OSPFv2, Hello, length 48
19:37:06.410525 IP 10.245.2.1 > 224.0.0.5: OSPFv2, Hello, length 44
19:37:10.411084 IP 10.245.2.254 > 224.0.0.5: OSPFv2, Hello, length 48
meaning that, once again, packets appear to be transmitted as they should be.
However, if I look at the debug output of quagga, on server 2:
ubuntu@ip-10-10-1-254:~$ journalctl -u ospfd
Dec 14 19:35:30 ip-10-10-2-254 ospfd[30785]: Hello sent to [224.0.0.5] via [tap0:10.245.2.254].
Dec 14 19:35:30 ip-10-10-2-254 ospfd[30785]: Hello sent to [224.0.0.5] via [tap3:10.245.3.2].
Dec 14 19:35:30 ip-10-10-2-254 ospfd[30785]: Hello sent to [224.0.0.5] via [tap4:10.245.4.2].
Dec 14 19:35:30 ip-10-10-2-254 ospfd[30785]: Hello sent to [224.0.0.5] via [tap5:10.245.5.2].
Dec 14 19:35:36 ip-10-10-2-254 ospfd[30785]: Hello received from [10.10.1.254] via [tap0:10.245.2.254]
Dec 14 19:35:36 ip-10-10-2-254 ospfd[30785]: src [10.245.2.1],
Dec 14 19:35:36 ip-10-10-2-254 ospfd[30785]: dst [224.0.0.5]
and so the ospfd process on server 2 receives the hello packet from server 1
Now, the same debug output on server 1:
Dec 14 19:36:56 ip-10-10-1-254 ospfd[30868]: Hello sent to [224.0.0.5] via [tap0:10.245.1.254].
Dec 14 19:36:56 ip-10-10-1-254 ospfd[30868]: Hello sent to [224.0.0.5] via [tap2:10.245.2.1].
Dec 14 19:36:56 ip-10-10-1-254 ospfd[30868]: Hello sent to [224.0.0.5] via [tap3:10.245.3.1].
Dec 14 19:36:56 ip-10-10-1-254 ospfd[30868]: Hello sent to [224.0.0.5] via [tap4:10.245.4.1].
Dec 14 19:36:56 ip-10-10-1-254 ospfd[30868]: Hello sent to [224.0.0.5] via [tap5:10.245.5.1].
Dec 14 19:36:58 ip-10-10-1-254 ospfd[30868]: Hello received from [10.10.4.254] via [tap4:10.245.4.1]
So, a hello received from server 4 on tap4, but never any from server 2 on tap2 (or, for that matter, server 3 or server 5); meaning that the OSPF process does not receive the packets seen on the tcpdump earlier.
I would like to state once again, that sometimes, it works, and all 5 OSPF routers manage to establish an adjacency with the other fours. However, I cannot figure out why, or how...
Things I have tried:
1. verified I have no iptables rule to discard these packets: all tables are empty
2. set the flag kernel flag rp_filter=0 for all interfaces: does not change anything
3. tried to turn multicast packets into unicast packets with iptables DNAT on each tap: the adjacency goes a bit further (exstart), but is stuck there
4. updated to OpenVPN 2.4.8 and latest Quagga. I am running on Ubuntu 18.04 LTS bionic.
Does anybody have an idea what I could do to make this work relying on the OSPF self-discovery, or see anything I have done wrong in the setup?
Or, in general, can anybody suggest a direction where I should look into to solve the issue?
Many thanks for the long read and insights 🙂
