weird routing issue

[offthedeepnd]

Hi All,

I've been using openvpn for years now and until now not had an issue.

I've got a standard setup, single server set up to accept multiple clients using tun interfaces. I have it set up so that a client connects from remote location and then I can connect to the client and networks behind the client from any systems behind the server. The openvpn server and client are running openvpn 2.2.2 on openbsd 5.3.

I see on my server packets from my home lan destined for the networks behind the clients going down the tun interface but on the client side, i don't see any packets coming down the tun interface.

From the openvpn server at home and any client behind it I can connect to the openvpn client but nothing beyond it.

My configs follow:

openvpn server (at home):
local 172.27.27.9
proto udp
port 1195
dev tun1
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/vpn01.entiret.com.crt
key /etc/openvpn/keys/vpn01.entiret.com.key
dh /etc/openvpn/keys/dh2048.pem
tls-auth /etc/openvpn/keys/ta.key 0
user _openvpn
group _openvpn
daemon openvpn
comp-lzo
chroot /var/empty
persist-key
persist-tun
keepalive 10 120
ifconfig-pool-persist /tmp/ipp.txt
status /var/log/openvpn-status.log
log /var/log/openvpn.log
log-append /var/log/openvpn.log
verb 5
server 192.168.7.0 255.255.255.0
route 192.168.250.0 255.255.255.252
route 10.0.0.0 255.0.0.0


openvpn client (at work):
client
remote vpn01.company.com
proto udp
port 1195
dev tun0
ca /etc/openvpn/keys/ca.crt
key /etc/openvpn/keys/me.key
cert /etc/openvpn/keys/me.crt
tls-auth /etc/openvpn/keys/ta.key 1
ns-cert-type server
user _openvpn
group _openvpn
daemon openvpn
comp-lzo
chroot /var/empty
persist-key
persist-tun
keepalive 10 120
status /var/log/openvpn-status.log
log /var/log/vpn.log
log-append /var/log/vpn.log
replay-persist replay.txt
verb 5


here are the routing tables:

openvpn server(home):
# netstat -rnf inet
Routing tables

Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
default 172.27.27.254 UGS 7 2353 - 8 bge0
10/8 192.168.7.2 UGS 0 2216 - 8 tun1
127/8 127.0.0.1 UGRS 0 0 33196 8 lo0
127.0.0.1 127.0.0.1 UH 1 0 33196 4 lo0
172.27.27/24 link#1 UC 1 0 - 4 bge0
172.27.27.254 00:d0:b7:8f:4e:54 UHLc 1 5 - 4 bge0
192.168.7/24 192.168.7.2 UGS 0 456 - 8 tun1
192.168.7.2 192.168.7.1 UH 4 0 - 4 tun1
192.168.250.0/30 192.168.7.2 UGS 0 0 - 8 tun1
224/4 127.0.0.1 URS 0 0 33196 8 lo0

Openvpn client(work):
# netstat -rnf inet
Routing tables

Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
default 10.0.4.1 UGS 5 1252 - 8 sis0
10.0.4/24 link#1 UC 1 0 - 4 sis0
10.0.4.1 00:22:be:9f:17:c3 UHLc 1 0 - 4 sis0
127/8 127.0.0.1 UGRS 0 0 33196 8 lo0
127.0.0.1 127.0.0.1 UH 1 0 33196 4 lo0
192.168.7.1/32 192.168.7.5 UGS 1 528 - 8 tun0
192.168.7.5 192.168.7.6 UH 1 0 - 4 tun0
224/4 127.0.0.1 URS 0 0 33196 8 lo0

The reason I think this is a routing issue is this, from the openvpn server and any system behind it, I can ping/ssh to 192.168.7.6, the openvpn client, but I can't get to anything beyond the client. When I tcpdump on the openvpn server and client when i'm pinging/sshing to 10.0.4.142 I see the packets going down the tun1 interface on the server but I don't see them on the client. Additonally, when tailing the server vpn log, i don't see any reads or writes. I'm lost as to where the traffic is going. Here are the logs:

from the openvpn server:

# ping 192.168.7.6
PING 192.168.7.6 (192.168.7.6): 56 data bytes
64 bytes from 192.168.7.6: icmp_seq=5 ttl=255 time=16.819 ms
64 bytes from 192.168.7.6: icmp_seq=6 ttl=255 time=14.630 ms
64 bytes from 192.168.7.6: icmp_seq=7 ttl=255 time=14.934 ms
64 bytes from 192.168.7.6: icmp_seq=8 ttl=255 time=15.088 ms
64 bytes from 192.168.7.6: icmp_seq=9 ttl=255 time=15.092 ms
64 bytes from 192.168.7.6: icmp_seq=10 ttl=255 time=15.413 ms
64 bytes from 192.168.7.6: icmp_seq=11 ttl=255 time=17.897 ms
--- 192.168.7.6 ping statistics ---
12 packets transmitted, 7 packets received, 41.7% packet loss
round-trip min/avg/max/std-dev = 14.630/15.696/17.897/1.112 ms

(notice the first 4 pings failed, however, the tail of the vpn log immediately started seeing the writes)

# tail -f /var/log/workvpn-openvpn.log

rWrWrWrWrWRrWRwrWRwrWRwrWRwrWRwrWRwrW

during this ping i see this doing a tcpdump:

openvpn server:
# tcpdump -ttt -e -nvi tun1 icmp
tcpdump: listening on tun1, link-type LOOP


Jul 21 23:30:38.335354 192.168.7.1 > 192.168.7.6: icmp: echo request (id:7005 seq:0) (ttl 255, id 28599, len 84)
Jul 21 23:30:39.344035 192.168.7.1 > 192.168.7.6: icmp: echo request (id:7005 seq:1) (ttl 255, id 56894, len 84)
Jul 21 23:30:40.354014 192.168.7.1 > 192.168.7.6: icmp: echo request (id:7005 seq:2) (ttl 255, id 47753, len 84)
Jul 21 23:30:40.368512 192.168.7.6 > 192.168.7.1: icmp: echo reply (id:7005 seq:2) (ttl 255, id 10281, len 84)
Jul 21 23:30:41.364013 192.168.7.1 > 192.168.7.6: icmp: echo request (id:7005 seq:3) (ttl 255, id 65349, len 84)
Jul 21 23:30:41.378832 192.168.7.6 > 192.168.7.1: icmp: echo reply (id:7005 seq:3) (ttl 255, id 21466, len 84)
Jul 21 23:30:42.374009 192.168.7.1 > 192.168.7.6: icmp: echo request (id:7005 seq:4) (ttl 255, id 32881, len 84)
Jul 21 23:30:42.391475 192.168.7.6 > 192.168.7.1: icmp: echo reply (id:7005 seq:4) (ttl 255, id 58687, len 84)
Jul 21 23:30:43.384009 192.168.7.1 > 192.168.7.6: icmp: echo request (id:7005 seq:5) (ttl 255, id 48671, len 84)
Jul 21 23:30:43.399288 192.168.7.6 > 192.168.7.1: icmp: echo reply (id:7005 seq:5) (ttl 255, id 22418, len 84)
Jul 21 23:30:44.394006 192.168.7.1 > 192.168.7.6: icmp: echo request (id:7005 seq:6) (ttl 255, id 8776, len 84)
Jul 21 23:30:44.411158 192.168.7.6 > 192.168.7.1: icmp: echo reply (id:7005 seq:6) (ttl 255, id 9464, len 84)


Openvpn Client:
# tcpdump -ttt -e -nvi tun0 icmp
tcpdump: listening on tun0, link-type LOOP
Jul 21 23:30:45.486883 192.168.7.1 > 192.168.7.6: icmp: echo request (id:7005 seq:2) (ttl 255, id 47753, len 84)
Jul 21 23:30:45.487031 192.168.7.6 > 192.168.7.1: icmp: echo reply (id:7005 seq:2) (ttl 255, id 10281, len 84)
Jul 21 23:30:46.496397 192.168.7.1 > 192.168.7.6: icmp: echo request (id:7005 seq:3) (ttl 255, id 65349, len 84)
Jul 21 23:30:46.496480 192.168.7.6 > 192.168.7.1: icmp: echo reply (id:7005 seq:3) (ttl 255, id 21466, len 84)
Jul 21 23:30:47.508653 192.168.7.1 > 192.168.7.6: icmp: echo request (id:7005 seq:4) (ttl 255, id 32881, len 84)
Jul 21 23:30:47.508717 192.168.7.6 > 192.168.7.1: icmp: echo reply (id:7005 seq:4) (ttl 255, id 58687, len 84)
Jul 21 23:30:48.517422 192.168.7.1 > 192.168.7.6: icmp: echo request (id:7005 seq:5) (ttl 255, id 48671, len 84)
Jul 21 23:30:48.517496 192.168.7.6 > 192.168.7.1: icmp: echo reply (id:7005 seq:5) (ttl 255, id 22418, len 84)
Jul 21 23:30:49.526864 192.168.7.1 > 192.168.7.6: icmp: echo request (id:7005 seq:6) (ttl 255, id 8776, len 84)
Jul 21 23:30:49.526928 192.168.7.6 > 192.168.7.1: icmp: echo reply (id:7005 seq:6) (ttl 255, id 9464, len 84)

When I try to ping or ssh to anything behind the openvpn client I never see packets on the client side of the vpn,

Openvpn Server during a ping:
# hostname
vpn01.company.com
# ping 10.0.4.142
PING 10.0.4.142 (10.0.4.142): 56 data bytes
--- 10.0.4.142 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss

# tcpdump -ttt -e -nvi tun1 icmp
tcpdump: listening on tun1, link-type LOOP
Jul 21 23:33:12.850554 192.168.7.1 > 10.0.4.142: icmp: echo request (id:d119 seq:0) (ttl 255, id 31726, len 84)
Jul 21 23:33:13.853813 192.168.7.1 > 10.0.4.142: icmp: echo request (id:d119 seq:1) (ttl 255, id 59524, len 84)
Jul 21 23:33:14.863798 192.168.7.1 > 10.0.4.142: icmp: echo request (id:d119 seq:2) (ttl 255, id 64683, len 84)
Jul 21 23:33:15.873794 192.168.7.1 > 10.0.4.142: icmp: echo request (id:d119 seq:3) (ttl 255, id 55113, len 84)
Jul 21 23:33:16.883792 192.168.7.1 > 10.0.4.142: icmp: echo request (id:d119 seq:4) (ttl 255, id 3809, len 84)


Openvpn client during same ping:
# hostname
vpn.company1.com
# tcpdump -ttt -e -nvi tun0 icmp
tcpdump: listening on tun0, link-type LOOP


Like i say, when i tail the vpn log on the server, there are no rwrwrw messages like when I ping direct to the client side of the vpn. The openvpn server looks to be sending the packets down the tunnel but they aren't getting written to the tunnel and the client side never sees them.

I don't think it's a firewall issue as here are my rules:

Openvpn server side:
# pfctl -sr
match out on tun1 inet from ! 192.168.7.1 to any nat-to 192.168.7.1
block drop log all
pass in on bge0 inet from <wired_win_hosts> to 172.20.203.0/24 flags S/SA keep state (if-bound)
pass in on bge0 inet from <wired_win_hosts> to 192.168.250.0/30 flags S/SA keep state (if-bound)
pass in on bge0 inet from <wired_win_hosts> to 10.0.0.0/8 flags S/SA keep state (if-bound)
pass in on bge0 inet from <wired_win_hosts> to 192.168.7.0/24 flags S/SA keep state (if-bound)
pass in on bge0 inet from <wlan_hosts> to 172.20.203.0/24 flags S/SA keep state (if-bound)
pass in on bge0 inet from <wlan_hosts> to 192.168.250.0/30 flags S/SA keep state (if-bound)
pass in on bge0 inet from <wlan_hosts> to 10.0.0.0/8 flags S/SA keep state (if-bound)
pass in on bge0 inet from <wlan_hosts> to 192.168.7.0/24 flags S/SA keep state (if-bound)
pass in on bge0 inet from <priv_hosts> to 172.20.203.0/24 flags S/SA keep state (if-bound)
pass in on bge0 inet from <priv_hosts> to 192.168.250.0/30 flags S/SA keep state (if-bound)
pass in on bge0 inet from <priv_hosts> to 10.0.0.0/8 flags S/SA keep state (if-bound)
pass in on bge0 inet from <priv_hosts> to 192.168.7.0/24 flags S/SA keep state (if-bound)
pass in on bge0 inet proto tcp from 172.27.7.0/24 to 172.27.27.9 port = 22 flags S/SA keep state (if-bound)
pass in on bge0 inet proto udp from <client_ip_address> to 172.27.27.9 port = 1195 keep state (if-bound)
pass out on bge0 all flags S/SA keep state (if-bound)
pass out on tun1 all flags S/SA keep state (if-bound)

Openvpn client side:
# pfctl -sr
match out on egress inet from ! 10.0.4.0/24 to any nat-to 10.0.4.7
block drop log all
pass in on sis0 inet proto tcp from 172.27.7.0/24 to 10.0.4.7 port = 22 flags S/SA
pass in on sis0 inet proto tcp from 10.0.4.215 to 10.0.4.7 port = 22 flags S/SA
pass in inet proto icmp all icmp-type echoreq
pass in on tun0 all flags S/SA
pass out on sis0 all flags S/SA

as you can see, I'm passing all traffic out on tun 1 from the server which connects to the tun0 interface on the client which I am also allowing all traffic in on. Also, when tailing the firewall logs I don't see anything getting blocked.

YES, packet forwarding is turned on for both machines!

What am i missing, it's driving me crazy.

Please let me know if there is anything further you need to see.

Thanks in advance,

Aaron Martinez

[maikcat]

if you want lan-to-lan connectivity you must:

1)enable ip forwarding on both openvpn server/client
2)create a ccd file named after clients common name cert field with contents:

Code: Select all

iroute remotelan 255.255.255.0

also you need to add to your server config

Code: Select all

route remotelan 255.255.255.0

Michael.