VPN up, routing looks good but packets vanishing

illwrk4u

Greetings all,

To extend on the subject, I am running OpenBSD 4.9( openvpn-2.1.4) on the
VPN server side and OpenBSD 5.0(openvpn-2.1.4) on the VPN client side. I
have successfully connected the VPN from the client to the server and can
ping both endpoints of the tunnel from either side. The problem occurs
when I try to ping through the tunnel from the server side to an
IP/network on the client side other than the vpn client itself.

My network looks (roughly) like this:

server side pvt net-----vpn server(vpn01)--- internet------->
---------------------------------------------------------------------------|
---------------------------------------------------------------------------|
client side pvt nets-----client vpn machine(apmobsd50)---<

VPN server 10.8.0.1---------VPN clent 10.8.0.6

From the Server side I can ping throught to the client just fine:

# hostname
vpn01
# ping 10.8.0.6
PING 10.8.0.6 (10.8.0.6): 56 data bytes
64 bytes from 10.8.0.6: icmp_seq=0 ttl=255 time=21.618 ms
64 bytes from 10.8.0.6: icmp_seq=1 ttl=255 time=15.959 ms
64 bytes from 10.8.0.6: icmp_seq=2 ttl=255 time=16.455 ms
64 bytes from 10.8.0.6: icmp_seq=3 ttl=255 time=16.183 ms
64 bytes from 10.8.0.6: icmp_seq=4 ttl=255 time=16.414 ms
--- 10.8.0.6 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 15.959/17.325/21.618/2.159 ms

# hostname
vpn01
# tcpdump -ttt -e -nvi tun1 icmp
tcpdump: listening on tun1, link-type LOOP
Feb 06 07:58:42.924204 10.8.0.1 > 10.8.0.6: icmp: echo request (id:fcef
seq:0) (ttl 255, id 45718, len 84)
Feb 06 07:58:42.945622 10.8.0.6 > 10.8.0.1: icmp: echo reply (id:fcef
seq:0) (ttl 255, id 35066, len 84)
Feb 06 07:58:43.925483 10.8.0.1 > 10.8.0.6: icmp: echo request (id:fcef
seq:1) (ttl 255, id 35693, len 84)
Feb 06 07:58:43.941367 10.8.0.6 > 10.8.0.1: icmp: echo reply (id:fcef
seq:1) (ttl 255, id 46280, len 84)
Feb 06 07:58:44.935548 10.8.0.1 > 10.8.0.6: icmp: echo request (id:fcef
seq:2) (ttl 255, id 27668, len 84)
Feb 06 07:58:44.951932 10.8.0.6 > 10.8.0.1: icmp: echo reply (id:fcef
seq:2) (ttl 255, id 31581, len 84)
Feb 06 07:58:45.945649 10.8.0.1 > 10.8.0.6: icmp: echo request (id:fcef
seq:3) (ttl 255, id 32461, len 84)
Feb 06 07:58:45.961761 10.8.0.6 > 10.8.0.1: icmp: echo reply (id:fcef
seq:3) (ttl 255, id 39723, len 84)
Feb 06 07:58:46.955749 10.8.0.1 > 10.8.0.6: icmp: echo request (id:fcef
seq:4) (ttl 255, id 11295, len 84)
Feb 06 07:58:46.972091 10.8.0.6 > 10.8.0.1: icmp: echo reply (id:fcef
seq:4) (ttl 255, id 4699, len 84)
^C
10 packets received by filter
0 packets dropped by kernel

# hostname
apmobsd50
# tcpdump -ttt -e -nvi tun0 icmp
tcpdump: listening on tun0, link-type LOOP
Feb 06 07:58:42.934197 10.8.0.1 > 10.8.0.6: icmp: echo request (id:fcef
seq:0) (ttl 255, id 45718, len 84)
tcpdump: WARNING: compensating for unaligned libpcap packets
Feb 06 07:58:42.934219 10.8.0.6 > 10.8.0.1: icmp: echo reply (id:fcef
seq:0) (ttl 255, id 35066, len 84)
Feb 06 07:58:43.930074 10.8.0.1 > 10.8.0.6: icmp: echo request (id:fcef
seq:1) (ttl 255, id 35693, len 84)
Feb 06 07:58:43.930086 10.8.0.6 > 10.8.0.1: icmp: echo reply (id:fcef
seq:1) (ttl 255, id 46280, len 84)
Feb 06 07:58:44.940225 10.8.0.1 > 10.8.0.6: icmp: echo request (id:fcef
seq:2) (ttl 255, id 27668, len 84)
Feb 06 07:58:44.940239 10.8.0.6 > 10.8.0.1: icmp: echo reply (id:fcef
seq:2) (ttl 255, id 31581, len 84)
Feb 06 07:58:45.950323 10.8.0.1 > 10.8.0.6: icmp: echo request (id:fcef
seq:3) (ttl 255, id 32461, len 84)
Feb 06 07:58:45.950336 10.8.0.6 > 10.8.0.1: icmp: echo reply (id:fcef
seq:3) (ttl 255, id 39723, len 84)
Feb 06 07:58:46.960871 10.8.0.1 > 10.8.0.6: icmp: echo request (id:fcef
seq:4) (ttl 255, id 11295, len 84)
Feb 06 07:58:46.960886 10.8.0.6 > 10.8.0.1: icmp: echo reply (id:fcef
seq:4) (ttl 255, id 4699, len 84)
^C
10 packets received by filter
0 packets dropped by kernel

When I try to ping from the server(vpn01) through the client(apmobsd50) to
a network/IP on the client side internal networks (using 10.5.29.92), the
packets look like they are going down the tunnel correctly but
they never show up as writing to the tunnel when I have logging turned up
and I never see them on the client side when listneing with tcpdump. I do
have routing information set up, done multiple ways, used both 10.8.0.1
and 10.8.0.2 as the gateway, on the VPN server side so the packets
will(should) know where to go.

# hostname
apmobsd50
# ping 10.5.29.92
PING 10.5.29.92 (10.5.29.92): 56 data bytes
64 bytes from 10.5.29.92: icmp_seq=0 ttl=59 time=8.958 ms
64 bytes from 10.5.29.92: icmp_seq=1 ttl=59 time=8.974 ms
--- 10.5.29.92 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 8.958/8.966/8.974/0.008 ms

# hostname
vpn01.entiret.com
# ping 10.5.29.92
PING 10.5.29.92 (10.5.29.92): 56 data bytes
--- 10.5.29.92 ping statistics ---
8 packets transmitted, 0 packets received, 100.0% packet loss

# hostname
vpn01
# tcpdump -ttt -e -nvi tun1 icmp
tcpdump: listening on tun1, link-type LOOP

Feb 06 08:07:33.556975 10.8.0.1 > 10.5.29.92: icmp: echo request (id:23f6
seq:0) (ttl 255, id 23031, len 84)
Feb 06 08:07:34.558122 10.8.0.1 > 10.5.29.92: icmp: echo request (id:23f6
seq:1) (ttl 255, id 33084, len 84)
Feb 06 08:07:35.568179 10.8.0.1 > 10.5.29.92: icmp: echo request (id:23f6
seq:2) (ttl 255, id 35684, len 84)
Feb 06 08:07:36.578276 10.8.0.1 > 10.5.29.92: icmp: echo request (id:23f6
seq:3) (ttl 255, id 57256, len 84)
Feb 06 08:07:37.588382 10.8.0.1 > 10.5.29.92: icmp: echo request (id:23f6
seq:4) (ttl 255, id 14535, len 84)
Feb 06 08:07:38.598477 10.8.0.1 > 10.5.29.92: icmp: echo request (id:23f6
seq:5) (ttl 255, id 64258, len 84)
Feb 06 08:07:39.608578 10.8.0.1 > 10.5.29.92: icmp: echo request (id:23f6
seq:6) (ttl 255, id 17072, len 84)
Feb 06 08:07:40.618674 10.8.0.1 > 10.5.29.92: icmp: echo request (id:23f6
seq:7) (ttl 255, id 15968, len 84)
^C
8 packets received by filter
0 packets dropped by kernel

# hostname
apmobsd50
# tcpdump -ttt -e -nvi tun0 icmp
tcpdump: listening on tun0, link-type LOOP

^C
0 packets received by filter
0 packets dropped by kernel

Firewalls don't appear to be the issue either:

# hostname
vpn01
PING 10.5.29.92 (10.5.29.92): 56 data bytes
--- 10.5.29.92 ping statistics ---
35 packets transmitted, 0 packets received, 100.0% packet loss

# hostname
vpn01
# tcpdump -ttt -e -nvi pflog0 icmp
tcpdump: WARNING: snaplen raised from 116 to 160
tcpdump: listening on pflog0, link-type PFLOG

^C
0 packets received by filter
0 packets dropped by kernel

# hostname
apmobsd50
# tcpdump -ttt -e -nvi pflog0 icmp
tcpdump: WARNING: snaplen raised from 116 to 160
tcpdump: listening on pflog0, link-type PFLOG
^C
271 packets received by filter
0 packets dropped by kernel

I can't figure out why openvpn doesn't see the packets traversing the
tunnel and using bandwidth. When I set logging to "verb 5", and am
able to ping from the server side to the client side, I see the
wrwrwrwrwrw activity in the logs, however when trying to go through the
vpn to 10.5.29.92, it doesn't log anything.

Here are my configs and network information.

(VPN SERVER)
# hostname
vpn01
# ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33200
priority: 0
groups: lo
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff000000
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:02:b3:97:13:3c
priority: 0
groups: egress
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet6 fe80::202:b3ff:fe97:133c%fxp0 prefixlen 64 scopeid 0x1
inet 192.168.27.9 netmask 0xffffff00 broadcast 192.168.27.255
enc0: flags=0<>
priority: 0
groups: enc
status: active
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33200
priority: 0
groups: pflog
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
priority: 0
groups: tun
status: active
inet 10.7.0.1 --> 10.7.0.2 netmask 0xffffffff
tun2: flags=9843<UP,BROADCAST,RUNNING,SIMPLEX,LINK0,MULTICAST> mtu 1500
lladdr fe:e1:ba:dd:29:91
priority: 0
groups: tun
status: active
inet 10.12.0.1 netmask 0xffffff00 broadcast 10.12.0.255
inet6 fe80::fce1:baff:fedd:2991%tun2 prefixlen 64 scopeid 0xb
tun3: flags=9843<UP,BROADCAST,RUNNING,SIMPLEX,LINK0,MULTICAST> mtu 1500
lladdr fe:e1:ba:df:6a:de
priority: 0
groups: tun
status: active
inet 10.10.0.1 netmask 0xffffff00 broadcast 10.10.0.255
inet6 fe80::fce1:baff:fedf:6ade%tun3 prefixlen 64 scopeid 0xc
tun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
priority: 0
groups: tun
status: active
inet 10.8.0.1 --> 10.8.0.2 netmask 0xffffffff
# netstat -rnf inet
Routing tables

Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
default 192.168.27.254 UGS 7 21714 - 8 fxp0
10.5.29/24 10.8.0.2 UGS 0 189 - 8 tun1
10.7.0/24 10.7.0.2 UGS 2 4116 - 8 tun0
10.7.0.2 10.7.0.1 UH 1 0 - 4 tun0
10.8.0/24 10.8.0.2 UGS 0 30 - 8 tun1
10.8.0.2 10.8.0.1 UH 2 0 - 4 tun1
10.10.0/24 link#12 UC 0 0 - 4 tun3
10.12.0/24 link#11 UC 0 0 - 4 tun2
127/8 127.0.0.1 UGRS 0 0 33200 8 lo0
127.0.0.1 127.0.0.1 UH 0 70 33200 4 lo0
192.168.27/24 link#1 UC 1 0 - 4 fxp0
192.168.27.254 00:03:47:0a:45:0c UHLc 1 11 - 4 fxp0
224/4 127.0.0.1 URS 0 0 33200 8 lo0

# cat vpn.conf
mode server
local 192.168.27.9
port 1195
#proto tcp
proto udp
dev tun1
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/vpn01.crt
key /etc/openvpn/wkeys/vpn01.key
dh /etc/openvpn/keys/dh2048.pem
tls-auth /etc/openvpn/keys/ta.key 0
server 10.8.0.0 255.255.255.0
route 10.5.29.0 255.255.255.0 10.8.0.2
ifconfig-pool-persist /tmp/ipp.txt
keepalive 10 120
comp-lzo
user _openvpn
group _openvpn
persist-key
persist-tun
chroot /var/empty
status /var/log/vpn-status.log
log-append /var/log/vpn.log
verb 2

#

(VPN CLIENT)
# hostname
apmobsd50

# ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33152
priority: 0
groups: lo
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet 127.0.0.1 netmask 0xff000000
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:25:64:ab:6c:0b
priority: 0
groups: egress
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet6 fe80::225:64ff:feab:6c0b%em0 prefixlen 64 scopeid 0x1
inet 10.6.116.161 netmask 0xfffffe00 broadcast 10.6.117.255
xl0: flags=8a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:60:97:9f:af:80
priority: 0
media: Ethernet autoselect (100baseTX full-duplex)
status: active
inet 192.168.13.254 netmask 0xffffff00 broadcast 192.168.13.255
inet6 fe80::260:97ff:fe9f:af80%xl0 prefixlen 64 scopeid 0x2
enc0: flags=0<>
priority: 0
groups: enc
status: active
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33152
priority: 0
groups: pflog
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
priority: 0
groups: tun
status: active
inet 10.8.0.6 --> 10.8.0.5 netmask 0xffffffff
tun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
priority: 0
groups: tun
status: active
inet 10.7.0.6 --> 10.7.0.5 netmask 0xffffffff

# netstat -rnf inet
Routing tables

Internet:
Destination Gateway Flags Refs Use Mtu Prio Iface
default 10.6.116.1 UGS 16 825487 - 8 em0
10.6.116/23 link#1 UC 2 0 - 4 em0
10.6.116.1 00:26:0b:de:3e:52 UHLc 2 1 - 4 em0
10.6.116.2 00:1f:9e:9b:d3:54 UHLc 0 0 - 4 em0
10.6.116.161 127.0.0.1 UGHS 0 0 33152 8 lo0
10.7.0.1/32 10.7.0.5 UGS 3 18699 - 8 tun1
10.7.0.5 10.7.0.6 UH 1 0 - 4 tun1
10.8.0.1/32 10.8.0.5 UGS 0 13916 - 8 tun0
10.8.0.5 10.8.0.6 UH 1 0 - 4 tun0
127/8 127.0.0.1 UGRS 0 0 33152 8 lo0
127.0.0.1 127.0.0.1 UH 2 64502 33152 4 lo0
192.168.13/24 link#2 UC 1 0 - 4 xl0
192.168.13.33 00:15:60:c4:42:9a UHLc 3 5448937 - L 4 xl0
224/4 127.0.0.1 URS 0 2 33152 8 lo0

# cat /etc/openvpn/client.conf
client
nobind
dev tun0
#proto tcp
proto udp
remote 24.26.229.61 1195
#remote 192.168.17.9 1195
persist-key
persist-tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/apmobsd50.crt
key /etc/openvpn/apmobsd50.key
tls-auth /etc/openvpn/ta.key 1
tls-client
remote-cert-tls server
comp-lzo
chroot /var/empty
status /var/log/vpn-status.log
log-append /var/log/vpn.log
verb 5

user _openvpn
group _openvpn
#

Any help greatly appreciated!!!

TIA,

Aaron

PS.
I have been subscribed to the openvpn user list mailing list since 2002. is it no more? I noticed I haven't gotten anything since about July.. Thanks again.

janjust

I have been subscribed to the openvpn user list mailing list since 2002. is it no more? I noticed I haven't gotten anything since about July.. Thanks again.

I've replied via the mailing list; the list has been quiet but there has definitely been traffic over the past months.

illwrk4u

I just wanted to post this here in case anyone else runs into this as well.

in order to reach a LAN subnet behind a client you will need to set up a
client-config file (CCD file) using the appropriate 'iroute' statement.
Read up at
http://openvpn.net/index.php/open-source/documentation/howto.html#scope
for more details.

Basically, the openvpn process needs to be told that the network
10.5.29.0/24 can be found via client 'apmobsd50'

HTH,

JJK

It was the addition of the ccd directory and the iroute statements in the common name file in that directory that was the key.

Thanks again Jan!

(here's the pudding)

# hostname
vpn01
# ping -c 5 10.5.29.92
PING 10.5.29.92 (10.5.29.92): 56 data bytes
64 bytes from 10.5.29.92: icmp_seq=0 ttl=58 time=36.175 ms
64 bytes from 10.5.29.92: icmp_seq=1 ttl=58 time=38.887 ms
64 bytes from 10.5.29.92: icmp_seq=2 ttl=58 time=35.332 ms
64 bytes from 10.5.29.92: icmp_seq=3 ttl=58 time=36.884 ms
64 bytes from 10.5.29.92: icmp_seq=4 ttl=58 time=38.760 ms
--- 10.5.29.92 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 35.332/37.207/38.887/1.424 ms

# hostname
vpn01
# tcpdump -ttt -e -nvi tun1 icmp
tcpdump: listening on tun1, link-type LOOP
Feb 07 11:03:58.313631 10.8.0.1 > 10.5.29.92: icmp: echo request (id:62c4 seq:0) (ttl 255, id 24112, len 84)
Feb 07 11:03:58.349649 10.5.29.92 > 10.8.0.1: icmp: echo reply (id:62c4 seq:0) (ttl 58, id 5962, len 84)
Feb 07 11:03:59.319786 10.8.0.1 > 10.5.29.92: icmp: echo request (id:62c4 seq:1) (ttl 255, id 23060, len 84)
Feb 07 11:03:59.358604 10.5.29.92 > 10.8.0.1: icmp: echo reply (id:62c4 seq:1) (ttl 58, id 5963, len 84)
Feb 07 11:04:00.329817 10.8.0.1 > 10.5.29.92: icmp: echo request (id:62c4 seq:2) (ttl 255, id 29059, len 84)
Feb 07 11:04:00.365084 10.5.29.92 > 10.8.0.1: icmp: echo reply (id:62c4 seq:2) (ttl 58, id 5964, len 84)
Feb 07 11:04:01.339922 10.8.0.1 > 10.5.29.92: icmp: echo request (id:62c4 seq:3) (ttl 255, id 26516, len 84)
Feb 07 11:04:01.376736 10.5.29.92 > 10.8.0.1: icmp: echo reply (id:62c4 seq:3) (ttl 58, id 5965, len 84)
Feb 07 11:04:02.350018 10.8.0.1 > 10.5.29.92: icmp: echo request (id:62c4 seq:4) (ttl 255, id 41159, len 84)
Feb 07 11:04:02.388712 10.5.29.92 > 10.8.0.1: icmp: echo reply (id:62c4 seq:4) (ttl 58, id 5966, len 84)

# hostname
apmobsd50
# tcpdump -ttt -e -nvi tun0 icmp
tcpdump: listening on tun0, link-type LOOP

Feb 07 11:03:58.329103 10.8.0.1 > 10.5.29.92: icmp: echo request (id:62c4 seq:0) (ttl 255, id 24112, len 84)
tcpdump: WARNING: compensating for unaligned libpcap packets
Feb 07 11:03:58.339375 10.5.29.92 > 10.8.0.1: icmp: echo reply (id:62c4 seq:0) (ttl 58, id 5962, len 84)
Feb 07 11:03:59.339379 10.8.0.1 > 10.5.29.92: icmp: echo request (id:62c4 seq:1) (ttl 255, id 23060, len 84)
Feb 07 11:03:59.348179 10.5.29.92 > 10.8.0.1: icmp: echo reply (id:62c4 seq:1) (ttl 58, id 5963, len 84)
Feb 07 11:04:00.343590 10.8.0.1 > 10.5.29.92: icmp: echo request (id:62c4 seq:2) (ttl 255, id 29059, len 84)
Feb 07 11:04:00.354609 10.5.29.92 > 10.8.0.1: icmp: echo reply (id:62c4 seq:2) (ttl 58, id 5964, len 84)
Feb 07 11:04:01.355560 10.8.0.1 > 10.5.29.92: icmp: echo request (id:62c4 seq:3) (ttl 255, id 26516, len 84)
Feb 07 11:04:01.366321 10.5.29.92 > 10.8.0.1: icmp: echo reply (id:62c4 seq:3) (ttl 58, id 5965, len 84)
Feb 07 11:04:02.366360 10.8.0.1 > 10.5.29.92: icmp: echo request (id:62c4 seq:4) (ttl 255, id 41159, len 84)
Feb 07 11:04:02.378161 10.5.29.92 > 10.8.0.1: icmp: echo reply (id:62c4 seq:4) (ttl 58, id 5966, len 84)