routing problem - TUN -> SRV -> default GW instead of LAN

mb76pl

Hi all,

I've been struggling with this for days.
My OpenVPN config is straight forward. Just accessing home's network, no fancy features.
But the problem is, that I can ping server's IP, but none of server's LAN machines (of course I can ping them from server).
Traffic from connected client goes to server, then is routed to server's default gateway instead of machine in LAN.
What am I missing ?

server: Ubuntu 14.04 desktop
OpenVPN version: OpenVPN 2.3.2 x86_64-pc-linux-gnu
AppArmor - disabled
192.168.10.0/24 route on 192.168.1.1 configred

server's netstat:

Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.10.0 192.168.10.2 255.255.255.0 UG 0 0 0 tun0
192.168.10.2 0.0.0.0 255.255.255.255 UH 0 0 0 tun0

tracert from windows7 client:

C:\>tracert -d 192.168.1.15
Tracing route to 192.168.1.15 over a maximum of 30 hops
1 4 ms 5 ms 5 ms 192.168.10.1
2 * * * Request timed out.
3 * ^C
C:\>

tcpdump on the server

root@test:/etc/openvpn# tcpdump -i tun0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 65535 bytes
07:33:41.159742 IP 192.168.10.6 > 192.168.1.15: ICMP echo request, id 1, seq 161, length 40
07:33:41.160895 IP 192.168.1.1 > 192.168.10.6: ICMP echo reply, id 19765, seq 161, length 40
07:33:45.850931 IP 192.168.10.6 > 192.168.1.15: ICMP echo request, id 1, seq 162, length 40
07:33:45.852214 IP 192.168.1.1 > 192.168.10.6: ICMP echo reply, id 13682, seq 162, length 40
^C

my server's config:

local 192.168.1.111
username-as-common-name
port 10443
proto tcp
dev tun
ca ca.crt
cert MB-AccessServer.crt
key MB-AccessServer.key # This file should be kept secret
dh dh2048.pem
server 192.168.10.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "route 192.168.1.0 255.255.255.0"
push "dhcp-option DOMAIN marcinbieniek.lan"
client-to-client
keepalive 10 120
tls-auth ta.key 0 # This file is secret
comp-lzo
max-clients 5
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
management 127.0.0.1 5094
log-append /var/log/openvpn.log
verb 3
mute 20
reneg-sec 32400
plugin /usr/lib/openvpn/openvpn-plugin-auth-pam.so openvpn

client's config (viscosity):

remote ******.com 10443 tcp-client
tls-client
pull
dev tun
nobind
persist-key
persist-tun
auth-user-pass
comp-lzo
ns-cert-type server
ca ca.crt
cert cert.crt
key key.key
tls-auth ta.key 1
resolv-retry infinite
route-delay 5
keepalive 10 120
reneg-sec 0
dev-node {3205EF1E-D3A2-44DF-8ABB-217D01CFF342}

mb76pl

cat /proc/sys/net/ipv4/ip_forward returns "1"

as I wrote earlier, default gateway route 192.168.10.0/24 class

VPN client network is: 192.168.10.0/24
Server's LAN network: 192.168.1.0/24

server's default gateway 192.168.1.1

Ok, so let me try to describe the problem another way:
Current situation as you can see in tcpdump is that traffic tries, but fails to go like this:
192.168.10.6 -> [192.168.10.2] -> 192.168.1.1 -> 192.168.1.40

shouldn't it go like this?:
192.168.10.6 -> [192.168.10.2 -> 192.168.1.111] -> 192.168.1.40

[] - means OpenVPN box

mb76pl

viscosity log:

mar 20 09:22:13: State changed to Connecting
mar 20 09:22:13: Viscosity 1.4.7 (1222)
mar 20 09:22:13: Running on Microsoft Windows 7 Professional 
mar 20 09:22:13: Bringing up interface...
mar 20 09:22:13: Checking reachability status of connection...
mar 20 09:22:13: Connection is reachable. Starting connection attempt.
mar 20 09:22:13: OpenVPN 2.3.2 Windows-MSVC [SSL (OpenSSL)] [LZO] [PKCS11] [eurephia] [IPv6] built on Dec 13 2013
mar 20 09:22:15: Control Channel Authentication: using 'ta.key' as a OpenVPN static key file
mar 20 09:22:15: Attempting to establish TCP connection with [AF_INET]XXX.XXX.XXX.XXX:10443 [nonblock]
mar 20 09:22:16: TCP connection established with [AF_INET]XXX.XXX.XXX.XXX:10443
mar 20 09:22:16: TCPv4_CLIENT link local: [undef]
mar 20 09:22:16: TCPv4_CLIENT link remote: [AF_INET]XXX.XXX.XXX.XXX:10443
mar 20 09:22:16: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
mar 20 09:22:22: [MB-AccessServer] Peer Connection Initiated with [AF_INET]XXX.XXX.XXX.XXX:10443
mar 20 09:22:25: do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
mar 20 09:22:25: open_tun, tt->ipv6=0
mar 20 09:22:25: TAP-WIN32 device [wichura.ovpn] opened: \\.\Global\{3205EF1E-D3A2-44DF-8ABB-217D01CFF342}.tap
mar 20 09:22:25: Notified TAP-Windows driver to set a DHCP IP/netmask of 192.168.10.6/255.255.255.252 on interface {3205EF1E-D3A2-44DF-8ABB-217D01CFF342} [DHCP-serv: 192.168.10.5, lease-time: 31536000]
mar 20 09:22:25: Successful ARP Flush on interface [29] {3205EF1E-D3A2-44DF-8ABB-217D01CFF342}
mar 20 09:22:30: Initialization Sequence Completed
mar 20 09:22:30: State changed to Connected

OpenVPN log:

Thu Mar 20 09:22:48 2014 us=963739 MULTI: multi_create_instance called
Thu Mar 20 09:22:48 2014 us=963881 Re-using SSL/TLS context
Thu Mar 20 09:22:48 2014 us=963960 LZO compression initialized
Thu Mar 20 09:22:48 2014 us=964317 Control Channel MTU parms [ L:1544 D:168 EF:68 EB:0 ET:0 EL:0 ]
Thu Mar 20 09:22:48 2014 us=964379 Data Channel MTU parms [ L:1544 D:1450 EF:44 EB:135 ET:0 EL:0 AF:3/1 ]
Thu Mar 20 09:22:48 2014 us=964461 Local Options String: 'V4,dev-type tun,link-mtu 1544,tun-mtu 1500,proto TCPv4_SERVER,comp-lzo,keydir 0,cipher BF-CBC,auth SHA1,keysize 128,tls-auth,key-method 2,t
ls-server'
Thu Mar 20 09:22:48 2014 us=964490 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1544,tun-mtu 1500,proto TCPv4_CLIENT,comp-lzo,keydir 1,cipher BF-CBC,auth SHA1,keysize 128,tls-auth,key-
method 2,tls-client'
Thu Mar 20 09:22:48 2014 us=964543 Local Options hash (VER=V4): 'bd577cd1'
Thu Mar 20 09:22:48 2014 us=964582 Expected Remote Options hash (VER=V4): 'ee93268d'
Thu Mar 20 09:22:48 2014 us=964630 TCP connection established with [AF_INET]31.61.138.19:5247
Thu Mar 20 09:22:48 2014 us=964661 TCPv4_SERVER link local: [undef]
Thu Mar 20 09:22:48 2014 us=964691 TCPv4_SERVER link remote: [AF_INET]31.61.138.19:5247
Thu Mar 20 09:22:49 2014 us=879601 31.61.138.19:5247 TLS: Initial packet from [AF_INET]31.61.138.19:5247, sid=695fe1d8 b6b831bd
Thu Mar 20 09:22:54 2014 us=553104 31.61.138.19:5247 VERIFY OK: depth=1, C=PL, ST=dolnoslaskie, L=Smolec, O=MarcinBieniek.com, OU=MB-AccessServer, CN=wichura.marcinbieniek.com, name=MB-AccessServer
, emailAddress=me@marcinbieniek.com
Thu Mar 20 09:22:54 2014 us=553535 31.61.138.19:5247 VERIFY OK: depth=0, C=PL, ST=dolnoslaskie, L=Smolec, O=MarcinBieniek.com, OU=MB-AccessServer, CN=marcinb, name=Marcin Bieniek, emailAddress=me@m
arcinbieniek.com
AUTH-PAM: BACKGROUND: received command code: 0
AUTH-PAM: BACKGROUND: USER: marcinb
AUTH-PAM: BACKGROUND: my_conv[0] query='Password: ' style=1
Thu Mar 20 09:22:56 2014 us=409144 31.61.138.19:5247 PLUGIN_CALL: POST /usr/lib/openvpn/openvpn-plugin-auth-pam.so/PLUGIN_AUTH_USER_PASS_VERIFY status=0
Thu Mar 20 09:22:56 2014 us=409223 31.61.138.19:5247 TLS: Username/Password authentication succeeded for username 'marcinb' [CN SET]
Thu Mar 20 09:22:56 2014 us=409412 31.61.138.19:5247 Data Channel Encrypt: Cipher 'BF-CBC' initialized with 128 bit key
Thu Mar 20 09:22:56 2014 us=409429 31.61.138.19:5247 Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Mar 20 09:22:56 2014 us=409497 31.61.138.19:5247 Data Channel Decrypt: Cipher 'BF-CBC' initialized with 128 bit key
Thu Mar 20 09:22:56 2014 us=409512 31.61.138.19:5247 Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Mar 20 09:22:56 2014 us=880737 31.61.138.19:5247 Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 2048 bit RSA
Thu Mar 20 09:22:56 2014 us=880834 31.61.138.19:5247 [marcinb] Peer Connection Initiated with [AF_INET]31.61.138.19:5247
Thu Mar 20 09:22:56 2014 us=880897 marcinb/31.61.138.19:5247 MULTI_sva: pool returned IPv4=192.168.10.6, IPv6=(Not enabled)
Thu Mar 20 09:22:56 2014 us=880979 marcinb/31.61.138.19:5247 MULTI: Learn: 192.168.10.6 -> marcinb/31.61.138.19:5247
Thu Mar 20 09:22:56 2014 us=881012 marcinb/31.61.138.19:5247 MULTI: primary virtual IP for marcinb/31.61.138.19:5247: 192.168.10.6
Thu Mar 20 09:22:59 2014 us=8578 marcinb/31.61.138.19:5247 PUSH: Received control message: 'PUSH_REQUEST'
Thu Mar 20 09:22:59 2014 us=8658 marcinb/31.61.138.19:5247 send_push_reply(): safe_cap=940
Thu Mar 20 09:22:59 2014 us=8739 marcinb/31.61.138.19:5247 SENT CONTROL [marcinb]: 'PUSH_REPLY,route 192.168.1.0 255.255.255.0,dhcp-option DOMAIN marcinbieniek.lan,route 192.168.10.0 255.255.255.0,
topology net30,ping 10,ping-restart 120,ifconfig 192.168.10.6 192.168.10.5' (status=1)
Thu Mar 20 09:22:59 2014 us=510385 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:22:59 2014 us=634414 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:22:59 2014 us=634504 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:22:59 2014 us=634553 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:22:59 2014 us=634598 marcinb/31.61.138.19:5247 MULTI: bad source address from client [::], packet dropped
Thu Mar 20 09:22:59 2014 us=634643 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:22:59 2014 us=634703 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:22:59 2014 us=757928 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:22:59 2014 us=844142 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:23:00 2014 us=79703 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:23:00 2014 us=164154 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:23:00 2014 us=584263 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:23:00 2014 us=684271 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:23:00 2014 us=684373 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:23:00 2014 us=811556 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:23:00 2014 us=811648 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:23:00 2014 us=811697 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:23:00 2014 us=811743 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:23:01 2014 us=85926 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:23:01 2014 us=354046 marcinb/31.61.138.19:5247 MULTI: bad source address from client [fe80::c5f5:e651:8726:9b57], packet dropped
Thu Mar 20 09:23:01 2014 us=938677 marcinb/31.61.138.19:5247 NOTE: --mute triggered...

mb76pl

yes, I did, and it seems to work correctly:

C:\>tracert -d 192.168.10.7

Tracing route to 192.168.10.7 over a maximum of 30 hops

  1     4 ms     3 ms     1 ms  192.168.1.1
  2     3 ms     1 ms     1 ms  192.168.1.111
  3     *     ^C
C:\>

mb76pl

i changed policies to ACCEPT few days ago to rule it out as possible cause

root:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere             tcp flags:!FIN,SYN,RST,ACK/SYN
ACCEPT     udp  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

maikcat

can you also please :

1)post the output of iptables -L -t nat on your openvpn server
2)tracert vpns server tun ip from one of your lans pc,
3)tracert vpn client tun ip from one of lans pc.
4)keep for testing ALL firewalls/antivirus disabled

also add a static route to your lan pc to send DIRECTLY traffic to vpn server (in case your router filters traffic).

Michael.

mb76pl

@debbie10t
unfortunately adding "topology subnet" to server's config didn't help.

@maikcat:
1)

root@:~# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
root@:~#

I'll be able to perform test 2-4 in 3 days, I'll post the results then.

mb76pl

Sorry it took a week, but i was really swamped.
So, @maikcat, I did the tests and here are the results:

2)

root@dm800se:~# traceroute -n 192.168.10.4
traceroute to 192.168.10.4 (192.168.10.4), 30 hops max, 38 byte packets
 1  192.168.1.1  0.706 ms  0.671 ms  0.667 ms
 2  192.168.1.111  0.655 ms  0.764 ms  0.699 ms
 3  *  *^C

traceroute to 192.168.10.1 (192.168.10.1), 30 hops max, 38 byte packets
 1  192.168.1.1  0.512 ms  0.669 ms  0.746 ms
 2  192.168.10.1  0.739 ms  0.839 ms  0.793 ms

4) antivir/firewall disabled

static route add on lan machine to send vpn class directly to VPN machine:

root@dm800se:~# ip route add 192.168.10.0/24 via 192.168.1.111

root@dm800se:~# ip ro list
default via 192.168.1.1 dev eth0
192.168.1.0/24 dev eth0  src 192.168.1.40
192.168.10.0/24 via 192.168.1.111 dev eth0

root@dm800se:~# traceroute -n 192.168.10.4
traceroute to 192.168.10.4 (192.168.10.4), 30 hops max, 38 byte packets
 1  192.168.1.111  0.299 ms  0.304 ms  0.030 ms
 2  *^C

root@dm800se:~# traceroute -n 192.168.10.1
traceroute to 192.168.10.1 (192.168.10.1), 30 hops max, 38 byte packets
 1  192.168.10.1  0.390 ms  0.172 ms  0.138 ms
root@dm800se:~#

maikcat

hello there,

dm800se? 😉
which image are you using? PLI?

i ve noticed you are testing against 192.168.10.4,
are you using topology subnet on your openvpn setup?
also, 192.168.10.4 has any firewall enabled on it?
can you post its routing table?

Michael.

mb76pl

yes, PLI 😃
This was the only linux running (other than VPN server) in my LAN at the time of testing.

I have topology subnet in my server's config
192.168.4.10 - no firewall
routing table

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0      192.168.4.1    192.168.4.119     10
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    306
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    306
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    306
      192.168.1.0    255.255.255.0     192.168.10.1     192.168.10.4     30
      192.168.4.0    255.255.255.0         On-link     192.168.4.119    266
    192.168.4.119  255.255.255.255         On-link     192.168.4.119    266
    192.168.4.255  255.255.255.255         On-link     192.168.4.119    266
     192.168.10.0    255.255.255.0         On-link      192.168.10.4    286
     192.168.10.4  255.255.255.255         On-link      192.168.10.4    286
   192.168.10.255  255.255.255.255         On-link      192.168.10.4    286
     192.168.56.0    255.255.255.0         On-link      192.168.56.1    276
     192.168.56.1  255.255.255.255         On-link      192.168.56.1    276
   192.168.56.255  255.255.255.255         On-link      192.168.56.1    276
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    306
        224.0.0.0        240.0.0.0         On-link      192.168.56.1    276
        224.0.0.0        240.0.0.0         On-link     192.168.4.119    266
        224.0.0.0        240.0.0.0         On-link      192.168.10.4    286
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    306
  255.255.255.255  255.255.255.255         On-link      192.168.56.1    276
  255.255.255.255  255.255.255.255         On-link     192.168.4.119    266
  255.255.255.255  255.255.255.255         On-link      192.168.10.4    286
===========================================================================
Persistent Routes:
  None

IPv6 Route Table
===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    306 ::1/128                  On-link
 25    276 fe80::/64                On-link
 12    266 fe80::/64                On-link
 12    266 fe80::2c2d:49e8:7c74:4a3e/128
                                    On-link
 25    276 fe80::4932:91a9:6fb3:dc8d/128
                                    On-link
  1    306 ff00::/8                 On-link
 25    276 ff00::/8                 On-link
 12    266 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

But, I think I craked it! Seems like the most relevant were the entries in openvpn's logfile:

Thu Mar 20 09:23:00 2014 us=684373 username/xxx.xxx.xxx.xxx:5247 MULTI: bad source address from client [fe80::c5f5:e651:xxxx:xxxx], packet dropped

Well, it clearly shows IPv6 address, so, because I don't give a damn about IPv6 on my setup - I tried disabling IPv6 on my client machine's viscosity virtual netowork adapter and - poof! - problem's gone 😃
I'll test it again few time just to make sure, but for now it seems like a valid solution.

Thank you guys for helping me 🙂