Firewall / NAT issue after sucessful connection.

Gaius380

Hello,

I have some issue with port forwarding / firewall after a successful connection between a smartphone (on a 4G mobile network) and an Ubuntu server with OpenVPN.

* Server *

Operating system:
Ubuntu 16.04

# uname -a
Linux hostname 4.4.0-75-generic #96-Ubuntu SMP Thu Apr 20 09:56:33 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Network setup:

$ ifconfig
eth1      Link encap:Ethernet  HWaddr 
          inet addr:192.168.0.30  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3487358 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3906812 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3594167158 (3.5 GB)  TX bytes:4006095175 (4.0 GB)
          Interrupt:17 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:8989 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8989 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:994910 (994.9 KB)  TX bytes:994910 (994.9 KB)

tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:10.8.0.1  P-t-P:10.8.0.1  Mask:255.255.255.0
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:208 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:12377 (12.3 KB)  TX bytes:0 (0.0 B)

sslh is also installed to redirect vpn request on port 443 to local 1194 (where OpenVPN is running)

cd /etc/openvpn
port 1194
proto tcp
dev tun
sndbuf 0
rcvbuf 0
ca ca.crt
cert server.crt
key server.key
dh dh.pem
tls-auth ta.key 0
topology subnet
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "redirect-gateway def1 bypass-dhcp"
push "dhcp-option DNS 8.8.8.8"
push "dhcp-option DNS 8.8.4.4"
keepalive 10 120
cipher AES-128-CBC
comp-lzo
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
verb 4
crl-verify crl.pem

Server log (at --verb 4 and client IP address removed)

nov. 01 15:51:31 hostname systemd[1]: Started OpenVPN connection to server.
nov. 01 15:51:31 hostname ovpn-server[7673]: NOTE: your local LAN uses the extremely common subnet address 192.168.0.x or 192.168.1.x.  Be aware that this might create routing conflicts
if you connect to the VPN server from public locations such as internet cafes that use the same subnet.
nov. 01 15:51:31 hostname ovpn-server[7673]: Diffie-Hellman initialized with 2048 bit key
nov. 01 15:51:31 hostname ovpn-server[7673]: Control Channel Authentication: using 'ta.key' as a OpenVPN static key file
nov. 01 15:51:31 hostname ovpn-server[7673]: Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
nov. 01 15:51:31 hostname ovpn-server[7673]: Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
nov. 01 15:51:31 hostname ovpn-server[7673]: TLS-Auth MTU parms [ L:1560 D:1182 EF:68 EB:0 ET:0 EL:3 ]
nov. 01 15:51:31 hostname ovpn-server[7673]: Socket Buffers: R=[87380->87380] S=[65536->65536]
nov. 01 15:51:31 hostname ovpn-server[7673]: TUN/TAP device tun0 opened
nov. 01 15:51:31 hostname ovpn-server[7673]: TUN/TAP TX queue length set to 100
nov. 01 15:51:31 hostname ovpn-server[7673]: do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
nov. 01 15:51:31 hostname ovpn-server[7673]: /sbin/ip link set dev tun0 up mtu 1500
nov. 01 15:51:31 hostname ovpn-server[7673]: /sbin/ip addr add dev tun0 10.8.0.1/24 broadcast 10.8.0.255
nov. 01 15:51:31 hostname ovpn-server[7673]: Data Channel MTU parms [ L:1560 D:1450 EF:60 EB:143 ET:0 EL:3 AF:3/1 ]
nov. 01 15:51:31 hostname ovpn-server[7673]: GID set to nogroup
nov. 01 15:51:31 hostname ovpn-server[7673]: UID set to nobody
nov. 01 15:51:31 hostname ovpn-server[7673]: Listening for incoming TCP connection on [undef]
nov. 01 15:51:31 hostname ovpn-server[7673]: TCPv4_SERVER link local (bound): [undef]
nov. 01 15:51:31 hostname ovpn-server[7673]: TCPv4_SERVER link remote: [undef]
nov. 01 15:51:31 hostname ovpn-server[7673]: MULTI: multi_init called, r=256 v=256
nov. 01 15:51:31 hostname ovpn-server[7673]: IFCONFIG POOL: base=10.8.0.2 size=252, ipv6=0
nov. 01 15:51:31 hostname ovpn-server[7673]: ifconfig_pool_read(), in='client,10.8.0.2', TODO: IPv6
nov. 01 15:51:31 hostname ovpn-server[7673]: succeeded -> ifconfig_pool_set()
nov. 01 15:51:31 hostname ovpn-server[7673]: IFCONFIG POOL LIST
nov. 01 15:51:31 hostname ovpn-server[7673]: client,10.8.0.2
nov. 01 15:51:31 hostname ovpn-server[7673]: MULTI: TCP INIT maxclients=1024 maxevents=1028
nov. 01 15:51:31 hostname ovpn-server[7673]: Initialization Sequence Complete

## connection from the client from here

nov. 01 16:02:39 hostname ovpn-server[7673]: MULTI: multi_create_instance called
nov. 01 16:02:39 hostname ovpn-server[7673]: Re-using SSL/TLS context
nov. 01 16:02:39 hostname ovpn-server[7673]: LZO compression initialized
nov. 01 16:02:39 hostname ovpn-server[7673]: Control Channel MTU parms [ L:1560 D:1182 EF:68 EB:0 ET:0 EL:3 ]
nov. 01 16:02:39 hostname ovpn-server[7673]: Data Channel MTU parms [ L:1560 D:1450 EF:60 EB:143 ET:0 EL:3 AF:3/1 ]
nov. 01 16:02:39 hostname ovpn-server[7673]: Local Options String: 'V4,dev-type tun,link-mtu 1560,tun-mtu 1500,proto TCPv4_SERVER,comp-lzo,keydir 0,cipher AES-128-CBC,auth SHA1,keysize 128,tls-auth,key-method 2,tls-server'
nov. 01 16:02:39 hostname ovpn-server[7673]: Expected Remote Options String: 'V4,dev-type tun,link-mtu 1560,tun-mtu 1500,proto TCPv4_CLIENT,comp-lzo,keydir 1,cipher AES-128-CBC,auth SHA1,keysize 128,tls-auth,key-method 2,tls-client'
nov. 01 16:02:39 hostname ovpn-server[7673]: Local Options hash (VER=V4): 'a642654b'
nov. 01 16:02:39 hostname ovpn-server[7673]: Expected Remote Options hash (VER=V4): '0bdd0804'
nov. 01 16:02:39 hostname ovpn-server[7673]: TCP connection established with [AF_INET]127.0.0.1:37444
nov. 01 16:02:39 hostname ovpn-server[7673]: TCPv4_SERVER link local: [undef]
nov. 01 16:02:39 hostname ovpn-server[7673]: TCPv4_SERVER link remote: [AF_INET]127.0.0.1:37444
nov. 01 16:02:39 hostname ovpn-server[7673]: 127.0.0.1:37444 TLS: Initial packet from [AF_INET]127.0.0.1:37444, sid=0d5e5219 6aac44b0
nov. 01 16:02:40 hostname ovpn-server[7673]: 127.0.0.1:37444 CRL CHECK OK: CN=ChangeMe
nov. 01 16:02:40 hostname ovpn-server[7673]: 127.0.0.1:37444 VERIFY OK: depth=1, CN=ChangeMe
nov. 01 16:02:40 hostname ovpn-server[7673]: 127.0.0.1:37444 CRL CHECK OK: CN=client
nov. 01 16:02:40 hostname ovpn-server[7673]: 127.0.0.1:37444 VERIFY OK: depth=0, CN=client
nov. 01 16:02:40 hostname ovpn-server[7673]: 127.0.0.1:37444 Data Channel Encrypt: Cipher 'AES-128-CBC' initialized with 128 bit key
nov. 01 16:02:40 hostname ovpn-server[7673]: 127.0.0.1:37444 Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
nov. 01 16:02:40 hostname ovpn-server[7673]: 127.0.0.1:37444 Data Channel Decrypt: Cipher 'AES-128-CBC' initialized with 128 bit key
nov. 01 16:02:40 hostname ovpn-server[7673]: 127.0.0.1:37444 Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
nov. 01 16:02:40 hostname ovpn-server[7673]: 127.0.0.1:37444 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
nov. 01 16:02:40 hostname ovpn-server[7673]: 127.0.0.1:37444 [client] Peer Connection Initiated with [AF_INET]127.0.0.1:37444
nov. 01 16:02:40 hostname ovpn-server[7673]: client/127.0.0.1:37444 MULTI_sva: pool returned IPv4=10.8.0.2, IPv6=(Not enabled)
nov. 01 16:02:40 hostname ovpn-server[7673]: client/127.0.0.1:37444 MULTI: Learn: 10.8.0.2 -> client/127.0.0.1:37444
nov. 01 16:02:40 hostname ovpn-server[7673]: client/127.0.0.1:37444 MULTI: primary virtual IP for client/127.0.0.1:37444: 10.8.0.2
nov. 01 16:02:41 hostname ovpn-server[7673]: client/127.0.0.1:37444 PUSH: Received control message: 'PUSH_REQUEST'
nov. 01 16:02:41 hostname ovpn-server[7673]: client/127.0.0.1:37444 send_push_reply(): safe_cap=940
nov. 01 16:02:41 hostname ovpn-server[7673]: client/127.0.0.1:37444 SENT CONTROL [client]: 'PUSH_REPLY,redirect-gateway def1 bypass-dhcp,dhcp-option DNS 8.8.8.8,dhcp-option DNS 8.8.4.4,route-gateway 10.8.0.1,topology subnet,ping 10,ping-restart 120,ifconfig 10.8.0.2 255.255.255.0' (status=1)

* Client *

Operating system:
Android 7.1.1, OpenVPN for Android version 0.6.73

client
dev tun
proto tcp
sndbuf 0
rcvbuf 0
remote <server-ip> 443
resolv-retry infinite
nobind
persist-key
persist-tun
remote-cert-tls server
cipher AES-128-CBC
comp-lzo
setenv opt block-outside-dns
key-direction 1
verb 3

The connection is successful (as it always has been), and I can ping machines on the server local network. So I dont think this is a client side issue.

But then, when I try to ping 8.8.8.8 for example (using a terminal emulator on Android), I've got no response whatsoever. Still, a tcpdump on the tun0 interface gives the following :

tcpdump -i tun0 icmp

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 262144 bytes
07:51:06.715970 IP 10.8.0.2 > google-public-dns-a.google.com: ICMP echo request, id 3, seq 1, length 64
07:51:07.696526 IP 10.8.0.2 > google-public-dns-a.google.com: ICMP echo request, id 3, seq 2, length 64
07:51:08.716327 IP 10.8.0.2 > google-public-dns-a.google.com: ICMP echo request, id 3, seq 3, length 64
07:51:09.709309 IP 10.8.0.2 > google-public-dns-a.google.com: ICMP echo request, id 3, seq 4, length 64

I've followed every official documentation to properly setup the firewall and the ip_forwarding setting :

/etc/ufw/before.rules :
CODE: SELECT ALL
#################################################
 12 # NAT (Network Address Translation) table rules
 13 *nat
 14 :POSTROUTING ACCEPT [0:0]
 15 -F
 16 # Allow traffic from clients to eth1
 17 -A POSTROUTING -s 10.8.0.0/24 -o eth1 -j MASQUERADE
 18
 19 # do not delete the "COMMIT" line or the NAT table rules above will not be processed
 20 COMMIT
 21 #################################################
 22


/etc/default/ufw
DEFAULT_FORWARD_POLICY="ACCEPT"


sysctl -a | grep "forwarding = 1"
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.eth1.forwarding = 1
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.tun0.forwarding = 1

The following thread on askubuntu might be related https://askubuntu.com/questions/898473/nat-using-iptables-on-ubuntu-16-04-doesnt-work, but there is no solution.

TinCanTech

Try ping 8.8.8.8 with this:

tcpdump -i eth1 icmp

eth1 ..

Gaius380

@TinCanTech thanks for your reply.
Well, there is nothing on eth1 :

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

TinCanTech

Did you enable IP Forwarding ?

Gaius380

Yes, that's the first thing I've checked :

sysctl -a | grep "forwarding = 1"
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.eth1.forwarding = 1
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.tun0.forwarding = 1

TinCanTech

Oops .. sorry I should have read more carefully.

Gaius380 wrote The connection is successful (as it always has been), and I can ping machines on the server local network

Can you take a tcpdump capture and post it showing that please.

Gaius380

Yeah sure :

sudo tcpdump -n -i eth1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
17:42:27.357823 IP 192.168.0.30 > 192.168.0.11: ICMP echo request, id 11, seq 1, length 64
17:42:27.358029 IP 192.168.0.11 > 192.168.0.30: ICMP echo reply, id 11, seq 1, length 64
17:42:28.424732 IP 192.168.0.30 > 192.168.0.11: ICMP echo request, id 11, seq 2, length 64
17:42:28.424938 IP 192.168.0.11 > 192.168.0.30: ICMP echo reply, id 11, seq 2, length 64
17:42:29.377301 IP 192.168.0.30 > 192.168.0.11: ICMP echo request, id 11, seq 3, length 64
17:42:29.377482 IP 192.168.0.11 > 192.168.0.30: ICMP echo reply, id 11, seq 3, length 64
17:42:30.357791 IP 192.168.0.30 > 192.168.0.11: ICMP echo request, id 11, seq 4, length 64
17:42:30.357995 IP 192.168.0.11 > 192.168.0.30: ICMP echo reply, id 11, seq 4, length 64
17:42:31.377712 IP 192.168.0.30 > 192.168.0.11: ICMP echo request, id 11, seq 5, length 64
17:42:31.377937 IP 192.168.0.11 > 192.168.0.30: ICMP echo reply, id 11, seq 5, length 64
17:42:32.377725 IP 192.168.0.30 > 192.168.0.11: ICMP echo request, id 11, seq 6, length 64
17:42:32.377914 IP 192.168.0.11 > 192.168.0.30: ICMP echo reply, id 11, seq 6, length 64
^C
12 packets captured
12 packets received by filter
0 packets dropped by kernel

(pinging the 192.168.0.11 machine on the local network)

TinCanTech

Please post output of route -n from the server.

Gaius380

TinCanTech wrote Please post output of route -n from the server.

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 eth1
10.8.0.0        0.0.0.0         255.255.255.0   U     0      0        0 tun0
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1

TinCanTech

All I can suggest is that you remove ufw and go back to iptables directly.

Then clear out all your firewall rules and insert only the documented rule:

iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE

Note: eth1 in your case ..

What version of openvpn is on the server ?
And I presume you keep your server up to date ? (apt-get update etc .. )

Gaius380

Just tried that :

ufw disable
iptables -F
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth1 -j MASQUERADE

iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  10.8.0.0/24          anywhere
SNAT       all  --  10.8.0.0/24          anywhere             to:192.168.0.30
MASQUERADE  all  --  10.8.0.0/24          anywhere

Still not working.

System and openvpn are up to date :

apt-cache policy openvpn
openvpn:
  Installed: 2.3.10-1ubuntu2.1
  Candidate: 2.3.10-1ubuntu2.1

TinCanTech

This will not make any difference the the forwarding problem but you can use openvpn 2.4
(Which comes with some new goodies)

Also, please post output of iptables-save

Gaius380

Ok I'll try that this weekend. Maybe another VPN server (I don't know any other). If this is an Ubuntu isssue, there is not much I can do, maybe I should tried to install another VPN server on another machine (e.g, on my laptop which runs on ArchLinux).
Thanks again for your help

Gaius380

@TinCanTech I may have found the solution... thanks to the iptable-save command which has displayed a huge number of lines related to some pgl service running... I've disabled pgl temporarily and now I can ping 8.8.8.8
I should have thought of that, sorry. Now, I just need to find the proper whitelisting rules for pgl.

Gaius380

For those who have the same issue, here is the solution :
Add

CHAINS="INPUT OUTPUT"

to /etc/pgl/pglcmd.conf.

By default, all the chains are filtered by pgl, including the FWD chain, which was the cause of this issue.

TinCanTech

Gaius380 wrote I've disabled pgl temporarily and now I can ping 8.8.8.8

Great 8-) thanks for letting us know.