OpenVPN Support Forum

Posted: **Mon Dec 21, 2015 5:26 pm**

I maintain a distribution of embedded Linux and use OpenVPN. It generally works perfectly but recently I encountered a weird problem on only one of my devices. The device has upgraded to latest version of distribution and since then all packets that are sent though the tunnel are lost (i.e. ping 10.4.0.1 results in 100% packet loss).

I investigated the problem a bit and came to a conclusion that these packets must be dropped somewhere in kernel, because:
- iptables is disabled
- I see packets in tcpdump sent to 10.4.0.1, but there are no corresponding entries in OpenVPN's logs (the only packets logged are probably the ping packets arriving from server side).
- routing table looks very similar on problematic device to regular devices:

Code: Select all

# ip ro
default dev ppp0  scope link 
10.4.0.1 dev tun0  proto kernel  scope link  src 10.4.0.18 
192.168.80.0/22 dev eth0  proto kernel  scope link  src 192.168.81.123 
192.168.202.0 dev ppp0  proto kernel  scope link  src 10.187.248.19

- ifconfig says packets are dropped:

Code: Select all

# ifconfig tun0
tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:10.4.0.18  P-t-P:10.4.0.1  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:304 errors:0 dropped:580 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:0 (0.0 B)  TX bytes:65089 (63.5 KiB)

What is the simpest way to debug this? I know there is something called systemtap that might be helpful, but maybe there are some other techniques... Has anyone encountered similar issue and knows what I could check?

These are my configs, client and server side respectively:

Code: Select all

syslog openvpn_gw_srv

client

proto tcp
port 8911

nobind
comp-lzo
mute-replay-warnings
ns-cert-type server
cipher BF-CBC

hand-window 170
ca   /etc/openvpn/ca.crt
cert /etc/openvpn/client.crt
key  /etc/openvpn/client.key

persist-key

ping-exit 92
reneg-sec 86400

management localhost 7012

Code: Select all

dev tun1
server 10.4.0.0 255.255.0.0
#proto udp
proto tcp

port 8894

ping  30

comp-lzo
mute-replay-warnings
cipher BF-CBC

dh   /etc/ssl/private/server/dh1024.pem
ca   /etc/ssl/private/server/ca.crt
cert /etc/ssl/private/server/server.crt
key  /etc/ssl/private/server/server.key

ifconfig-pool-persist ipp.txt
ifconfig-pool-linear

persist-key
persist-tun

reneg-sec 86400
reneg-pkts 0

ping-restart 0

status openvpn-status.log
verb 5
management localhost 7012
daemon server-openvpn

hand-window 600

The port mismatch is not a mistake, there is a port forwarding condigured on the server.

Posted: **Mon Dec 21, 2015 6:29 pm**

It will be worth checking your OpenVPN logs (at --verb 4) for errors ..

Posted: **Tue Dec 22, 2015 11:03 am**

There is nothing suspicious in logs. I also tested using OpenVPN 2.3.9 and results are the same (originally I used 2.3.0).

It is interesting that VPN works as expected when I configure client to connect over eth0 instead of ppp0 by changing remote address and port. It is even more interesting that when I revert configuration to the default VPN is also working.

Posted: **Tue Dec 22, 2015 12:09 pm**

mass85 wrote:There is nothing suspicious in logs

We will take your word on that ..

mass85 wrote:It is interesting that VPN works as expected when I configure client to connect over eth0 instead of ppp0 by changing remote address and port

Details not included ..

Posted: **Tue Dec 22, 2015 12:28 pm**

Traffic wrote:
mass85 wrote:It is interesting that VPN works as expected when I configure client to connect over eth0 instead of ppp0 by changing remote address and port
Details not included ..

Details follow.

The default config is the one I pasted in my first post and is not altered in described scenario. By default in my system openvpn is run using command openvpn --dev tun0 --remote my.public.address.pl --config /etc/openvpn/openvpn.conf. Then I killed openvpn and run openvpn --dev tun3 --remote 192.168.82.121 --config /etc/openvpn/openvpn.conf --port 8894, so that client connected to the very same openvpn server process on the same machine as in the default scenario. ping 10.4.0.1 worked fine. Then I run openvpn in the defulat way and ping also worked. Then I rebooted system, openvpn was started automatically in the deafult way and ping also worked, so the "fix" was persistent...

Posted: **Tue Dec 22, 2015 12:44 pm**

mass85 wrote:The default config is the one I pasted in my first post and is not altered in described scenario. By default in my system openvpn is run using command openvpn --dev tun0 --remote my.public.address.pl --config /etc/openvpn/openvpn.conf. Then I killed openvpn and run openvpn --dev tun3 --remote 192.168.82.121 --config /etc/openvpn/openvpn.conf --port 8894

More details not included in your initial post ..

mass85 wrote:so the "fix" was persistent...

You mean you fixed it ?

It is difficult to follow the problem when you do not post the full details.

Posted: **Tue Dec 22, 2015 1:04 pm**

Do you suggest to edit first post and add what I mentioned in my previous one?

By "fix" I mean the act of running openvpn process with different remote and port, because since then the VPN works fine even with the default remote and port, even after reboot.

Posted: **Tue Dec 22, 2015 4:14 pm**

One question:

Traffic wrote:Then I killed openvpn

How did you do that ?

Posted: **Tue Dec 22, 2015 4:52 pm**

Traffic wrote:One question:
Traffic wrote:Then I killed openvpn
How did you do that ?

I did that with command killall openvpn.

My further investigation revealed that the Send-Q gets full on the VPN's client size:

Code: Select all

# netstat -nt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 192.168.81.123:22       192.168.81.24:49876     ESTABLISHED 
tcp        0      0 127.0.0.1:7012          127.0.0.1:33536         ESTABLISHED 
tcp        0  53775 10.190.61.227:46525     <THE VPN SERVER IP>:8911     ESTABLISHED 
tcp        0      0 127.0.0.1:33536         127.0.0.1:7012          ESTABLISHED

I can see in client's logs that when I issue ping 10.4.0.5 on the server side, packets are read. So it looks like a one way communication. Because netstat -nt does not show any problems with buffers utilization on the server side, I think the whole problem might be related to the ports redirection, perhaps it somehow saturates just after TLS handshake is finished and no more packets in this TCP connection get redirected to VPN server. I will investigate more tomorrow.

Posted: **Tue Dec 22, 2015 5:54 pm**

mass85 wrote:I did that with command killall openvpn.

Check your logs to make sure the VPN routes are torn down correctly ..

Also, according to the configs you posted:

mass85 wrote:I can see in client's logs that when I issue ping 10.4.0.5

10.4.0.5 is not ping-able by design .. it is an endpoint to the tunnel, for use by OpenVPN only.

Posted: **Wed Dec 23, 2015 9:16 am**

Traffic wrote:
mass85 wrote:I did that with command killall openvpn.
Check your logs to make sure the VPN routes are torn down correctly ..

Yes, they are:

Code: Select all

>LOG:1450860900,N,event_wait : Interrupted system call (code=4)
>LOG:1450860900,,TCP/UDP: Closing socket
>LOG:1450860900,,Closing TUN/TAP interface
>LOG:1450860900,I,/sbin/ip addr del dev tun0 local 10.4.0.5 peer 10.4.0.1

Traffic wrote:10.4.0.5 is not ping-able by design .. it is an endpoint to the tunnel, for use by OpenVPN only.

This destroys my world, can you elaborate a bit more? In my experience with OpenVPN pinging peer has always worked from both sides: client and server. Moreover my applications working above VPN use those addresses as destination, so application on the VPN server side sends UDP packets to 10.4.0.5 port 11040, whereas application on the VPN client side sends UDP packets to 10.4.0.1 port 11040 and it has always worked like this.

Posted: **Thu Dec 24, 2015 12:12 am**

mass85 wrote:OpenVPN pinging peer has always worked from both sides

Yes .. but 10.4.0.5 is not the peer .. 10.4.0.1 is the server

See --topology in The Manual v23x

OpenVPN Support Forum

All packets dropped at client side - how to debug?

All packets dropped at client side - how to debug?

Re: All packets dropped at client side - how to debug?

Re: All packets dropped at client side - how to debug?

Re: All packets dropped at client side - how to debug?

Re: All packets dropped at client side - how to debug?

Re: All packets dropped at client side - how to debug?

Re: All packets dropped at client side - how to debug?

Re: All packets dropped at client side - how to debug?

Re: All packets dropped at client side - how to debug?

Re: All packets dropped at client side - how to debug?

Re: All packets dropped at client side - how to debug?

Re: All packets dropped at client side - how to debug?