[bug] Slow client-connect script leads to traffic lockup

pgoose

Hi everybody.

I have discovered that slow client-connect and client-disconnect scripts during a client connection leads to traffic lockup for all clients connected to a server.
For a test purposes I created a dummy sleep.sh script:

#!/bin/bash
sleep 15
exit 0

And here is ping to 8.8.8.8 from the first client (Client A) connected to the server:

64 bytes from 8.8.8.8: icmp_seq=30533 ttl=49 time=62.578 ms
64 bytes from 8.8.8.8: icmp_seq=30534 ttl=49 time=60.343 ms
64 bytes from 8.8.8.8: icmp_seq=30535 ttl=49 time=60.523 ms
64 bytes from 8.8.8.8: icmp_seq=30536 ttl=49 time=60.212 ms
64 bytes from 8.8.8.8: icmp_seq=30537 ttl=49 time=61.006 ms
64 bytes from 8.8.8.8: icmp_seq=30538 ttl=49 time=60.870 ms
64 bytes from 8.8.8.8: icmp_seq=30539 ttl=49 time=60.844 ms
64 bytes from 8.8.8.8: icmp_seq=30540 ttl=49 time=63.178 ms  <------------------- Client B started connection and sleep.sh executed via client-connect
64 bytes from 8.8.8.8: icmp_seq=30541 ttl=49 time=136.251 ms
64 bytes from 8.8.8.8: icmp_seq=30542 ttl=49 time=102.526 ms
64 bytes from 8.8.8.8: icmp_seq=30543 ttl=49 time=60.604 ms
64 bytes from 8.8.8.8: icmp_seq=30544 ttl=49 time=70.402 ms
Request timeout for icmp_seq 96082
Request timeout for icmp_seq 96083
Request timeout for icmp_seq 96084
Request timeout for icmp_seq 96085
Request timeout for icmp_seq 96086
Request timeout for icmp_seq 96087
Request timeout for icmp_seq 96088
Request timeout for icmp_seq 96089
Request timeout for icmp_seq 96090
Request timeout for icmp_seq 96091
Request timeout for icmp_seq 96092
Request timeout for icmp_seq 96093
Request timeout for icmp_seq 96094
64 bytes from 8.8.8.8: icmp_seq=30545 ttl=49 time=14630.349 ms
64 bytes from 8.8.8.8: icmp_seq=30546 ttl=49 time=13629.352 ms
64 bytes from 8.8.8.8: icmp_seq=30551 ttl=49 time=8624.299 ms
64 bytes from 8.8.8.8: icmp_seq=30548 ttl=49 time=11627.413 ms
64 bytes from 8.8.8.8: icmp_seq=30549 ttl=49 time=10626.649 ms
64 bytes from 8.8.8.8: icmp_seq=30547 ttl=49 time=12628.408 ms
64 bytes from 8.8.8.8: icmp_seq=30550 ttl=49 time=9625.859 ms
64 bytes from 8.8.8.8: icmp_seq=30558 ttl=49 time=1617.810 ms
64 bytes from 8.8.8.8: icmp_seq=30552 ttl=49 time=7623.460 ms
64 bytes from 8.8.8.8: icmp_seq=30553 ttl=49 time=6622.398 ms
64 bytes from 8.8.8.8: icmp_seq=30559 ttl=49 time=617.378 ms
64 bytes from 8.8.8.8: icmp_seq=30555 ttl=49 time=4620.089 ms
64 bytes from 8.8.8.8: icmp_seq=30554 ttl=49 time=5673.718 ms
64 bytes from 8.8.8.8: icmp_seq=30556 ttl=49 time=3672.284 ms
64 bytes from 8.8.8.8: icmp_seq=30557 ttl=49 time=2671.990 ms
64 bytes from 8.8.8.8: icmp_seq=30560 ttl=49 time=149.811 ms
64 bytes from 8.8.8.8: icmp_seq=30561 ttl=49 time=110.026 ms
64 bytes from 8.8.8.8: icmp_seq=30562 ttl=49 time=107.493 ms
64 bytes from 8.8.8.8: icmp_seq=30563 ttl=49 time=61.229 ms
64 bytes from 8.8.8.8: icmp_seq=30564 ttl=49 time=89.112 ms
64 bytes from 8.8.8.8: icmp_seq=30565 ttl=49 time=88.898 ms
64 bytes from 8.8.8.8: icmp_seq=30566 ttl=49 time=60.797 ms  <------------------- Client B connected and sleep.sh finished
64 bytes from 8.8.8.8: icmp_seq=30567 ttl=49 time=60.472 ms
64 bytes from 8.8.8.8: icmp_seq=30568 ttl=49 time=61.226 ms
64 bytes from 8.8.8.8: icmp_seq=30569 ttl=49 time=60.482 ms
64 bytes from 8.8.8.8: icmp_seq=30570 ttl=49 time=60.457 ms

Server config:

lport                   443
mode                    server
proto                   tcp-server
tls-server
dev                     tun
script-security         2
ca                      /etc/openvpn/cert/ca.crt
cert                    /etc/openvpn/cert/server.crt
key                     /etc/openvpn/cert/server.key  # This file should be kept secret
dh                      /etc/openvpn/cert/dh2048.pem
cipher                  AES-256-CBC
client-cert-not-required
username-as-common-name
float
ifconfig                172.16.1.1 172.16.1.2
ifconfig-pool           172.16.1.2 172.16.1.254 #255.255.255.0
route                   172.16.1.0 255.255.255.0
comp-lzo                yes
persist-key
persist-tun
tun-mtu                 1500
mssfix
push                    "dhcp-option DNS 172.16.1.1"
push                    "dhcp-option DNS 8.8.8.8"
push                    "route-delay 2"
push                    "redirect-gateway def1"
client-connect          /opt/sleep.sh
client-disconnect       /opt/sleep.sh
auth-user-pass-verify   /opt/auth.sh via-env
management             127.0.0.1 7500
verb                    3

Tested on OpenVPN 2.2.1 from ubuntu 12.04 repo and manually compiled 2.2.2, 2.3b1.

Could anyone test it before bug reporting?

Mimiko

Yes, I'm configming this as a bug. Moreover, if during this client's waiting you will try to stop OpenVPN, the batch will be processed until all trying clients will be passed.

janjust

this is a known shortcoming of OpenVPN; unfortunately it is not easy to overcome, as openvpn is single threaded at the moment. You will have to wait for OpenVPN 3.0 (whenever that comes out) for this behaviour to change.

krzee

JJK talks about this issue in his book. He also talks about the fact that you can use the plugin interface instead of scripting interface to get around this problem.
The scripting interface runs in the same thread, and blocks openvpn
The plugin interface runs in its own thread, so it does not block openvpn.

pent1ckel

Hi everybody,

I know that this thread is very very old but I didn't wanted to open a new one. I run into the same problem and I tried to use a plugin for client-connect/client-disconnect but still everything stucks as long as the plugin is doing something.

Can somebody please verify whether the statement that plugins shouldn't block is still right or something changed over years.

Thanks a lot,
Bernd

TinCanTech

pent1ckel wroteCan somebody please verify whether the statement that plugins shouldn't block is still right or something changed over years

Nothing has changed in the current version of openvpn 2.x and will not change until openvpn 3.x (which is still some way off)

However,

pent1ckel wroteI tried to use a plugin for client-connect/client-disconnect but still everything stucks as long as the plugin is doing something

If you are calling a plugin via --client-connect/disconnect this will cause a block at the server.

Please read the HOWTO:
https://openvpn.net/index.php/open-source/documentation/howto.html#auth

Note the section: Using Shared Object or DLL Plugins

I believe this: plugin /usr/share/openvpn/plugin/lib/openvpn-auth-pam.so login is the fastest solution.

For further information please read HOWTO: Request Help ! and start your own thread.

dazo

Let's clarify this properly once and for all.

1. Yes, --client-connect scripts will block during authentication. To avoid blocking other clients, it is crucially important that the authentication script executes and returns a result as fast as possible.
2. Yes, --plugin may avoid blocking other clients during authentication. But that depends entirely on the plug-in you use. If the plug-in does not implement deferred authentication, it will block just as --client-connect. Only plug-ins which implements deferred authentication will not block other clients.
3. The openvpn-auth-pam.so plug-in does not implement deferred authentication. So if the PAM subsystem is slow to provide an authentication result, it will block other clients.