Scalability, UDP buffer code 105, learn-address problem

[jnorthco]

I'm having what seem to be scalability issues with OpenVPN that I can't seem to figure out.

I have an OpenVPN server that uses certificates. I am running two instances of the server, one UDP on port 1194, and one TCP on port 443. My clients are configured to try UDP first, but then try TCP since some of the clients are on networks where outbound connections are restricted. Both servers share the same authentication information. I have also added to the client configuration two extra connections that point to the same server IP as the first two - we once had a DNS outage that took down our VPN, so to avoid that we now have redundant entries on separate DNS servers.

UDP connections come in on tun1, and TCP on tun2.

In order to get the same IP for each client all the time, I have a client configuration directory. These (and the private network range) are both shared between the UDP and TCP servers. Also, I have a custom script that puts in the correct route (to tun1 or tun2) when each client connects, and removes it when they disconnect.

My intention here is to have all clients permanently connected - I am attempting to monitor the health of these clients over the VPN, so they need to be connected always.

This setup seemed to have been working well for several months, but recently I started having problems with clients disconnecting for no particular reason. I found lots of the following in my logs:

Code: Select all

Oct 21 10:25:17 vpn.xx.com ovpn-server[1470]: player17/206.228.220.132:49139 write UDPv4 []: No buffer space available (code=105)

I'm not sure whether or not this is related, I was not looking for these entries before the problem started. I'm now up to about 100 clients, so I am assuming that the increase in clients is somehow the root of the problem.

I also sometimes see that clients are connected, but the route for their connection has not been added by my script as I think it should. Once I add the route manually, then I can communicate with the client again.

Client and both server configurations are included below.

Does anybody have any ideas on:

* how to solve the UDP write errors

I've tried already increasing sndbuf, rcvbuf, and txqueuelen in the OpenVPN configs. I've also tried playing with /proc/sys/net/ipv4/udp_rmem_min, /proc/sys/net/ipv4/udp_wmem_min, and /proc/sys/vm/min_free_kbytes without any apparent effect.

* how to make sure that the special per-client routes are always added

* any other issues with having many clients that might be causing my problems

I can't see anything else in the logs that seems relevant, but maybe I don't know what to look for.

* any suggestions on how to do this better?

Thanks,

James

=============

Server config (Debian Lenny, OpenVPN 2.1~rc11-1)

====UDP server====

Code: Select all

port 1194
proto udp
sndbuf 262144
rcvbuf 262144
txqueuelen 500
dev tun1
topology subnet
ca easy-rsa/keys/ca.crt
cert easy-rsa/keys/server.crt
key easy-rsa/keys/server.key
crl-verify crl.pem
dh easy-rsa/keys/dh2048.pem
server 172.27.0.0 255.255.0.0
ifconfig-pool-persist ipp.txt 0
client-config-dir ccd
learn-address /etc/openvpn/udp-route.sh
script-security 2
keepalive 10 120
tls-auth ta.key 0 # This file is secret
cipher AES-128-CBC
comp-lzo
persist-key
persist-tun
status openvpn-status.log
verb 5
management 127.0.0.1 5000

====TCP server====

Code: Select all

local xx.yy.zz.110
port 443
proto tcp
tcp-queue-limit 256
dev tun2
topology subnet
ca easy-rsa/keys/ca.crt
cert easy-rsa/keys/server.crt
key easy-rsa/keys/server.key  # This file should be kept secret
crl-verify crl.pem
dh easy-rsa/keys/dh2048.pem
server 172.27.0.0 255.255.0.0
ifconfig-pool-persist ipp.txt 0
client-config-dir ccd
keepalive 10 120
tls-auth ta.key 0 # This file is secret
cipher AES-128-CBC   # AES
comp-lzo
persist-key
persist-tun
status openvpn-status-tcp.log
learn-address /etc/openvpn/tcp-route.sh
script-security 2
verb 3
management 127.0.0.1 5001

====udp-route.sh====

Code: Select all

#!/bin/bash

if [[ $1 = 'add' ]]; then
	ip route add $2 dev tun1 src 172.27.0.1
fi
if [[ $1 = 'delete' ]]; then
	ip route del $2 dev tun1 src 172.27.0.1
fi

====tcp-route.sh====

Code: Select all

#!/bin/bash

if [[ $1 = 'add' ]]; then
	ip route add $2 dev tun2 src 172.27.0.1
fi
if [[ $1 = 'delete' ]]; then
	ip route del $2 dev tun2 src 172.27.0.1
fi

====client config====

Code: Select all

client
dev tun
topology subnet

<connection>
remote vpn.xx.com 1194 udp
</connection>

<connection>
remote vpn.xx.com 443 tcp
</connection>

<connection>
remote vpn.yy.com 1194 udp
</connection>

<connection>
remote vpn.yy.com 443 tcp
</connection>

resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
ca ca.crt
cert client.crt
key client.key
remote-cert-tls server
tls-auth ta.key 1
cipher AES-128-CBC
comp-lzo
verb 3