Max number of concurrent users for this setup

drodl

We're running a VPN service based on openvpn

I was wandering if you can say, from your experience, how many concurrent "average" openvpn sessions can this hold (not compromising usability):

Dell PowerEdge R210
Intel(R) Core(TM) i3 CPU 540 @ 3.07GHz (4 cores)
2GB RAM
1Gbps connection, 4 IP addresses
Linux 2.6 (Debian 6)

Thanks in advance.

Mimiko

The number of concuretn conenctions vatly depend of the tun-connection setup (routing or bridging), the number of OpenVPN processes, the bandwith of internet connection. By default one OpenVPN process run in one-thread mode, using only one logical CPU.

drodl

server is running multiple instances of openvpn server, on different ips. http traffic is redirected through a proxy (running on same server).

there is about 25 users connected to each instance (total 150) - more users can connect, but the service becomes very slow and stops being usable. with a single openvpn instance it's the same: with about 150 concurrent users connected, service becomes unusable.

openvpn config: dev tun, proto udp, fragment 1200, mssfix (tried with proto tcp before, was worse).

tried some sysctl tuning and some QoS using iptables and tc. using "yeah" tcp_congestion_control

peak stats from munin:
firewall throughput: 10k packets/sec received, 5k forwarded.
tcp connections established: 3k
tcp connections time_wait: 6k
udp connections: 9k
proxy requests per second: 25
cpu usage: 25%
memory usage: 35%
network usage: 6%

Any idea where could bottleneck be? And how far theoretically we can push the concurrent users on the above mentioned hardware?
Thanks

Mimiko

Any idea where could bottleneck be? And how far theoretically we can push the concurrent users on the above mentioned hardware?

The best way to tweak is compile the source to your needs by eliminating unneeded options.

defsdoor

I'm guessing you have a 4 core CPU and it's 25% usage is one core loaded to the max.

You said you have multiple instances of openvpn running - bind each to a different core.

drodl

Mimiko wrote The best way to tweak is compile the source to your needs by eliminating unneeded options.

it doesn't seem that openvpn is the bottleneck, it's just the entry point to the server... the problem might be OS related, or... hardware related, or both
Would be interesting to know how far can you push max users on this type of hardware... Any experience with this?

defsdoor wrote I'm guessing you have a 4 core CPU and it's 25% usage is one core loaded to the max.

You said you have multiple instances of openvpn running - bind each to a different core.

there are 7 instances of openvpn running.
exact core loads on peak would be something like:
core 1: 70%
core 2: 5%
core 3: 20%
core 4: 5%
that's average 25% usage. most of the time - cpu usage is lower than that and the load on a single openvpn instance is lower than 20% cpu usage.

dazo

Please make sure that you really have pinned each of the OpenVPN processes to different CPUs. The taskset command can be used to do this. As OpenVPN is not multi-threaded, spreading OpenVPN processes to separate CPUs will most likely give the best performance.

From what we have heard from other users over the years is that the load is low and performance very good up to approx 150 users per OpenVPN process. From that point, the performance begins to decrease. Those clients have then all mostly been very active clients, with a lot of traffic. If you clients have less traffic, you might experience that your OpenVPN process can handle some more clients for slowing down.

And before you ask, yes, we're thinking about how to make OpenVPN multi-threaded. However, in many situations it is not efficient to have a thread per client - as applications usually perform worse if you have more threads than CPU cores. This will anyway not be implemented before OpenVPN 3.

drodl

dazo wrotePlease make sure that you really have pinned each of the OpenVPN processes to different CPUs. The taskset command can be used to do this. As OpenVPN is not multi-threaded, spreading OpenVPN processes to separate CPUs will most likely give the best performance.

From what we have heard from other users over the years is that the load is low and performance very good up to approx 150 users per OpenVPN process. From that point, the performance begins to decrease. Those clients have then all mostly been very active clients, with a lot of traffic. If you clients have less traffic, you might experience that your OpenVPN process can handle some more clients for slowing down.

And before you ask, yes, we're thinking about how to make OpenVPN multi-threaded. However, in many situations it is not efficient to have a thread per client - as applications usually perform worse if you have more threads than CPU cores. This will anyway not be implemented before OpenVPN 3.

ok, thanks for suggestion about pinning openvpn processes to cores - but it doesn't change much. cpu usage is about the same (close to nothing)
the number of users on each openvpn instance is not even close to 150, it's about 25

when totally there is less than 100 users connected on all instances - everything works great! when there is more than 100 users - everything goes slow (or doesn't go at all - endless loading on webpages from internet via proxy).

chrjae

Could it be the open file limit? ulimit -n or ulimit -S -n. I don't have experience using openvpn with many users, so that's just an idea.