Hi all, I have a couple of setups sofar, but all of them support about 10-20 bridged/routed clients. Now I need to build a solution for thousands of non-agressive clients. I'm starting this topic as I hope it might help others scale their solutions too. I need openvpn because I need to encrypt IPv6 traffic over IPv4, with clients using dynamic IP-s. Clients will send just a 5-10 KB per hour. I couldn't find an easier tunneling/encryption solution sofar. I started by benchmarking different machines and got strange results: type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes blowfish cbc 77699.61k 80955.58k 81224.52k 81504.88k 81274.16k Intel XeonCPU E5-2620@2GHz blowfish cbc 106308.12k 111811.31k 113280.94k 114006.13k 113715.88k Intel Corе i5-2500K CPU@3.3GHz [b]blowfish cbc 247483648.00k 282553152.00k 292705792.00k 297224192.00k 292716544.00k Intel Xeon CPU 2.80GHz[/b] aes-128 cbc 99747.24k 108173.49k 110013.72k 110066.30k 110909.20k Intel Corе i5-2500K CPU@3.3GHz aes-128 cbc 74038.08k 105149.77k 120364.86k 124562.92k 127773.21k Intel Xeon CPU 2.80GHz [b]aes-128 cbc 75810.92k 121050.52k 137173.90k 146973.69k 147916.80k Intel XeonCPU E5-2620@2GHz[/b] sign verify sign/s verify/s rsa 1024 bits 0.001815s 0.000076s 551.0 13108.5 Intel XeonCPU E5-2620@2GHz rsa 1024 bits 0.000217s 0.000012s 4618.6 80487.6 Intel Corе i5-2500K CPU@3.3GHz [b]rsa 1024 bits 0.000000s 0.000000s 2727000.0 65302000.0 Intel Xeon CPU 2.80GHz[/b] rsa 2048 bits 0.009252s 0.000223s 108.1 4481.8 Intel XeonCPU E5-2620@2GHz rsa 2048 bits 0.001345s 0.000042s 743.5 23746.2 Intel Corе i5-2500K CPU@3.3GHz [b]rsa 2048 bits 0.000002s 0.000000s 522000.0 21705000.0 Intel Xeon CPU 2.80GHz[/b] dsa 1024 bits 0.000768s 0.000926s 1301.6 1079.8 Intel XeonCPU E5-2620@2GHz dsa 1024 bits 0.000131s 0.000142s 7606.4 7020.2 Intel Corе i5-2500K CPU@3.3GHz [b]dsa 1024 bits 0.000000s 0.000000s 6359000.0 5445000.0 Intel Xeon CPU 2.80GHz[/b] dsa 2048 bits 0.002258s 0.002697s 442.8 370.7 Intel XeonCPU E5-2620@2GHz dsa 2048 bits 0.000414s 0.000489s 2413.9 2045.0 Intel Corе i5-2500K CPU@3.3GHz [b]dsa 2048 bits 0.000000s 0.000001s 2151000.0 1738000.0 Intel Xeon CPU 2.80GHz[/b] As you see one of the machines (IBM eserver xSeries 345 shows 2600 times greater performance for blowfish. I thought there might be some hardware acceleration, but no module was loaded: lsmod | grep -e pad -e via Anything else to check on this machine? I also wanted to know if: 1. Running a single openvpn server process per core (quad core = four openvpn server instances) is a good practice 2. I read that for 1000 clients I need 1,5 GB RAM. [url]https://forums.openvpn.net/topic11683.html[/url] Anybody else running 1000 clients on single instance? 3. It is obvious that I have to increase the re-keying interval as it will consume CPU and I want to avoid it as much as possible. That's why I can: - Disable --reneg-sec by setting it to 0 and using --reneg-bytes to some value which will randomize rekeying as traffic is almost equal on all lines (5-10%) difference - Increase --reneg-sec to 43200 (12 hours). How can I evaluate a maximum value without compromising security? 4. Reduce verbosity. The official Docu kinda puzzles me: # 0 is silent, except for fatal errors # 4 is reasonable for general usage # 5 and 6 can help to debug connection problems # 9 is extremely verbose verb 3 They select 3 but I don't see any info about 1,2 or 3. I guess I could also do with 1 or 2 5. Further reduce logging overload on by sending the info to --syslog 6. UDP is mandatory I believe 7. Using the above mentioned test I guess CFC-BF will be the right solution for speed and reasonable security 8. My clients will be using GPRs. I guess I will have to finetune the [b]mssfix[/b] and [b]fragment[/b] once I make a test with [b]mtu-test[/b] 9. On the networking side in my case I cannot benefit by using net.inet.ip.fastforwarding=1, because I will be routing ipv6 over my bridged tunnels. 10. Finally I need to make sure to rate limit the new incoming udp "connections" to some reasonable value in case my server need to go down and up in order not to start serving thousands of clients in the first minute or so I'm also considering the new IPv6 support in 2.3, but I wonder it if this new feature will be as stable and fast as good old ipv4 tunnel. Your corrections and suggestions are very welcome

IPv6 over IPv4 for one thousand clients

windwalker78

Hi all,

I have a couple of setups sofar, but all of them support about 10-20 bridged/routed clients.
Now I need to build a solution for thousands of non-agressive clients. I'm starting this topic as I hope it might help others scale their solutions too.
I need openvpn because I need to encrypt IPv6 traffic over IPv4, with clients using dynamic IP-s. Clients will send just a 5-10 KB per hour. I couldn't find an easier tunneling/encryption solution sofar.

I started by benchmarking different machines and got strange results:

type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
blowfish cbc 77699.61k 80955.58k 81224.52k 81504.88k 81274.16k Intel XeonCPU E5-2620@2GHz
blowfish cbc 106308.12k 111811.31k 113280.94k 114006.13k 113715.88k Intel Corе i5-2500K CPU@3.3GHz
blowfish cbc 247483648.00k 282553152.00k 292705792.00k 297224192.00k 292716544.00k Intel Xeon CPU 2.80GHz

aes-128 cbc 99747.24k 108173.49k 110013.72k 110066.30k 110909.20k Intel Corе i5-2500K CPU@3.3GHz
aes-128 cbc 74038.08k 105149.77k 120364.86k 124562.92k 127773.21k Intel Xeon CPU 2.80GHz
aes-128 cbc 75810.92k 121050.52k 137173.90k 146973.69k 147916.80k Intel XeonCPU E5-2620@2GHz
sign verify sign/s verify/s
rsa 1024 bits 0.001815s 0.000076s 551.0 13108.5 Intel XeonCPU E5-2620@2GHz
rsa 1024 bits 0.000217s 0.000012s 4618.6 80487.6 Intel Corе i5-2500K CPU@3.3GHz
rsa 1024 bits 0.000000s 0.000000s 2727000.0 65302000.0 Intel Xeon CPU 2.80GHz
rsa 2048 bits 0.009252s 0.000223s 108.1 4481.8 Intel XeonCPU E5-2620@2GHz
rsa 2048 bits 0.001345s 0.000042s 743.5 23746.2 Intel Corе i5-2500K CPU@3.3GHz
rsa 2048 bits 0.000002s 0.000000s 522000.0 21705000.0 Intel Xeon CPU 2.80GHz
dsa 1024 bits 0.000768s 0.000926s 1301.6 1079.8 Intel XeonCPU E5-2620@2GHz
dsa 1024 bits 0.000131s 0.000142s 7606.4 7020.2 Intel Corе i5-2500K CPU@3.3GHz
dsa 1024 bits 0.000000s 0.000000s 6359000.0 5445000.0 Intel Xeon CPU 2.80GHz
dsa 2048 bits 0.002258s 0.002697s 442.8 370.7 Intel XeonCPU E5-2620@2GHz
dsa 2048 bits 0.000414s 0.000489s 2413.9 2045.0 Intel Corе i5-2500K CPU@3.3GHz
dsa 2048 bits 0.000000s 0.000001s 2151000.0 1738000.0 Intel Xeon CPU 2.80GHz

As you see one of the machines (IBM eserver xSeries 345 shows 2600 times greater performance for blowfish.
I thought there might be some hardware acceleration, but no module was loaded:
lsmod | grep -e pad -e via

Anything else to check on this machine?

I also wanted to know if:
1. Running a single openvpn server process per core (quad core = four openvpn server instances) is a good practice
2. I read that for 1000 clients I need 1,5 GB RAM. https://forums.openvpn.net/topic11683.html Anybody else running 1000 clients on single instance?
3. It is obvious that I have to increase the re-keying interval as it will consume CPU and I want to avoid it as much as possible. That's why I can:
- Disable --reneg-sec by setting it to 0 and using --reneg-bytes to some value which will randomize rekeying as traffic is almost equal on all lines (5-10%) difference
- Increase --reneg-sec to 43200 (12 hours). How can I evaluate a maximum value without compromising security?
4. Reduce verbosity. The official Docu kinda puzzles me:
# 0 is silent, except for fatal errors
# 4 is reasonable for general usage
# 5 and 6 can help to debug connection problems
# 9 is extremely verbose
verb 3
They select 3 but I don't see any info about 1,2 or 3. I guess I could also do with 1 or 2
5. Further reduce logging overload on by sending the info to --syslog
6. UDP is mandatory I believe
7. Using the above mentioned test I guess CFC-BF will be the right solution for speed and reasonable security
8. My clients will be using GPRs. I guess I will have to finetune the mssfix and fragment once I make a test with mtu-test
9. On the networking side in my case I cannot benefit by using net.inet.ip.fastforwarding=1, because I will be routing ipv6 over my bridged tunnels.
10. Finally I need to make sure to rate limit the new incoming udp "connections" to some reasonable value in case my server need to go down and up in order not to start serving thousands of clients in the first minute or so

I'm also considering the new IPv6 support in 2.3, but I wonder it if this new feature will be as stable and fast as good old ipv4 tunnel.

Your corrections and suggestions are very welcome

windwalker78

I think I identified the reason for blowfish cbc to be so fast:

openssl speed bf-cbc
Doing blowfish cbc for 3s on 16 size blocks: 16519562 blowfish cbc's in 2.99s
Doing blowfish cbc for 3s on 64 size blocks: 4517321 blowfish cbc's in 3.00s
Doing blowfish cbc for 3s on 256 size blocks: 1149005 blowfish cbc's in 3.00s
Doing blowfish cbc for 3s on 1024 size blocks: 290213 blowfish cbc's in 2.99s
Doing blowfish cbc for 3s on 8192 size blocks: 35707 blowfish cbc's in 2.99s
OpenSSL 0.9.8k 25 Mar 2009
built on: Thu Oct 7 00:47:50 UTC 2010
options:bn(64,32) md2(int) rc4(idx,int) des(ptr,risc1,16,long) aes(partial) blowfish(idx)
compiler: cc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DTERMIO -O3 -march=i686 -Wa,--noexecstack -g -Wall -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DSHA1_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM
available timing options: TIMES TIMEB HZ=100 [sysconf value]
timing function used: times
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
blowfish cbc 88398.99k 96369.51k 98048.43k 99390.67k 97830.01k

Obviously the total makes the wrong calculation while diving processed/sec and of course aes-128-cbc seems better.