We don't have any kind of MFA. Radius is ok, all other (non openvpnas) our servers have no problems. Also I checked radius answers using tcpdump at openvpnas, I saw ~15ms time of answers. At the time the server is in this strange state, there is almost no radius requests I see in tcpdump.
In the server log it looks so:
Code: Select all
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: "2022-10-19 04:10:55 10.255.255.150:32650 PUSH: Received control message: 'PUSH_REQUEST'"
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 peer info: IV_VER=3.git::081bfebe'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 peer info: IV_PLAT=ios'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 peer info: IV_NCP=2'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 peer info: IV_TCPNL=1'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 peer info: IV_PROTO=30'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 peer info: IV_CIPHERS=AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 peer info: UV_ASCLI_VER=3.3.2-5086'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 peer info: UV_PLAT_REL=16.0.2'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 peer info: UV_UUID=D6D361A8-E7DF-4ECF-9306-A7A64A6208F5'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 peer info: IV_GUI_VER=net.openvpn.connect.ios_3.3.2-5086'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 peer info: IV_SSO=webauth,openurl,crtext'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 peer info: IV_HWADDR=D6D361A8-E7DF-4ECF-9306-A7A64A6208F5'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 peer info: IV_SSL=OpenSSL_1.1.1n__15_Mar_2022'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: "2022-10-19 04:10:55 10.255.255.150:58979 TLS: Username/Password authentication deferred for username 'XXXXXXXXXX' "
2022-10-19T08:10:55+0400 [stdout#info] Warning: OpenVPN daemon job Queue size (~156085) at or above limit (auth.module.max_jobs_in_parallel=64).
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:54545 peer info: IV_VER=3.git::081bfebe'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:54545 peer info: IV_PLAT=ios'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:54545 peer info: IV_NCP=2'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:54545 peer info: IV_TCPNL=1'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:54545 peer info: IV_PROTO=30'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:54545 peer info: IV_CIPHERS=AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305:AES-256-CBC'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:54545 peer info: UV_ASCLI_VER=3.3.2-5086'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:54545 peer info: UV_PLAT_REL=15.6.1'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:54545 peer info: UV_UUID=3F53FE1C-D40C-4AC6-B5A4-C5A7E0004043'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:54545 peer info: IV_GUI_VER=net.openvpn.connect.ios_3.3.2-5086'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:54545 peer info: IV_SSO=webauth,openurl,crtext'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:54545 peer info: IV_HWADDR=3F53FE1C-D40C-4AC6-B5A4-C5A7E0004043'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:54545 peer info: IV_SSL=OpenSSL_1.1.1n__15_Mar_2022'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: "2022-10-19 04:10:55 10.255.255.150:54545 TLS: Username/Password authentication deferred for username 'XXXXXXXXXXX' "
2022-10-19T08:10:55+0400 [stdout#info] Warning: OpenVPN daemon job Queue size (~156086) at or above limit (auth.module.max_jobs_in_parallel=64).
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 0] OUT: "2022-10-19 04:10:55 10.255.255.150:4488 PUSH: Received control message: 'PUSH_REQUEST'"
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 0] OUT: '2022-10-19 04:10:55 10.255.255.150:49291 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 ECDHE-RSA-AES256-GCM-SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 0] OUT: '2022-10-19 04:10:55 10.255.255.150:49291 [XXXXXXXX] Peer Connection Initiated with [AF_INET]10.255.255.150:49291'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 0] OUT: "2022-10-19 04:10:55 10.255.255.150:49291 PUSH: Received control message: 'PUSH_REQUEST'"
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 ECDHE-RSA-AES256-GCM-SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:55 10.255.255.150:58979 [XXXXXXXXX] Peer Connection Initiated with [AF_INET]10.255.255.150:58979 (via [AF_INET]XX.XX.XX.XX%ens224)'
2022-10-19T08:10:55+0400 [stdout#info] [OVPN 4] OUT: "2022-10-19 04:10:55 10.255.255.150:58979 PUSH: Received control message: 'PUSH_REQUEST'"
2022-10-19T08:10:56+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:56 10.255.255.150:54545 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 ECDHE-RSA-AES256-GCM-SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256'
2022-10-19T08:10:56+0400 [stdout#info] [OVPN 4] OUT: '2022-10-19 04:10:56 10.255.255.150:54545 [XXXXXXXX] Peer Connection Initiated with [AF_INET]10.255.255.150:54545 (via [AF_INET]XX.XX.XX.XX%ens224)'
2022-10-19T08:10:56+0400 [stdout#info] [OVPN 4] OUT: "2022-10-19 04:10:56 10.255.255.150:54545 PUSH: Received control message: 'PUSH_REQUEST'"
2022-10-19T08:10:56+0400 [stdout#info] [OVPN 5] OUT: "2022-10-19 04:10:56 10.255.255.150:39174 PUSH: Received control message: 'PUSH_REQUEST'"
2022-10-19T08:10:56+0400 [stdout#info] [OVPN 4] OUT: "2022-10-19 04:10:56 10.255.255.150:47584 PUSH: Received control message: 'PUSH_REQUEST'"
2022-10-19T08:10:56+0400 [stdout#info] [OVPN 4] OUT: "2022-10-19 04:10:56 10.255.255.150:63825 PUSH: Received control message: 'PUSH_REQUEST'"
2022-10-19T08:10:56+0400 [stdout#info] [OVPN 6] OUT: "2022-10-19 04:10:56 10.255.255.150:56525 PUSH: Received control message: 'PUSH_REQUEST'"
2022-10-19T08:10:56+0400 [stdout#info] [OVPN 7] OUT: '2022-10-19 04:10:56 10.255.255.150:61082 SIGTERM[soft,delayed-exit] received, client-instance exiting'
2022-10-19T08:10:56+0400 [stdout#info] Warning: OpenVPN daemon job Queue size (~156087) at or above limit (auth.module.max_jobs_in_parallel=64).
2022-10-19T08:10:56+0400 [stdout#info] [OVPN 4] OUT: "2022-10-19 04:10:56 10.255.255.150:61126 PUSH: Received control message: 'PUSH_REQUEST'"
2022-10-19T08:10:56+0400 [stdout#info] [OVPN 4] OUT: "2022-10-19 04:10:56 10.255.255.150:47726 PUSH: Received control message: 'PUSH_REQUEST'"
As soon as I restarted the server I see high rate of radius request-answers in tcpdump with 15ms answer time. It looks like something was freezed inside the server, without any external signs. Even cpu has low load in this freezed state.