Page 1 of 1

OpenVPN tunnel crashes with OpenWrt

Posted: Mon Jan 06, 2020 8:02 pm
by dmitriyrsl
Hi!
I`m using OpenVPN for NVR video from IP cam.
My network diagram:
Image

With a small load tunnel works fine.
But when I open IP cam web interface with video stream, bandwidth reach 3-4MBit/s and after 1-3 minutes tunnel crashes.
I expect stable tunnel from OpenWrt till 8MBit/s, but it crashes on 3-4MBit/s.
If I close IP cam web interface tunnel restores.

My configs:
Server config

port 53
proto udp
dev tun
ca ca.crt
cert server.crt
key server.key
dh dh.pem
topology subnet
server 10.82.0.0 255.255.255.0
ifconfig-pool-persist /var/log/openvpn/ipp.txt
client-config-dir ccd
client-to-client
keepalive 5 20
tls-auth ta.key 0 # This file is secret
cipher AES-256-CBC
persist-key
persist-tun
status /var/log/openvpn/openvpn-status.log
log /var/log/openvpn/openvpn.log
verb 4
explicit-exit-notify 1
auth SHA256
route 10.82.2.0 255.255.255.0 10.82.0.2
route 10.82.3.0 255.255.255.0 10.82.0.3


ccd_microtik

ifconfig-push 10.82.0.3 255.255.255.0
iroute 10.82.3.0 255.255.255.0
push "route 10.82.2.0 255.255.255.0 10.82.0.1"


ccd_xiaomi

ifconfig-push 10.82.0.2 255.255.255.0
iroute 10.82.2.0 255.255.255.0
push "route 10.82.3.0 255.255.255.0 10.82.0.1"


Microtik and Xiaomi

client
dev tun
proto udp
remote <SERVER_IP> 53
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
ca ca.crt
cert client.crt
key client.key
remote-cert-tls server
cipher AES-256-CBC
auth SHA256
verb 3
key-direction 1
sndbuf 0
rcvbuf 0
<ca>
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
</ca>
<cert>
Certificate:
...
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
</cert>
<key>
<tls-auth>
-----BEGIN OpenVPN Static key V1-----
...
-----END OpenVPN Static key V1-----
</tls-auth>


What can be the reason of this crashes? Performance issues on OpenWRT routers on incorrect configuration of OpenVPN?
Do I need more powerful hardware?

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Mon Jan 06, 2020 8:16 pm
by Pippin
Please describe "crash" better.
Anything in the logs at verb 4?
Do I need more powerful hardware?
What are the CPU specs of the involved devices?
You sure it's Mbit not Mbyte?

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Mon Jan 06, 2020 8:32 pm
by dmitriyrsl
crash is loss of pings. server log contains lines beginning from:
Mon Jan 6 21:04:04 2020 us=99342 MULTI: multi_create_instance called
and later:
Mon Jan 6 21:04:24 2020 us=75424 xiaomi/<XIAOMI_IP>:49349 [xiaomi] Inactivity timeout (--ping-restart), restarting
Xiaomi Mini WiFi has: MediaTek MT7620A CPU at 580MHz
Mikrotik Hap has: Qualcomm Atheros QCA9531 at 650MHz
Both tunnels can crash. Not always in same time.

Yes I`m sure, It`s MBits. I see it on my Windows 7, System monitor. And H.264 stream from IP cam must be around 2-4MBit.

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Mon Jan 06, 2020 9:15 pm
by Pippin
Xiaomi Mini WiFi has: MediaTek MT7620A CPU at 580MHz
Mikrotik Hap has: Qualcomm Atheros QCA9531 at 650MHz
Those do not look so powerful.
1. How about the Debian CPU?

2. Are normal speeds ok, like a file transfer not using VPN? (from NVR to NVR client and vice versa).

3. Does it get better using

Code: Select all

cipher AES-128-CBC
auth SHA1
?

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Mon Jan 06, 2020 9:27 pm
by dmitriyrsl
1 Debian has Intel Pentium 4 at 3GHz.
2 I use RDP through OpenVPN tunnel and it works fine
3 I see test where 128 and 256 bit has small difference for processor load.

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Mon Jan 06, 2020 9:40 pm
by TinCanTech
dmitriyrsl wrote:
Mon Jan 06, 2020 8:02 pm
Hi!
I`m using OpenVPN for NVR video from IP cam.
My network diagram:
http://joxi.ru/krDpYpRfKzYNar.jpg
Your diagram is 404.
dmitriyrsl wrote:
Mon Jan 06, 2020 8:02 pm
With a small load tunnel works fine.
But when I open IP cam web interface with video stream, bandwidth reach 3-4MBit/s and after 1-3 minutes tunnel crashes.
I expect stable tunnel from OpenWrt till 8MBit/s, but it crashes on 3-4MBit/s.
If I close IP cam web interface tunnel restores.

<Snip>

ccd_microtik

ifconfig-push 10.82.0.3 255.255.255.0
iroute 10.82.3.0 255.255.255.0
push "route 10.82.2.0 255.255.255.0 10.82.0.1"


ccd_xiaomi

ifconfig-push 10.82.0.2 255.255.255.0
iroute 10.82.2.0 255.255.255.0
push "route 10.82.3.0 255.255.255.0 10.82.0.1"


<Snip>

What can be the reason of this crashes? Performance issues on OpenWRT routers on incorrect configuration of OpenVPN?

Do I need more powerful hardware?
Because you have two clients, it looks like your server is trying to VPN a video stream between two different clients .. if so, Yes, you need better hardware ..

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Mon Jan 06, 2020 9:47 pm
by dmitriyrsl
TinCanTech wrote:
Mon Jan 06, 2020 9:40 pm
dmitriyrsl wrote:
Mon Jan 06, 2020 8:02 pm
Hi!
I`m using OpenVPN for NVR video from IP cam.
My network diagram:
http://joxi.ru/krDpYpRfKzYNar.jpg
Your diagram is 404.

hm.... I check it now, and it opens... Mat be you can open this link?: http://prntscr.com/qk5xrg

dmitriyrsl wrote:
Mon Jan 06, 2020 8:02 pm
With a small load tunnel works fine.
But when I open IP cam web interface with video stream, bandwidth reach 3-4MBit/s and after 1-3 minutes tunnel crashes.
I expect stable tunnel from OpenWrt till 8MBit/s, but it crashes on 3-4MBit/s.
If I close IP cam web interface tunnel restores.

<Snip>

ccd_microtik

ifconfig-push 10.82.0.3 255.255.255.0
iroute 10.82.3.0 255.255.255.0
push "route 10.82.2.0 255.255.255.0 10.82.0.1"


ccd_xiaomi

ifconfig-push 10.82.0.2 255.255.255.0
iroute 10.82.2.0 255.255.255.0
push "route 10.82.3.0 255.255.255.0 10.82.0.1"


<Snip>

What can be the reason of this crashes? Performance issues on OpenWRT routers on incorrect configuration of OpenVPN?

Do I need more powerful hardware?
Because you have two clients, it looks like your server is trying to VPN a video stream between two different clients .. if so, Yes, you need better hardware ..
I started iperf test from xiaomi side to server:
[ 4] 24.00-25.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 25.00-26.00 sec 1.25 MBytes 10.5 Mbits/sec
[ 4] 26.00-27.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 27.00-28.00 sec 896 KBytes 7.34 Mbits/sec
[ 4] 28.00-29.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 29.00-30.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 30.00-31.00 sec 640 KBytes 5.24 Mbits/sec
[ 4] 31.00-32.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 32.00-33.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 33.00-34.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 34.00-35.00 sec 896 KBytes 7.35 Mbits/sec
[ 4] 35.00-36.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 36.00-37.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 37.00-38.00 sec 1.25 MBytes 10.5 Mbits/sec
[ 4] 38.00-39.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 39.00-40.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 40.00-41.00 sec 1.25 MBytes 10.5 Mbits/sec
[ 4] 41.00-42.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 42.00-43.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 43.00-44.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 44.00-45.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 45.00-46.00 sec 1.25 MBytes 10.5 Mbits/sec
[ 4] 46.00-47.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 47.00-48.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 48.00-49.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 49.00-50.00 sec 1.25 MBytes 10.5 Mbits/sec
[ 4] 50.00-51.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 51.00-52.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 52.00-53.00 sec 1.12 MBytes 9.45 Mbits/sec
[ 4] 53.00-54.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 54.00-55.00 sec 640 KBytes 5.24 Mbits/sec
[ 4] 55.00-56.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 56.00-57.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 57.00-58.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 58.00-59.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 59.00-60.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 60.00-61.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 61.00-62.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 62.00-63.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 63.00-64.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 64.00-65.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 65.00-66.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 66.00-67.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 67.00-68.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 68.00-69.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 69.00-70.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 70.00-71.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 71.00-72.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 72.00-73.00 sec 0.00 Bytes 0.00 bits/sec
iperf3: error - unable to write to stream socket: Connection reset by peer
At this test I use only one tunnel

At same time I check load average on xiaomi router and it was in maximum:
23:39:55 up 1 day, 6:12, load average: 0.55, 0.17, 0.06
something strange....

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Mon Jan 06, 2020 10:09 pm
by Pippin
Would still try this combination to see if it gets better:

Code: Select all

cipher AES-128-CBC
auth SHA1[code]
Hashing (auth) can have quite an impact on CPU...

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Mon Jan 06, 2020 10:43 pm
by TinCanTech
Technically, AES-256-GCM is better .. hash and cipher in one pass, as I understand it.
Note: NCP ciphers is in effect here .. ;)

but --auth SHA1 is a decent proposal .. not enough for video though ..

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Mon Jan 06, 2020 11:24 pm
by Pippin
OpenWrt, did not think about that one, NCP.
not enough for video though
With regards too...security or speed?

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Mon Jan 06, 2020 11:52 pm
by TinCanTech
Pippin wrote:
Mon Jan 06, 2020 11:24 pm
OpenWrt, did not think about that one, NCP.
not enough for video though
With regards too...security or speed?
Speed ..

You saw the new diagram upload, right ? :ugeek:
http://prntscr.com/qk5xrg

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Tue Jan 07, 2020 6:59 am
by dmitriyrsl
I tested:
Auth test
auth SHA1
cypher AES-128-CBC


iperf output:

Code: Select all

[  4]  61.00-62.00  sec  1.12 MBytes  9.44 Mbits/sec
[  4]  62.00-63.00  sec  1.25 MBytes  10.5 Mbits/sec
[  4]  63.00-64.00  sec  1.00 MBytes  8.39 Mbits/sec
[  4]  64.00-65.00  sec  1.25 MBytes  10.5 Mbits/sec
[  4]  65.00-66.00  sec  1.12 MBytes  9.45 Mbits/sec
[  4]  66.00-67.00  sec  1.25 MBytes  10.5 Mbits/sec
[  4]  67.00-68.00  sec  1.12 MBytes  9.44 Mbits/sec
[  4]  68.00-69.00  sec  1.25 MBytes  10.5 Mbits/sec
[  4]  69.00-70.00  sec  1.12 MBytes  9.44 Mbits/sec
[  4]  70.00-71.00  sec  1.25 MBytes  10.5 Mbits/sec
[  4]  71.00-72.00  sec  1.12 MBytes  9.44 Mbits/sec
[  4]  72.00-73.00  sec  1.25 MBytes  10.5 Mbits/sec
[  4]  73.00-74.00  sec   512 KBytes  4.19 Mbits/sec
[  4]  74.00-75.00  sec  0.00 Bytes  0.00 bits/sec
[  4]  75.00-76.00  sec  0.00 Bytes  0.00 bits/sec
[  4]  76.00-77.00  sec  0.00 Bytes  0.00 bits/sec
[  4]  77.00-78.04  sec  0.00 Bytes  0.00 bits/sec
this is top output on Debian server:
https://prnt.sc/qkadtt

this is top output on Xiaomi Mini Wifi:
http://prntscr.com/qkaeei

As you can see, nothing changes. tunnel still crashes after 70 seconds.
top output on xiaomi shows processor load around 100% but Debian server load only on 7%.

Its all? Only hardware change on client side can help?

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Tue Jan 07, 2020 11:30 am
by TinCanTech
To prove openvpn is not "crashing" please see your logs at verb 4, Post them if you like.

The problem is most likely a bottleneck at your server internet link.

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Tue Jan 07, 2020 11:47 am
by dmitriyrsl
TinCanTech wrote:
Tue Jan 07, 2020 11:30 am
To prove openvpn is not "crashing" please see your logs at verb 4, Post them if you like.

The problem is most likely a bottleneck at your server internet link.
server logs at verb 4 I posted above. Client logs on OpenWRT in system log:

Code: Select all

Tue Jan  7 13:41:44 2020 daemon.notice openvpn(xiaomi)[4359]: [openvpn] Inactivity timeout (--ping-restart), restarting
Tue Jan  7 13:41:44 2020 daemon.notice openvpn(xiaomi)[4359]: TCP/UDP: Closing socket
Tue Jan  7 13:41:44 2020 daemon.notice openvpn(xiaomi)[4359]: SIGUSR1[soft,ping-restart] received, process restarting
Tue Jan  7 13:41:44 2020 daemon.notice openvpn(xiaomi)[4359]: Restart pause, 5 second(s)
Tue Jan  7 13:41:49 2020 daemon.notice openvpn(xiaomi)[4359]: Re-using SSL/TLS context
Internet connection on both side 50MBit and higher.

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Tue Jan 07, 2020 11:55 am
by TinCanTech
The tunnel is not crashing but your internet connection is .. hence the --ping timeout.

I doubt it will help but you could try "keepalive 30 120" or something like that ..

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Tue Jan 07, 2020 12:01 pm
by dmitriyrsl
TinCanTech wrote:
Tue Jan 07, 2020 11:55 am
The tunnel is not crashing but your internet connection is .. hence the --ping timeout.

I doubt it will help but you could try "keepalive 30 120" or something like that ..
Router heavy loaded and doesn`t have time to process WAN connection? May be I can limit processor usage by openvpn client? In this case tunnel will have less speed but it will don`t crash. Is it posible?

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Tue Jan 07, 2020 12:24 pm
by TinCanTech
The fact is your server WAN is saturated.

Try lowering the resolution of your IP Cam feed.

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Tue Jan 07, 2020 2:46 pm
by dmitriyrsl
TinCanTech wrote:
Tue Jan 07, 2020 12:24 pm
The fact is your server WAN is saturated.

Try lowering the resolution of your IP Cam feed.
Currently i test tunnel using iperf, without ip cam traffic. But in future I need to transfer traffic from 4 ip cams.
When I add to client config:

Nice

nice 10


Time from begin of iperf test to crash, grown from 60-70sec to 90-95sec.
During test cpu usage by openvpn was at maximum 89%. But despite this, tunnel crashed.
Maybe it's not CPU loading, but because some network queue is overloaded?

My client config already has sndbuf and rcvbuf options equal to 0, may be i need to change something more?

Re: OpenVPN tunnel crashes with OpenWrt

Posted: Fri Jan 24, 2020 1:41 pm
by thompsonmax
Hello everyone. I am new here. Interesting thread, thanks for the information :?