OpenVPN tunnel crashes with OpenWrt

This forum is for general conversation and user-user networking.

Moderators: TinCanTech, TinCanTech, TinCanTech, TinCanTech, TinCanTech, TinCanTech

Post Reply
dmitriyrsl
OpenVpn Newbie
Posts: 8
Joined: Mon Jan 06, 2020 7:47 pm

OpenVPN tunnel crashes with OpenWrt

Post by dmitriyrsl » Mon Jan 06, 2020 8:02 pm

Hi!
I`m using OpenVPN for NVR video from IP cam.
My network diagram:
Image

With a small load tunnel works fine.
But when I open IP cam web interface with video stream, bandwidth reach 3-4MBit/s and after 1-3 minutes tunnel crashes.
I expect stable tunnel from OpenWrt till 8MBit/s, but it crashes on 3-4MBit/s.
If I close IP cam web interface tunnel restores.

My configs:
Server config

port 53
proto udp
dev tun
ca ca.crt
cert server.crt
key server.key
dh dh.pem
topology subnet
server 10.82.0.0 255.255.255.0
ifconfig-pool-persist /var/log/openvpn/ipp.txt
client-config-dir ccd
client-to-client
keepalive 5 20
tls-auth ta.key 0 # This file is secret
cipher AES-256-CBC
persist-key
persist-tun
status /var/log/openvpn/openvpn-status.log
log /var/log/openvpn/openvpn.log
verb 4
explicit-exit-notify 1
auth SHA256
route 10.82.2.0 255.255.255.0 10.82.0.2
route 10.82.3.0 255.255.255.0 10.82.0.3


ccd_microtik

ifconfig-push 10.82.0.3 255.255.255.0
iroute 10.82.3.0 255.255.255.0
push "route 10.82.2.0 255.255.255.0 10.82.0.1"


ccd_xiaomi

ifconfig-push 10.82.0.2 255.255.255.0
iroute 10.82.2.0 255.255.255.0
push "route 10.82.3.0 255.255.255.0 10.82.0.1"


Microtik and Xiaomi

client
dev tun
proto udp
remote <SERVER_IP> 53
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
ca ca.crt
cert client.crt
key client.key
remote-cert-tls server
cipher AES-256-CBC
auth SHA256
verb 3
key-direction 1
sndbuf 0
rcvbuf 0
<ca>
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
</ca>
<cert>
Certificate:
...
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
</cert>
<key>
<tls-auth>
-----BEGIN OpenVPN Static key V1-----
...
-----END OpenVPN Static key V1-----
</tls-auth>


What can be the reason of this crashes? Performance issues on OpenWRT routers on incorrect configuration of OpenVPN?
Do I need more powerful hardware?

User avatar
Pippin
Forum Team
Posts: 1201
Joined: Wed Jul 01, 2015 8:03 am
Location: irc://irc.libera.chat:6697/openvpn

Re: OpenVPN tunnel crashes with OpenWrt

Post by Pippin » Mon Jan 06, 2020 8:16 pm

Please describe "crash" better.
Anything in the logs at verb 4?
Do I need more powerful hardware?
What are the CPU specs of the involved devices?
You sure it's Mbit not Mbyte?
I gloomily came to the ironic conclusion that if you take a highly intelligent person and give them the best possible, elite education, then you will most likely wind up with an academic who is completely impervious to reality.
Halton Arp

dmitriyrsl
OpenVpn Newbie
Posts: 8
Joined: Mon Jan 06, 2020 7:47 pm

Re: OpenVPN tunnel crashes with OpenWrt

Post by dmitriyrsl » Mon Jan 06, 2020 8:32 pm

crash is loss of pings. server log contains lines beginning from:
Mon Jan 6 21:04:04 2020 us=99342 MULTI: multi_create_instance called
and later:
Mon Jan 6 21:04:24 2020 us=75424 xiaomi/<XIAOMI_IP>:49349 [xiaomi] Inactivity timeout (--ping-restart), restarting
Xiaomi Mini WiFi has: MediaTek MT7620A CPU at 580MHz
Mikrotik Hap has: Qualcomm Atheros QCA9531 at 650MHz
Both tunnels can crash. Not always in same time.

Yes I`m sure, It`s MBits. I see it on my Windows 7, System monitor. And H.264 stream from IP cam must be around 2-4MBit.

User avatar
Pippin
Forum Team
Posts: 1201
Joined: Wed Jul 01, 2015 8:03 am
Location: irc://irc.libera.chat:6697/openvpn

Re: OpenVPN tunnel crashes with OpenWrt

Post by Pippin » Mon Jan 06, 2020 9:15 pm

Xiaomi Mini WiFi has: MediaTek MT7620A CPU at 580MHz
Mikrotik Hap has: Qualcomm Atheros QCA9531 at 650MHz
Those do not look so powerful.
1. How about the Debian CPU?

2. Are normal speeds ok, like a file transfer not using VPN? (from NVR to NVR client and vice versa).

3. Does it get better using

Code: Select all

cipher AES-128-CBC
auth SHA1
?
I gloomily came to the ironic conclusion that if you take a highly intelligent person and give them the best possible, elite education, then you will most likely wind up with an academic who is completely impervious to reality.
Halton Arp

dmitriyrsl
OpenVpn Newbie
Posts: 8
Joined: Mon Jan 06, 2020 7:47 pm

Re: OpenVPN tunnel crashes with OpenWrt

Post by dmitriyrsl » Mon Jan 06, 2020 9:27 pm

1 Debian has Intel Pentium 4 at 3GHz.
2 I use RDP through OpenVPN tunnel and it works fine
3 I see test where 128 and 256 bit has small difference for processor load.

TinCanTech
OpenVPN Protagonist
Posts: 11137
Joined: Fri Jun 03, 2016 1:17 pm

Re: OpenVPN tunnel crashes with OpenWrt

Post by TinCanTech » Mon Jan 06, 2020 9:40 pm

dmitriyrsl wrote:
Mon Jan 06, 2020 8:02 pm
Hi!
I`m using OpenVPN for NVR video from IP cam.
My network diagram:
http://joxi.ru/krDpYpRfKzYNar.jpg
Your diagram is 404.
dmitriyrsl wrote:
Mon Jan 06, 2020 8:02 pm
With a small load tunnel works fine.
But when I open IP cam web interface with video stream, bandwidth reach 3-4MBit/s and after 1-3 minutes tunnel crashes.
I expect stable tunnel from OpenWrt till 8MBit/s, but it crashes on 3-4MBit/s.
If I close IP cam web interface tunnel restores.

<Snip>

ccd_microtik

ifconfig-push 10.82.0.3 255.255.255.0
iroute 10.82.3.0 255.255.255.0
push "route 10.82.2.0 255.255.255.0 10.82.0.1"


ccd_xiaomi

ifconfig-push 10.82.0.2 255.255.255.0
iroute 10.82.2.0 255.255.255.0
push "route 10.82.3.0 255.255.255.0 10.82.0.1"


<Snip>

What can be the reason of this crashes? Performance issues on OpenWRT routers on incorrect configuration of OpenVPN?

Do I need more powerful hardware?
Because you have two clients, it looks like your server is trying to VPN a video stream between two different clients .. if so, Yes, you need better hardware ..

dmitriyrsl
OpenVpn Newbie
Posts: 8
Joined: Mon Jan 06, 2020 7:47 pm

Re: OpenVPN tunnel crashes with OpenWrt

Post by dmitriyrsl » Mon Jan 06, 2020 9:47 pm

TinCanTech wrote:
Mon Jan 06, 2020 9:40 pm
dmitriyrsl wrote:
Mon Jan 06, 2020 8:02 pm
Hi!
I`m using OpenVPN for NVR video from IP cam.
My network diagram:
http://joxi.ru/krDpYpRfKzYNar.jpg
Your diagram is 404.

hm.... I check it now, and it opens... Mat be you can open this link?: http://prntscr.com/qk5xrg

dmitriyrsl wrote:
Mon Jan 06, 2020 8:02 pm
With a small load tunnel works fine.
But when I open IP cam web interface with video stream, bandwidth reach 3-4MBit/s and after 1-3 minutes tunnel crashes.
I expect stable tunnel from OpenWrt till 8MBit/s, but it crashes on 3-4MBit/s.
If I close IP cam web interface tunnel restores.

<Snip>

ccd_microtik

ifconfig-push 10.82.0.3 255.255.255.0
iroute 10.82.3.0 255.255.255.0
push "route 10.82.2.0 255.255.255.0 10.82.0.1"


ccd_xiaomi

ifconfig-push 10.82.0.2 255.255.255.0
iroute 10.82.2.0 255.255.255.0
push "route 10.82.3.0 255.255.255.0 10.82.0.1"


<Snip>

What can be the reason of this crashes? Performance issues on OpenWRT routers on incorrect configuration of OpenVPN?

Do I need more powerful hardware?
Because you have two clients, it looks like your server is trying to VPN a video stream between two different clients .. if so, Yes, you need better hardware ..
I started iperf test from xiaomi side to server:
[ 4] 24.00-25.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 25.00-26.00 sec 1.25 MBytes 10.5 Mbits/sec
[ 4] 26.00-27.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 27.00-28.00 sec 896 KBytes 7.34 Mbits/sec
[ 4] 28.00-29.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 29.00-30.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 30.00-31.00 sec 640 KBytes 5.24 Mbits/sec
[ 4] 31.00-32.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 32.00-33.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 33.00-34.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 34.00-35.00 sec 896 KBytes 7.35 Mbits/sec
[ 4] 35.00-36.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 36.00-37.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 37.00-38.00 sec 1.25 MBytes 10.5 Mbits/sec
[ 4] 38.00-39.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 39.00-40.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 40.00-41.00 sec 1.25 MBytes 10.5 Mbits/sec
[ 4] 41.00-42.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 42.00-43.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 43.00-44.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 44.00-45.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 45.00-46.00 sec 1.25 MBytes 10.5 Mbits/sec
[ 4] 46.00-47.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 47.00-48.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 48.00-49.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 49.00-50.00 sec 1.25 MBytes 10.5 Mbits/sec
[ 4] 50.00-51.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 51.00-52.00 sec 1.00 MBytes 8.39 Mbits/sec
[ 4] 52.00-53.00 sec 1.12 MBytes 9.45 Mbits/sec
[ 4] 53.00-54.00 sec 1.12 MBytes 9.44 Mbits/sec
[ 4] 54.00-55.00 sec 640 KBytes 5.24 Mbits/sec
[ 4] 55.00-56.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 56.00-57.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 57.00-58.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 58.00-59.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 59.00-60.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 60.00-61.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 61.00-62.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 62.00-63.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 63.00-64.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 64.00-65.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 65.00-66.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 66.00-67.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 67.00-68.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 68.00-69.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 69.00-70.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 70.00-71.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 71.00-72.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 72.00-73.00 sec 0.00 Bytes 0.00 bits/sec
iperf3: error - unable to write to stream socket: Connection reset by peer
At this test I use only one tunnel

At same time I check load average on xiaomi router and it was in maximum:
23:39:55 up 1 day, 6:12, load average: 0.55, 0.17, 0.06
something strange....

User avatar
Pippin
Forum Team
Posts: 1201
Joined: Wed Jul 01, 2015 8:03 am
Location: irc://irc.libera.chat:6697/openvpn

Re: OpenVPN tunnel crashes with OpenWrt

Post by Pippin » Mon Jan 06, 2020 10:09 pm

Would still try this combination to see if it gets better:

Code: Select all

cipher AES-128-CBC
auth SHA1[code]
Hashing (auth) can have quite an impact on CPU...
I gloomily came to the ironic conclusion that if you take a highly intelligent person and give them the best possible, elite education, then you will most likely wind up with an academic who is completely impervious to reality.
Halton Arp

TinCanTech
OpenVPN Protagonist
Posts: 11137
Joined: Fri Jun 03, 2016 1:17 pm

Re: OpenVPN tunnel crashes with OpenWrt

Post by TinCanTech » Mon Jan 06, 2020 10:43 pm

Technically, AES-256-GCM is better .. hash and cipher in one pass, as I understand it.
Note: NCP ciphers is in effect here .. ;)

but --auth SHA1 is a decent proposal .. not enough for video though ..

User avatar
Pippin
Forum Team
Posts: 1201
Joined: Wed Jul 01, 2015 8:03 am
Location: irc://irc.libera.chat:6697/openvpn

Re: OpenVPN tunnel crashes with OpenWrt

Post by Pippin » Mon Jan 06, 2020 11:24 pm

OpenWrt, did not think about that one, NCP.
not enough for video though
With regards too...security or speed?
I gloomily came to the ironic conclusion that if you take a highly intelligent person and give them the best possible, elite education, then you will most likely wind up with an academic who is completely impervious to reality.
Halton Arp

TinCanTech
OpenVPN Protagonist
Posts: 11137
Joined: Fri Jun 03, 2016 1:17 pm

Re: OpenVPN tunnel crashes with OpenWrt

Post by TinCanTech » Mon Jan 06, 2020 11:52 pm

Pippin wrote:
Mon Jan 06, 2020 11:24 pm
OpenWrt, did not think about that one, NCP.
not enough for video though
With regards too...security or speed?
Speed ..

You saw the new diagram upload, right ? :ugeek:
http://prntscr.com/qk5xrg

dmitriyrsl
OpenVpn Newbie
Posts: 8
Joined: Mon Jan 06, 2020 7:47 pm

Re: OpenVPN tunnel crashes with OpenWrt

Post by dmitriyrsl » Tue Jan 07, 2020 6:59 am

I tested:
Auth test
auth SHA1
cypher AES-128-CBC


iperf output:

Code: Select all

[  4]  61.00-62.00  sec  1.12 MBytes  9.44 Mbits/sec
[  4]  62.00-63.00  sec  1.25 MBytes  10.5 Mbits/sec
[  4]  63.00-64.00  sec  1.00 MBytes  8.39 Mbits/sec
[  4]  64.00-65.00  sec  1.25 MBytes  10.5 Mbits/sec
[  4]  65.00-66.00  sec  1.12 MBytes  9.45 Mbits/sec
[  4]  66.00-67.00  sec  1.25 MBytes  10.5 Mbits/sec
[  4]  67.00-68.00  sec  1.12 MBytes  9.44 Mbits/sec
[  4]  68.00-69.00  sec  1.25 MBytes  10.5 Mbits/sec
[  4]  69.00-70.00  sec  1.12 MBytes  9.44 Mbits/sec
[  4]  70.00-71.00  sec  1.25 MBytes  10.5 Mbits/sec
[  4]  71.00-72.00  sec  1.12 MBytes  9.44 Mbits/sec
[  4]  72.00-73.00  sec  1.25 MBytes  10.5 Mbits/sec
[  4]  73.00-74.00  sec   512 KBytes  4.19 Mbits/sec
[  4]  74.00-75.00  sec  0.00 Bytes  0.00 bits/sec
[  4]  75.00-76.00  sec  0.00 Bytes  0.00 bits/sec
[  4]  76.00-77.00  sec  0.00 Bytes  0.00 bits/sec
[  4]  77.00-78.04  sec  0.00 Bytes  0.00 bits/sec
this is top output on Debian server:
https://prnt.sc/qkadtt

this is top output on Xiaomi Mini Wifi:
http://prntscr.com/qkaeei

As you can see, nothing changes. tunnel still crashes after 70 seconds.
top output on xiaomi shows processor load around 100% but Debian server load only on 7%.

Its all? Only hardware change on client side can help?

TinCanTech
OpenVPN Protagonist
Posts: 11137
Joined: Fri Jun 03, 2016 1:17 pm

Re: OpenVPN tunnel crashes with OpenWrt

Post by TinCanTech » Tue Jan 07, 2020 11:30 am

To prove openvpn is not "crashing" please see your logs at verb 4, Post them if you like.

The problem is most likely a bottleneck at your server internet link.

dmitriyrsl
OpenVpn Newbie
Posts: 8
Joined: Mon Jan 06, 2020 7:47 pm

Re: OpenVPN tunnel crashes with OpenWrt

Post by dmitriyrsl » Tue Jan 07, 2020 11:47 am

TinCanTech wrote:
Tue Jan 07, 2020 11:30 am
To prove openvpn is not "crashing" please see your logs at verb 4, Post them if you like.

The problem is most likely a bottleneck at your server internet link.
server logs at verb 4 I posted above. Client logs on OpenWRT in system log:

Code: Select all

Tue Jan  7 13:41:44 2020 daemon.notice openvpn(xiaomi)[4359]: [openvpn] Inactivity timeout (--ping-restart), restarting
Tue Jan  7 13:41:44 2020 daemon.notice openvpn(xiaomi)[4359]: TCP/UDP: Closing socket
Tue Jan  7 13:41:44 2020 daemon.notice openvpn(xiaomi)[4359]: SIGUSR1[soft,ping-restart] received, process restarting
Tue Jan  7 13:41:44 2020 daemon.notice openvpn(xiaomi)[4359]: Restart pause, 5 second(s)
Tue Jan  7 13:41:49 2020 daemon.notice openvpn(xiaomi)[4359]: Re-using SSL/TLS context
Internet connection on both side 50MBit and higher.

TinCanTech
OpenVPN Protagonist
Posts: 11137
Joined: Fri Jun 03, 2016 1:17 pm

Re: OpenVPN tunnel crashes with OpenWrt

Post by TinCanTech » Tue Jan 07, 2020 11:55 am

The tunnel is not crashing but your internet connection is .. hence the --ping timeout.

I doubt it will help but you could try "keepalive 30 120" or something like that ..

dmitriyrsl
OpenVpn Newbie
Posts: 8
Joined: Mon Jan 06, 2020 7:47 pm

Re: OpenVPN tunnel crashes with OpenWrt

Post by dmitriyrsl » Tue Jan 07, 2020 12:01 pm

TinCanTech wrote:
Tue Jan 07, 2020 11:55 am
The tunnel is not crashing but your internet connection is .. hence the --ping timeout.

I doubt it will help but you could try "keepalive 30 120" or something like that ..
Router heavy loaded and doesn`t have time to process WAN connection? May be I can limit processor usage by openvpn client? In this case tunnel will have less speed but it will don`t crash. Is it posible?

TinCanTech
OpenVPN Protagonist
Posts: 11137
Joined: Fri Jun 03, 2016 1:17 pm

Re: OpenVPN tunnel crashes with OpenWrt

Post by TinCanTech » Tue Jan 07, 2020 12:24 pm

The fact is your server WAN is saturated.

Try lowering the resolution of your IP Cam feed.

dmitriyrsl
OpenVpn Newbie
Posts: 8
Joined: Mon Jan 06, 2020 7:47 pm

Re: OpenVPN tunnel crashes with OpenWrt

Post by dmitriyrsl » Tue Jan 07, 2020 2:46 pm

TinCanTech wrote:
Tue Jan 07, 2020 12:24 pm
The fact is your server WAN is saturated.

Try lowering the resolution of your IP Cam feed.
Currently i test tunnel using iperf, without ip cam traffic. But in future I need to transfer traffic from 4 ip cams.
When I add to client config:

Nice

nice 10


Time from begin of iperf test to crash, grown from 60-70sec to 90-95sec.
During test cpu usage by openvpn was at maximum 89%. But despite this, tunnel crashed.
Maybe it's not CPU loading, but because some network queue is overloaded?

My client config already has sndbuf and rcvbuf options equal to 0, may be i need to change something more?

thompsonmax
OpenVpn Newbie
Posts: 2
Joined: Fri Jan 24, 2020 1:35 pm

Re: OpenVPN tunnel crashes with OpenWrt

Post by thompsonmax » Fri Jan 24, 2020 1:41 pm

Hello everyone. I am new here. Interesting thread, thanks for the information :?

Post Reply