OpenVPN Tap Bridge Crashes ASUS RT-AX86U Router

AndyL733

Hello,

I hope somebody can help here. Thank you in advance! I have also reached out to ASUS support (useless) as well as the asuswrt-merlin project (they're trying but I don't think they really understand the issue).

For several years, I have successfully run an Ethernet Bridge over OpenVPN between two locations -- one in Boston, USA and the other in Madrid, Spain. I need the Layer 2 broadcast support, which is why I have this set up. My router in Boston is the RT-AC86U currently running OpenVPN 2.6.0 with ASUSWRT-MERLIN FW 386.10, and my router in Madrid is the RT-AC68U running OpenVPN 2.6.3 with ASUSWRT-MERLIN FW 386.11.

I am trying to update my Madrid router to the much newer and more powerful ASUS RT-AX86U PRO but the same configuration that works on the older RT-AC68U crashes the RT-AX86U PRO. I get a kernel panic just after the tap device is opened and the OpenVPN initialization is completed. For what it's worth, in the RT-AX86U PRO vpn web interface, just after the OpenVPN client has started there is a warning saying VPN failed and "routing conflict".

The way I have my VPN set up, both sides are on the same subnet -- 192.168.15.x 255.255.255.0. DHCP servers on each side give out addresses -- on the Boston side from 100-179 and on the Madrid side from 180-254. I have a script running that, via edtables, blocks each side from seeing the other side's DHCP server, and with the working setup Nmap confirms that the ports are blocked correctly.

I don't understand why the Bridge CLIENT configuration works with the old router and not with the new one. With the new router, I have the exact same issue whether I run the stock ASUS FW (which has OpenVPN 2.4.12), or whether I upgrade it to the ASUSWRT-MERLIN FW (which has OpenVPN 2.6.3). Several times, by tailing the syslog over SSH, I have captured part of the kernel panic -- but never the whole dump. This is what it looks like with the stock ASUS FW:

May 27 03:52:33 vpnclient5[8521]: OpenVPN 2.4.12 arm-buildroot-linux-gnueabi [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Oct 12 2022
May 27 03:52:33 vpnclient5[8521]: library versions: OpenSSL 1.1.1n 15 Mar 2022, LZO 2.10
May 27 03:52:33 vpnclient5[8522]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
May 27 03:52:33 vpnclient5[8522]: TCP/UDP: Preserving recently used remote address: [AF_INET](My remote IP address -- replace by text for security):1194
May 27 03:52:33 vpnclient5[8522]: Socket Buffers: R=[524288->524288] S=[524288->524288]
May 27 03:52:33 vpnclient5[8522]: UDP link local: (not bound)
May 27 03:52:33 vpnclient5[8522]: UDP link remote: [AF_INET](My remote IP address -- replace by text for security):1194
May 27 03:52:33 vpnclient5[8522]: TLS: Initial packet from [AF_INET](My remote IP address -- replace by text for security):1194, sid=141f2619 fa8bcb51
May 27 03:52:33 vpnclient5[8522]: VERIFY OK: depth=1, C=TW, ST=TW, L=Taipei, O=ASUS, CN=RT-AC86U, emailAddress=me@myhost.mydomain
May 27 03:52:33 vpnclient5[8522]: VERIFY KU OK
May 27 03:52:33 vpnclient5[8522]: Validating certificate extended key usage
May 27 03:52:33 vpnclient5[8522]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
May 27 03:52:33 vpnclient5[8522]: VERIFY EKU OK
May 27 03:52:33 vpnclient5[8522]: VERIFY OK: depth=0, C=TW, ST=TW, L=Taipei, O=ASUS, CN=RT-AC86U, emailAddress=me@myhost.mydomain
May 27 03:52:34 vpnclient5[8522]: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1586', remote='link-mtu 1602'
May 27 03:52:34 vpnclient5[8522]: WARNING: 'tun-mtu' is used inconsistently, local='tun-mtu 1532', remote='tun-mtu 1500'
May 27 03:52:34 vpnclient5[8522]: WARNING: 'cipher' is used inconsistently, local='cipher BF-CBC', remote='cipher AES-128-CBC'
May 27 03:52:34 vpnclient5[8522]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 1024 bit RSA
May 27 03:52:34 vpnclient5[8522]: [RT-AC86U] Peer Connection Initiated with [AF_INET](My remote IP address -- deleted for privacy):1194
May 27 03:52:35 vpnclient5[8522]: SENT CONTROL [RT-AC86U]: 'PUSH_REQUEST' (status=1)
May 27 03:52:35 vpnclient5[8522]: PUSH: Received control message: 'PUSH_REPLY,route 0.0.0.0 255.255.255.255 net_gateway,route-gateway dhcp,ping 15,ping-restart 60,peer-id 0,cipher AES-256-GCM'
May 27 03:52:35 vpnclient5[8522]: OPTIONS IMPORT: timers and/or timeouts modified
May 27 03:52:35 vpnclient5[8522]: OPTIONS IMPORT: route options modified
May 27 03:52:35 vpnclient5[8522]: OPTIONS IMPORT: route-related options modified
May 27 03:52:35 vpnclient5[8522]: OPTIONS IMPORT: peer-id set
May 27 03:52:35 vpnclient5[8522]: OPTIONS IMPORT: adjusting link_mtu to 1657
May 27 03:52:35 vpnclient5[8522]: OPTIONS IMPORT: data channel crypto options modified
May 27 03:52:35 vpnclient5[8522]: Data Channel: using negotiated cipher 'AES-256-GCM'
May 27 03:52:35 vpnclient5[8522]: Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
May 27 03:52:35 vpnclient5[8522]: Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
May 27 03:52:35 vpnclient5[8522]: TUN/TAP device tap15 opened
May 27 03:52:35 vpnclient5[8522]: TUN/TAP TX queue length set to 100
May 27 03:52:35 vpnclient5[8522]: /etc/openvpn/ovpnc-up 5 tap15 1500 1585 init
May 27 03:52:35 vpnclient5: WARNING: Replace default vpn gateway by using 0.0.0.0/1 and 128.0.0.0/1
May 27 03:52:35 vpnclient5[8522]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
May 27 03:52:35 vpnclient5[8522]: Initialization Sequence Completed
May 27 03:52:47 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000000002b8
May 27 03:52:47 kernel: Mem abort info:
May 27 03:52:47 kernel: ESR = 0x96000007
May 27 03:52:47 kernel: Exception class = DABT (current EL), IL = 32 bits
May 27 03:52:47 kernel: SET = 0, FnV = 0
May 27 03:52:47 kernel: EA = 0, S1PTW = 0
May 27 03:52:47 kernel: Data abort info:
May 27 03:52:47 kernel: ISV = 0, ISS = 0x00000007
May 27 03:52:47 kernel: CM = 0, WnR = 0
May 27 03:52:47 kernel: user pgtable: 4k pages, 39-bit VAs, pgdp = 000000000963c35a
May 27 03:52:47 kernel: [00000000000002b8] pgd=000000003390f003, pud=000000003390f003, pmd=000000002171a003, pte=0000000000000000

The ovpn client configuration file was generated by my router in Boston. For the sake of this post, I have removed the certificates and any private information. Is there anything wrong with the configuration?

# Config generated by Asuswrt-Merlin 386.10, requires OpenVPN 2.4.0 or newer.client

dev tap
# Windows needs the TAP-Win32 adapter name
# from the Network Connections panel
# if you have more than one.
;dev-node MyTap
proto udp
remote (My remote DYNDNS Host Name, deleted for privacy) 1194
resolv-retry infinite
nobind
float
ncp-ciphers AES-256-GCM:AES-128-GCM:AES-256-CBC:AES-128-CBC
auth SHA256
comp-lzo adaptive
keepalive 15 60
remote-cert-tls server

Do you have any ideas what might be causing the kernel panic and how to solve it?

Thank you for reading this and helping! (And sorry if I didn't format this post correctly. I tried to use the oconf= option but it the preview it didn't seem to work)
Andy

AndyL733

For what it's worth:

I created a Server bridge on the RT-AX86U Pro Spain side just to see if the tap driver itself crashed the kernel. It did NOT. I also updated my Boston router to asuswrt-merlin 386.11 so it is running OpenVPN 2.6.3. The client OVPN exported by the Boston router still causes the Spain router to kernel panic.

I guess my final step will be to try to reverse the roles of the two routers and see if that works (given that the server seems to run okay on the RT-AX86U Pro side).

AndyL733

I guess my final step will be to try to reverse the roles of the two routers and see if that works (given that the server seems to run okay on the RT-AX86U Pro side).

Okay, so I tried this experiment. I enabled OpenVPN server on the Spain side (on the RT-AX86U Pro) and exported the client OVPN config file. Then I imported it on the Boston side (on the RT-AC86U) after shutting off the OpenVPN server on that router. As soon as I tried to connect to Spain, the RT-AX86U Pro (the new router in Spain) crashed with the kernel panic. So in other words, the RT-AC86U Pro crashes whether it is the bridge server or the bridge client.

So, I think this lends further evidence to the following idea -- that USING the tap adapter on the RT-AX86U Pro (sending data over the tap interface as opposed to just setting up the Server with it) causes the kernel panic.

Here is the complete log from the Spain RT-AX86U Pro when it is the bridge server rather than the bridge client:

admin@RT-AX86U_Pro:/tmp/home/root# tail -f /jffs/syslog.log
May 27 13:15:54 dropbear[5150]: Password auth succeeded for 'admin' from 192.168.15.211:56178
May 27 13:16:12 rc_service: httpd 2448:notify_rc restart_chpass;restart_vpnserver1
May 27 13:16:13 ovpn-server1[5264]: OpenVPN 2.6.3 arm-buildroot-linux-gnueabi [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
May 27 13:16:13 ovpn-server1[5264]: library versions: OpenSSL 1.1.1t 7 Feb 2023, LZO 2.10
May 27 13:16:13 ovpn-server1[5265]: NOTE: when bridging your LAN adapter with the TAP adapter, note that the new bridge adapter will often take on its own IP address that is different from what the LAN adapter was previously set to
May 27 13:16:13 ovpn-server1[5265]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
May 27 13:16:13 ovpn-server1[5265]: Diffie-Hellman initialized with 2048 bit key
May 27 13:16:13 ovpn-server1[5265]: TUN/TAP device tap21 opened
May 27 13:16:13 ovpn-server1[5265]: TUN/TAP TX queue length set to 1000
May 27 13:16:13 ovpn-server1[5265]: ovpn-up 1 server tap21 1500 0 init
May 27 13:16:13 ovpn-server1[5265]: Socket Buffers: R=[524288->524288] S=[524288->524288]
May 27 13:16:13 ovpn-server1[5265]: UDPv4 link local (bound): [AF_INET][undef]:1194
May 27 13:16:13 ovpn-server1[5265]: UDPv4 link remote: [AF_UNSPEC]
May 27 13:16:13 ovpn-server1[5265]: MULTI: multi_init called, r=256 v=256
May 27 13:16:13 ovpn-server1[5265]: Initialization Sequence Completed
May 27 13:23:27 wlceventd: wlceventd_proc_event(530): eth6: Auth 84:F3:EB:86:88:FE, status: Successful (0), rssi:0
May 27 13:23:27 wlceventd: wlceventd_proc_event(559): eth6: Assoc 84:F3:EB:86:88:FE, status: Successful (0), rssi:-45
May 27 13:23:32 wlceventd: wlceventd_proc_event(511): eth6: Disassoc 84:F3:EB:86:88:FE, status: 0, reason: Disassociated because sending station is leaving (or has left) BSS (8), rssi:0
May 27 13:23:32 wlceventd: wlceventd_proc_event(494): eth6: Deauth_ind 84:F3:EB:86:88:FE, status: 0, reason: Deauthenticated because sending station is leaving (or has left) IBSS or ESS (3), rssi:0
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 VERIFY OK: depth=1, C=TW, ST=TW, L=Taipei, O=ASUS, CN=RT-AX86U_Pro, emailAddress=me@myhost.mydomain
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 VERIFY OK: depth=0, C=TW, ST=TW, L=Taipei, O=ASUS, CN=client, emailAddress=me@myhost.mydomain
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_VER=2.6.3
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_PLAT=linux
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_TCPNL=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_MTU=1600
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_NCP=2
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_CIPHERS=AES-256-GCM:AES-128-GCM:AES-256-CBC:AES-128-CBC:CHACHA20-POLY1305
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_PROTO=990
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_LZO_STUB=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_COMP_STUB=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_COMP_STUBv2=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 TLS: tls_multi_process: initial untrusted session promoted to trusted
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 1024 bit RSA, signature: RSA-SHA256
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 [client] Peer Connection Initiated with [AF_INET](Boston client IP -- removed for privacy):38229 (via [AF_INET]88.1.204.132%ppp0)
May 27 13:24:26 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: no dynamic or static remote--ifconfig address is available for client/(Boston client IP -- removed for privacy):38229
May 27 13:24:26 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 SENT CONTROL [client]: 'PUSH_REPLY,route 0.0.0.0 255.255.255.255 net_gateway,route-gateway dhcp,ping 15,ping-restart 60,peer-id 0,cipher AES-256-GCM,protocol-flags cc-exit tls-ekm dyn-tls-crypt,tun-mtu 1500' (status=1)
May 27 13:24:26 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 4a:b1:c3:00:a3:5f@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:27 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 Data Channel: cipher 'AES-256-GCM', peer-id: 0
May 27 13:24:27 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 Timers: ping 15, ping-restart 120
May 27 13:24:27 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 Protocol options: protocol-flags cc-exit tls-ekm dyn-tls-crypt
May 27 13:24:31 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 3c:ec:ef:d9:23:15@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:31 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 4c:ed:fb:8f:dd:28@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:31 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 54:e6:fc:f9:56:d0@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:31 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 00:a0:de:fd:ad:41@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:34 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000000002b8
May 27 13:24:34 kernel: Mem abort info:
May 27 13:24:34 kernel: ESR = 0x96000007
May 27 13:24:34 kernel: Exception class = DABT (current EL), IL = 32 bits
May 27 13:24:34 kernel: SET = 0, FnV = 0
May 27 13:24:34 kernel: EA = 0, S1PTW = 0
May 27 13:24:34 kernel: Data abort info:
May 27 13:24:34 kernel: ISV = 0, ISS = 0x00000007
May 27 13:24:34 kernel: CM = 0, WnR = 0
May 27 13:24:34 kernel: user pgtable: 4k pages, 39-bit VAs, pgdp = 000000005aa2c669
May 27 13:24:34 kernel: [00000000000002b8] pgd=000000002535b003, pud=000000002535b003, pmd=0000000025556003, pte=0000000000000000