OpenVPN - OpenSSL cryptodev failure. How to throubleshoot

[drbrains]

I'm trying to use the AES engine build into then Mediatek MT7628 SoC. The OS is the latest Lede (OpenWRT) build. It is using OpenVPN 2.4.4 in combination with OpenSSL 1.0.2n. The cryptodev module is v1.9. All compiled from source using the Lede toolchains.

Encrypting and Decrypting a file using the OpenSSL-utils is working normally. To ensure no defects in the hardware-engine cross-checked by encrypting using HW and decrypting using software and visa-versa. Doing a compare between the original file and the decrypted output.

I want to use it with OpenVPN but I can't seem to be able to figure out where it goes wrong. It seems buffer-related, but I'm not an expert/developer.
I modified the crypto.c the dirty way to give me some more clues by outputting the buffer length cause that seemed the be my primary problem. I also changed the starting "for-loop" to 100 to rule out very small packages. The first run, shows my source and output buffer are the same but the test still fails. The next run shows different errors but all have a "0" buffer lenght in common.

For reference the discussion about this on the Lede-forum: https://forum.lede-project.org/t/openvp ... ot/10008/7

Code: Select all

root@OpenWrt:/tmp# openvpn --test-crypto --secret key --cipher AES-256-CBC --engine cryptodev
Sun Dec 17 11:56:01 2017 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Sun Dec 17 11:56:01 2017 OpenVPN 2.4.4 mipsel-openwrt-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
Sun Dec 17 11:56:02 2017 library versions: OpenSSL 1.0.2n  7 Dec 2017, LZO 2.10
Sun Dec 17 11:56:02 2017 OpenVPN 2.4.4 mipsel-openwrt-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
Sun Dec 17 11:56:03 2017 Initializing OpenSSL support for engine 'cryptodev'
Sun Dec 17 11:56:03 2017 Entering OpenVPN crypto self-test mode.
Sun Dec 17 11:56:03 2017 TESTING ENCRYPT/DECRYPT of packet length=100
Sun Dec 17 11:56:03 2017 OpenVPNEnc. buf.len=148
Sun Dec 17 11:56:03 2017 OpenVPNDec buf.len=100
Sun Dec 17 11:56:03 2017 SELF TEST FAILED, pos=0 in=87 out=75
Sun Dec 17 11:56:03 2017 Exiting due to fatal error
root@OpenWrt:/tmp# openvpn --test-crypto --secret key --cipher AES-256-CBC --engine cryptodev
Sun Dec 17 11:56:44 2017 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Sun Dec 17 11:56:44 2017 OpenVPN 2.4.4 mipsel-openwrt-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
Sun Dec 17 11:56:44 2017 library versions: OpenSSL 1.0.2n  7 Dec 2017, LZO 2.10
Sun Dec 17 11:56:44 2017 OpenVPN 2.4.4 mipsel-openwrt-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
Sun Dec 17 11:56:45 2017 Initializing OpenSSL support for engine 'cryptodev'
Sun Dec 17 11:56:45 2017 Entering OpenVPN crypto self-test mode.
Sun Dec 17 11:56:45 2017 TESTING ENCRYPT/DECRYPT of packet length=100
Sun Dec 17 11:56:45 2017 OpenVPNEnc. buf.len=148
Sun Dec 17 11:56:45 2017 Authenticate/Decrypt packet error: bad packet ID (may be a replay): [ #3800655692 / time = (2990300221) Tue Aug 28 16:28:45 1928 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
Sun Dec 17 11:56:45 2017 OpenVPNDec buf.len=0
Sun Dec 17 11:56:45 2017 SELF TEST FAILED, src.len=100 buf.len=0
Sun Dec 17 11:56:45 2017 Exiting due to fatal error
root@OpenWrt:/tmp# openvpn --test-crypto --secret key --cipher AES-256-CBC --engine cryptodev
Sun Dec 17 11:56:51 2017 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
Sun Dec 17 11:56:51 2017 OpenVPN 2.4.4 mipsel-openwrt-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
Sun Dec 17 11:56:51 2017 library versions: OpenSSL 1.0.2n  7 Dec 2017, LZO 2.10
Sun Dec 17 11:56:51 2017 OpenVPN 2.4.4 mipsel-openwrt-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
Sun Dec 17 11:56:52 2017 Initializing OpenSSL support for engine 'cryptodev'
Sun Dec 17 11:56:52 2017 Entering OpenVPN crypto self-test mode.
Sun Dec 17 11:56:52 2017 TESTING ENCRYPT/DECRYPT of packet length=100
Sun Dec 17 11:56:52 2017 OpenVPNEnc. buf.len=148
Sun Dec 17 11:56:52 2017 Authenticate/Decrypt packet error: bad packet ID (may be a replay): [ #2887217725 / time = (4121479353) Fri Jul  3 00:54:17 1964 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
Sun Dec 17 11:56:52 2017 OpenVPNDec buf.len=0
Sun Dec 17 11:56:52 2017 SELF TEST FAILED, src.len=100 buf.len=0
Sun Dec 17 11:56:52 2017 Exiting due to fatal error
root@OpenWrt:/tmp# 

[Pippin]

AFAIK, OpenVPN just passes instructions to OpenSSL, the rest is up to OpenSSL.
No need to use --engine cryptodev in config as OpenSSL version 1.0.0 and up will use it automatically when available.

Did you do the OpenSSL speedtests?
Without hardware crypto support:

Code: Select all

env OPENSSL_ia32cap=0 openssl speed -elapsed -evp aes-256-cbc -multi 4

With harware crypto support:

Code: Select all

openssl speed -elapsed -evp aes-256-cbc -multi 4

Compare.

In second terminal can do

Code: Select all

top -s 1 -aSCHIP

to see where CPU time is spend.

[drbrains]

Running the first test with the "env OPENSSL_ia32cap=0" part crashes the OS. Without Hardware it will do the "multi 4" part. Using the Hardware Crypto crashes half way during the test when using "multi 4". I'm using interrupts in the driver, so that might cause problems when using multi threads? Its a single core SoC so that might be a driver problem.

Code: Select all

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt SNAPSHOT, r5574-60a39e8
 -----------------------------------------------------
root@OpenWrt:~# openssl speed -elapsed -evp aes-256-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 953504 aes-256-cbc's in 2.99s
Doing aes-256-cbc for 3s on 64 size blocks: 268574 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 69517 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 17551 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 2192 aes-256-cbc's in 3.00s
OpenSSL 1.0.2n  7 Dec 2017
built on: reproducible build, date unspecified
options:bn(64,32) rc4(ptr,char) des(idx,cisc,2,long) aes(partial) blowfish(ptr) 
compiler: mipsel-openwrt-linux-musl-gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -I/home/drbrains/Lede/staging_dir/target-mipsel_24kc_musl/usr/include -I/home/drbrains/Lede/staging_dir/target-mipsel_24kc_musl/include -I/home/drbrains/Lede/staging_dir/toolchain-mipsel_24kc_gcc-6.3.0_musl/usr/include -I/home/drbrains/Lede/staging_dir/toolchain-mipsel_24kc_gcc-6.3.0_musl/include/fortify -I/home/drbrains/Lede/staging_dir/toolchain-mipsel_24kc_gcc-6.3.0_musl/include -znow -zrelro -DOPENSSL_SMALL_FOOTPRINT -DHAVE_CRYPTODEV -DUSE_CRYPTODEV_DIGESTS -DOPENSSL_NO_ERR -DTERMIOS -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -msoft-float -iremap/home/drbrains/Lede/build_dir/target-mipsel_24kc_musl/openssl-1.0.2n:openssl-1.0.2n -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -fpic -I/home/drbrains/Lede/package/libs/openssl/include -ffunction-sections -fdata-sections -fomit-frame-pointer -Wall -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DAES_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc       5102.36k     5729.58k     5932.12k     5990.74k     5985.62k
root@OpenWrt:~# modprobe mtk_aes
root@OpenWrt:~# openssl speed -elapsed -evp aes-256-cbc
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 140779 aes-256-cbc's in 2.99s
Doing aes-256-cbc for 3s on 64 size blocks: 145404 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 141959 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 97313 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 35108 aes-256-cbc's in 3.00s
OpenSSL 1.0.2n  7 Dec 2017
built on: reproducible build, date unspecified
options:bn(64,32) rc4(ptr,char) des(idx,cisc,2,long) aes(partial) blowfish(ptr) 
compiler: mipsel-openwrt-linux-musl-gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -I/home/drbrains/Lede/staging_dir/target-mipsel_24kc_musl/usr/include -I/home/drbrains/Lede/staging_dir/target-mipsel_24kc_musl/include -I/home/drbrains/Lede/staging_dir/toolchain-mipsel_24kc_gcc-6.3.0_musl/usr/include -I/home/drbrains/Lede/staging_dir/toolchain-mipsel_24kc_gcc-6.3.0_musl/include/fortify -I/home/drbrains/Lede/staging_dir/toolchain-mipsel_24kc_gcc-6.3.0_musl/include -znow -zrelro -DOPENSSL_SMALL_FOOTPRINT -DHAVE_CRYPTODEV -DUSE_CRYPTODEV_DIGESTS -DOPENSSL_NO_ERR -DTERMIOS -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -msoft-float -iremap/home/drbrains/Lede/build_dir/target-mipsel_24kc_musl/openssl-1.0.2n:openssl-1.0.2n -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -fpic -I/home/drbrains/Lede/package/libs/openssl/include -ffunction-sections -fdata-sections -fomit-frame-pointer -Wall -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DAES_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc        753.33k     3101.95k    12113.83k    33216.17k    95868.25k
root@OpenWrt:~#