OpenVPN FIPS support (almost there)
Posted: Tue Nov 21, 2017 5:06 pm
I've been working on building OpenVPN+OpenSSL for FIPS support. I feel like I've just about gotten it done, but I've run into an issue that's beyond a simple build sequence and was looking for some guidance.
The issue is that the OpenSSL FIPS Object Module v2.0 forbids MD5 once an application enables FIPS mode (eg: FIPS_set_mode(1)). But the code src/openvpn/ssl.c:tls1_PRF() makes a call to md_kt_get("MD5") without checking the return code. This results in a SIGSEGV a few lines later, when the function calls tls1_P_hash(md5, ...).
I'm not exactly sure how to resolve this. I tried to blindly just disable the code that attempts to call md_kt_get("MD5") and instead rely on the results of out2 (which are filled with SHA1 digest values). This prevents the SIGSEGV, but causes the tunnels to be unusable "AEAD Decrypt error: cipher final failed" (note: I did NOT expect this to succeed -- I only did it confirm I'd found the bit of code causing trouble). It's clear I can't just drop the MD5 call.
Does anyone have any advice on the right way to proceed?
Once we have this working, I'd be happy to share my work with the Community as either "HowTo's" or even a code diff.
The issue is that the OpenSSL FIPS Object Module v2.0 forbids MD5 once an application enables FIPS mode (eg: FIPS_set_mode(1)). But the code src/openvpn/ssl.c:tls1_PRF() makes a call to md_kt_get("MD5") without checking the return code. This results in a SIGSEGV a few lines later, when the function calls tls1_P_hash(md5, ...).
I'm not exactly sure how to resolve this. I tried to blindly just disable the code that attempts to call md_kt_get("MD5") and instead rely on the results of out2 (which are filled with SHA1 digest values). This prevents the SIGSEGV, but causes the tunnels to be unusable "AEAD Decrypt error: cipher final failed" (note: I did NOT expect this to succeed -- I only did it confirm I'd found the bit of code causing trouble). It's clear I can't just drop the MD5 call.
Does anyone have any advice on the right way to proceed?
Once we have this working, I'd be happy to share my work with the Community as either "HowTo's" or even a code diff.