learn-address fork fails when auth-user-pass-verify deferred

petiepooo

Hello all.

Thanks for your effort in sharing and supporting the greatest VPN software out there! I have a rather peculiar edge case that's failing on me.

I'm running an OpenVPN 2.2.1 server. I know that's old, but it's what comes with Ubuntu 12.04.5 and I haven't had time to build a new binary and test that it works in my production environment.

I've enabled two-factor authentication using DuoSecurity's platform, available at https://github.com/duosecurity/duo_openvpn. It's nice in that it performs a push notification to your mobile device upon login attempt. However, my VPN link would fail to route as soon as I enabled the plugin, and I narrowed it down to the following logs in my verb 6 output (199.1xx.xx.xx is client IP, 10.2x.xx.xx is VIP).

Fri Jan 29 14:58:11 2016 us=340854 PLUGIN_INIT: POST /opt/duo/duo_openvpn.so '[/opt/duo/duo_openvpn.so] [integration key redacted] [xxx secret key redacted xxxx] [api-20xxxxxx.duosecurity.com]' intercepted=PLUGIN_AUTH_USER_PASS_VERIFY
...
Fri Jan 29 14:59:09 2016 us=34136 199.1xx.xx.xx:60581 PLUGIN_CALL: POST /opt/duo/duo_openvpn.so/PLUGIN_AUTH_USER_PASS_VERIFY status=2
Fri Jan 29 14:59:09 2016 us=34164 199.1xx.xx.xx:60581 TLS: Username/Password authentication deferred for username 'user-redacted'
...
Fri Jan 29 14:59:19 2016 us=255422 cn-redacted/199.1xx.xx.xx:60581 MULTI_sva: pool returned IPv4=10.2x.xx.x, IPv6=40a7:xxxx:ipv6:redacted:xxxx:1
Fri Jan 29 14:59:19 2016 us=257419 cn-redacted/199.1xx.xx.xx:60581 WARNING: Failed running command (--learn-address): external program fork failed
Fri Jan 29 14:59:19 2016 us=257466 cn-redacted/199.1xx.xx.xx:60581 MULTI: Learn FAILED: 10.2x.xx.x -> cn-redacted/199.1xx.xx.xx:60581
...
Fri Jan 29 14:59:41 2016 us=470485 cn-redacted/199.1xx.xx.xx:60581 MULTI: bad source address from client [10.2x.xx.x], packet dropped
Fri Jan 29 14:59:41 2016 us=470521 cn-redacted/199.1xx.xx.xx:60581 UDPv4 WRITE [69] to [AF_INET]199.1xx.xx.xx:60581: P_DATA_V1 kid=0 DATA len=68
Fri Jan 29 14:59:42 2016 us=647792 cn-redacted/199.1xx.xx.xx:60581 UDPv4 READ [133] from [AF_INET]199.1xx.xx.xx:60581: P_DATA_V1 kid=0 DATA len=132
Fri Jan 29 14:59:42 2016 us=647846 cn-redacted/199.1xx.xx.xx:60581 MULTI: bad source address from client [10.2x.xx.x], packet dropped
...

(All further P_DATA_V1 READ calls followed by bad source address notice)

This is using Ubuntu's packaged learn-address.sh script that simply does "exec /bin/true". If I remove the learn-address directive from the server config, everything works. I'm not currently relying on an external learn-address script, so that's my resolution for now.

This is a bit of a peculiar edge case, and I know it was found on an older version. I've looked through the change logs since then to see if anything popped out at me, but nothing did. I haven't dug into the source yet to see if code in that area has changed between 2.2.1 an the current version. I've also searched the forums for learn-address and came up empty.

Has this issue been discussed and/or addressed yet to anyone's knowledge?

Traffic

petiepooo wroteI'm running an OpenVPN 2.2.1 server <...>

It is your security ..

petiepooo wroteFri Jan 29 14:59:19 2016 us=257419 cn-redacted/199.1xx.xx.xx:60581 WARNING: Failed running command (--learn-address): external program fork failed

petiepooo wroteThis is using Ubuntu's packaged learn-address.sh script that simply does "exec /bin/true".

Why would you do this ?

petiepooo wroteIf I remove the learn-address directive from the server config, everything works.

Just remove it and move on ..

petiepooo

Traffic, thanks for your curt and prompt response.

Traffic wrote
petiepooo wroteThis is using Ubuntu's packaged learn-address.sh script that simply does "exec /bin/true".
Why would you do this ?

If you're referring to the contents or presence of the learn-address.sh script, _I_ didn't do this. I assume one of the debian or ubuntu package maintainers did, presumably to make it one step easier for a user to implement a script that uses the learn-address feature.

On the other hand, if you're asking why one would use the learn-address directive, it allows one to respond to a new client coming online and add firewall rules or deny access at the network level.

Enabling that script by default, however, does expose a potential hole in OpenVPN's test regime. I would be surprised if learn-address is explicitly tested with a plugin that intercepts auth-user-pass-verify and responds with OPENVPN_PLUGIN_FUNC_DEFERRED. That's a small hole, granted, but the error appears on the surface to be a simple problem with forking the new process.

Traffic wrote
petiepooo wroteIf I remove the learn-address directive from the server config, everything works.
Just remove it and move on ..

Is that the official response, then; that learn-address is not supported with a plugin that defers auth-user-pass-verify? What if I'd like to use the learn-address functionality?

To my original question, has anyone else run into this particular combination of features?

Traffic

petiepooo wroteIf you're referring to the contents or presence of the learn-address.sh script, _I_ didn't do this. I assume one of the debian or ubuntu package maintainers did, presumably to make it one step easier

That is what I am referring to and it looks like an ubuntu bug ..

petiepooo wroteOn the other hand, if you're asking why one would use the learn-address directive, it allows one to respond to a new client coming online and add firewall rules or deny access at the network level

Not with that crazy script .. all that script does is drop all necessary information. I have never seen this script on any OpenVPN documentation and when I run tests it causes openvpn to fail .. Hence:

Traffic wroteJust remove it and move on ..

petiepooo wroteIs that the official response, then; that learn-address is not supported with a plugin that defers auth-user-pass-verify? What if I'd like to use the learn-address functionality?

No where as it implied or documented that such a setup is not supported. If you want to use --learn-address use it correctly.

Also, https://duo.com/docs/openvpn makes no mention of using --learn-address .. so the two are not linked.

Finally, as it is an old version of OpenVPN (v2.2.1 dated 08-May-2013) I urge you to upgrade it .. You can get the latest version from OpenVPN Repos:
https://openvpn.net/index.php/open-source/downloads.html

petiepooo

Traffic wroteNot with that crazy script .. all that script does is drop all necessary information. I have never seen this script on any OpenVPN documentation and when I run tests it causes openvpn to fail .. Hence:
Traffic wroteJust remove it and move on ..

Traffic, it's not the script. Either your test was faulty (did you set script-security to 2?) or you encountered the same issue I did.

The script is just a null test, as it simply returns SUCCESS (a value of 0). It's a template put in place for people to add their actions to; a time saver so that people don't have to set learn-address in the configs, modify their script-security setting, and create the script from scratch. I'll agree that it's a waste of resources to spawn another process just to have it return a value, but that little exercise should not cause harm, right? FWIW, setting learn-address to /bin/true also reports failure to fork when the plugin is active.

I'll also go back to the fact that it works just fine without the plugin. And on the other two openvpn services I'm running on the same server, both of which use that same script. Also, I see output from all three services if I add a line to append its received parameters to a file before exec'ing /bin/true. IOW, the script is being executed even though the openvpn process reports a failure to fork.

Here's the real culprit. When the duosec plugin is loaded, it sets SIG_CHLD to SIG_IGN at https://github.com/duosecurity/duo_openvpn/blob/master/duo_openvpn.c#L64, which causes the call to waitpid() at https://github.com/OpenVPN/openvpn/blob/release/2.2/misc.c#L531 to return ECHILD, even though fork() was successful and the script ran. See the "Errors" section of http://linux.die.net/man/2/waitpid for details.

The error handling in openvpn_execve() is minimal, as the status in the second parameter of waitpid() is immediately overwritten and errno is never checked. Not strictly an openvpn bug, but there is room for improvement, IMHO...

TL;DR: It's a bug in the duosec plugin. They may need to daemonize the child process and let init clean it up rather than just ignoring SIG_CHLD. I'll submit an issue to them.

Debugging other people's code is fun!

Traffic

petiepooo wroteTraffic, it's not the script. Either your test was faulty (did you set script-security to 2?) or you encountered the same issue I did.

*sigh* I would say your test is faulty .. :geek:

petiepooo wroteThe script is just a null test, as it simply returns SUCCESS

This effectively disables openvpn from learning any addresses - Thereby causing openvpn to fail to function.

petiepooo wroteDebugging other people's code is fun!

If you pay me .. otherwise, no thanks.

But you can keep battering your head against that wall for as long as you want to ..

petiepooo

Traffic wrote
petiepooo wroteThe script is just a null test, as it simply returns SUCCESS
This effectively disables openvpn from learning any addresses - Thereby causing openvpn to fail to function.

Are you just trolling me now? Citation, please!

Here's my countering citation directly from the manpage section on --learn-address (bold highlight added by me):

On "add" or "update" methods, if the script returns a failure code (non-zero), OpenVPN will reject the address and will not modify its internal routing table.

Or if you prefer, explore the platform_system_ok() function at https://github.com/OpenVPN/openvpn/blob/master/src/openvpn/platform.c#L211 and how that would affect the return value of openvpn_execve_check() at https://github.com/OpenVPN/openvpn/blob/master/src/openvpn/misc.c#L260. (hint: a return value of 0 results in it returning true, cascading up to learn_addr_multi() at https://github.com/OpenVPN/openvpn/blob/master/src/openvpn/multi.c#L1054)

I would have just not replied, but, you know, duty calls... https://xkcd.com/386/

And I'm going to bed now. Good night.

Traffic

petiepooo wroteI would have just not replied, but, you know, duty calls... https://xkcd.com/386/

:lol: :lol: :lol: :lol: :lol: 8-)

I will reply to your other misconceptions in due course ....

Bear in mind you are running an old (unsupported) version.

Traffic

petiepooo wrote
Traffic wrote
petiepooo wroteThe script is just a null test, as it simply returns SUCCESS
This effectively disables openvpn from learning any addresses - Thereby causing openvpn to fail to function.
Are you just trolling me now? Citation, please!

Server:

learn-address /etc/openvpn/learnaddtest.sh

/etc/openvpn/learnaddtest.sh:

exec /bin/true

Server log:

Sat Feb  6 11:24:13 2016 us=290739 client1/x.x.x.x:2230 WARNING: Failed running command (--learn-address): could not execute external program
Sat Feb  6 11:24:13 2016 us=290792 client1/x.x.x.x:2230 MULTI: Learn FAILED: 10.8.0.101 -> client1/x.x.x.x:2230

I have tested this more rigorously than an average problem :mrgreen:

$ openvpn --version
OpenVPN 2.3_git iproute2 [git:master/417fe4a72c82accc+] x86_64-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH] [IPv6] built on Jan 29 2016
library versions: OpenSSL 1.0.1k 8 Jan 2015, LZO 2.08
Originally developed by James Yonan
Copyright (C) 2002-2010 OpenVPN Technologies, Inc. <sales@openvpn.net>
Compile time defines: enable_async_push=no enable_comp_stub=no enable_crypto=yes enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=yes enable_fragment=yes enable_iproute2=yes enable_libtool_lock=yes enable_lz4=yes enable_lzo=yes enable_management=yes enable_multi=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=no enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_server=yes enable_shared=yes enable_shared_with_static_runtimes=no enable_small=no enable_static=yes enable_strict=no enable_strict_options=no enable_systemd=no enable_werror=no enable_win32_dll=yes enable_x509_alt_username=no with_crypto_library=openssl with_gnu_ld=yes with_mem_check=no with_plugindir='$(libdir)/openvpn/plugins' with_sysroot=no

petiepooo

I'm sorry you can't get the learn-address feature to work for you. It still works fine for me. In fact, once the faulty plugin is patched, I expect it to work on all three servers, rather than just the two without the plugin.

Did you forget to enable script-security with a setting of 2 or higher? That's the mistake most people who see that warning have made.

Is the shell script executable bit set? It could be because you omitted the shebang; try adding "#!/bin/bash" as the first line (that's what I have as well). Without a shebang, it's just a text file, and the system loader doesn't know what interpreter to use.

For a better test, I'd suggest first setting up a test with a learn-address script of your choice to make sure you have the rest of your environment configured correctly. Only then can you logically argue that it's the script and not something else. Assuming it's the script on your first error is a straw-man fallacy.

Or, if you'd like me to help debug your environment, I'll need more complete logs including a full config.. :mrgreen:

Also, I have to apologize; I didn't mean to bad-mouth Ubuntu or Debian. The script we're discussing doesn't seem to be their default after all. I'm not sure where it originated from on my system, but I must have copied it from some example config somewhere.. If it were my original, I think I would have used "exit 0" rather than "exec /bin/true" to use a bash built-in rather than load an external binary. In any case, it had been working as is for more than a year and a half before I tried adding the faulty duosec plugin.

Traffic

Cutting to the chase .. (Please excuse my obvious curtness, I am simply saving us the time)

You appear to be having problems using a plugin (duo) along side a script used for --learn-address

I am not going to summarise the intricacies of this problem for four reasons:

We do not provide support for third party apps/plugins etc.
You have already made it clear that you have a solution.
The script you are/were using is essentially pointless and you did not document it clearly.
(I was already aware of the #!/bin/sh issue but instead chose to accurately follow your post)
You are using openvpn v2.2.1, which is now almost three years out of date.
(You have clear instructions how to update this regardless of your distro repo)

You now have two options:

Update openvpn and try again.
Report the problem as is to the plugin author.

Sorry to be short but I want to get to the resolution .. not be analysing ***old (openvpn) and ***foreign (plugin) code. If you are determined to discuss this with hard core developers then I suggest the developer mailing list .. but you have been warned ***above.

On the other hand:

petiepooo wroteI would have just not replied, but, you know, duty calls... https://xkcd.com/386/

This did make me laugh .. I admit, I am guilty (both objectively and subjectively) 8-)

petiepooo wroteAre you just trolling me now?

A little guilty of this too .. please accept my apologies. :oops:

petiepooo wroteDid you forget to ..

Thanks for your help .. perhaps you have not noticed my post count and the implications there of .. 😃
Try the https://forums.openvpn.net/newposts.html link for a few days ..

petiepooo wroteOr if you prefer, explore the platform_system_ok() function ..

You are clearly skilled with programming .. perhaps you can become a contributor to openvpn .. :geek:

petiepooo wroteAlso, I have to apologize; I didn't mean to bad-mouth Ubuntu or Debian

I am quite sure they accept your apology 😉

Rest assured, my test rigs are configured to test (at short notice) almost anything you can throw at me! With the obvious exception of third party plugins, especially those that make money out of Free Open Source Software and can clearly afford the necessary resources to do the job themselves.

I would highly recommend that you subscribe to the developer and user mailing lists to get a clear idea of how active this project is and get some very useful tips and tricks not otherwise documented.

I will keep up with your thread but will unlikely have anything further to add.
kind regards

Traffic

Typical .. turns out there is something to add:

Depending on how you use --daemon to daemonize openvpn can cause this fork failure:

Starting openvpn from the command line [sudo openvpn --config server.conf --daemon openvpn-server] causes:

WARNING: Failed running command (--client-connect): external program fork failed

Starting openvpn via the service [sudo service openvpn start] does not cause fork failure ..

Perhaps you can test this idea with --learn-address to see if it is also effected ..

There are a number of partially related bug reports but I think I may have to open a new one for this .. :roll:

petiepooo

There's a push request pending for the duosecurity/duo_openvpn repository on github that resolves my issue. I can confirm that learn-address scripts are working again with those commits included.

The plugin was ignoring SIG_CHLD, which the wait() manpage clearly states will result in a call to pidwait() failing. The reason SIG_CHLD was being ignored is to eliminate zombie processes once the authentication cycle was complete. The normal and accepted way of doing this is to make the child a daemon and let init reap.

I don't think OpenVPN is at fault here. The plugin was not following standard *nix process management practices. If you've found another issue, that's great, but I don't think it's directly related. It may, however, explain why we were getting such different results... 🙂

Thanks.

Traffic

Thank you for your detailed reply 8-)