2.4 clients subtle-failing against a 2.2 server

This forum is for admins who are looking to build or expand their OpenVPN setup.

Moderators: TinCanTech, TinCanTech, TinCanTech, TinCanTech, TinCanTech, TinCanTech

Forum rules
Please use the [oconf] BB tag for openvpn Configurations. See viewtopic.php?f=30&t=21589 for an example.
Post Reply
ratnix
OpenVpn Newbie
Posts: 12
Joined: Wed Mar 07, 2018 11:06 pm

2.4 clients subtle-failing against a 2.2 server

Post by ratnix » Wed May 16, 2018 10:23 pm

I'm stuck on a 2.2 server (I know! I inherited an abandoned vpn. Going to 2.4 is planned but not immediately an option).

2.2 and 2.3 clients connect to the server and are completely happy.
2.4 clients connecting to the server get connected and... well. They have varying and intermittent issues. pings work pretty much universally the whole time, but ssh or web things will work/break/work/break and be frustratingly unreliable. Sometimes they will timeout in under a few minutes. Sometimes they will linger on without dying but be unusable. Almost all connections will become unusable within 30 minutes. Detuning to 2.3 and using the same config on the same client box removes all issues.

My question is, what is different about a 2.4 client that would have it behave this differently from a 2.3 client, with an identical config?

Things I've looked into:
  • I'm aware of comp-lzo being deprecated, but it's still valid (and handled in my future 2.4 server)
  • I turned the verbosity to 6, traffic appears to be passing whether an ssh command is working or failing.
  • With verbosity at 6, the client parameters have some differences when I run it as 2.3 or 2.4, but nothing jumps out at me.
  • There's no client firewall in play, and no drops/blocks observed on the server's iptables.
Thanks for looking.

Server-on-2.2

script-security 2
up /etc/openvpn/udp/openvpn-up
down /etc/openvpn/udp/openvpn-down
port 1194
proto udp
dev tun0
ca /etc/openvpn/udp/keys/ca.crt
cert /etc/openvpn/udp/keys/server.crt
key /etc/openvpn/udp/keys/server.key
dh /etc/openvpn/udp/keys/dh.pem
crl-verify /etc/openvpn/udp/keys/crl.pem
server 10.8.248.0 255.255.252.0
client-connect /etc/openvpn/udp/plugins/client-connect
learn-address /usr/lib/openvpn/plugins/netfilter_openvpn.sh
push "dhcp-option DNS 10.8.72.15"
push "dhcp-option DOMAIN company.com"
keepalive 10 120
duplicate-cn
tls-auth /etc/openvpn/udp/keys/ta.key 0
cipher AES-256-CBC
comp-lzo
max-clients 255
user openvpn
group openvpn
persist-key
persist-tun
status /var/log/openvpn/udp-status.log
log-append /var/log/openvpn/udp-openvpn.log
verb 4
mute 20
plugin /usr/lib/openvpn/plugins/duo_openvpn.so /usr/lib/openvpn/plugins/duo_openvpn.py
script-security 3
management /var/run/openvpn-udp.socket unix
management-client-user root


Client-on-2.3-or-2.4

remote vpn.company.com 1194 udp

auth-user-pass
persist-key
tls-client
tls-auth private/ta.key 1
pull
ca private/ca.crt
dev tun
persist-tun
cert private/cert.crt
comp-lzo no
nobind
key private/key.key
cipher AES-256-CBC
resolv-retry infinite

up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf
script-security 2

TinCanTech
OpenVPN Protagonist
Posts: 11137
Joined: Fri Jun 03, 2016 1:17 pm

Re: 2.4 clients subtle-failing against a 2.2 server

Post by TinCanTech » Thu May 17, 2018 10:00 am

Openvpn 2.2 is at least 5 years old and is no longer supported.

ratnix
OpenVpn Newbie
Posts: 12
Joined: Wed Mar 07, 2018 11:06 pm

Re: 2.4 clients subtle-failing against a 2.2 server

Post by ratnix » Thu May 17, 2018 2:03 pm

I'm painfully aware of 2.2's age. My question is more about where 2.4 would have lost interop in such a low-key manner vs how 2.3 is behaving, since cross-version interoperability is (while not guaranteed) at least a usual consideration in development, and there's nothing so far to indicate where the watershed of incompatibility came in.

TinCanTech
OpenVPN Protagonist
Posts: 11137
Joined: Fri Jun 03, 2016 1:17 pm

Re: 2.4 clients subtle-failing against a 2.2 server

Post by TinCanTech » Thu May 17, 2018 4:02 pm

ratnix wrote:
Thu May 17, 2018 2:03 pm
I'm painfully aware of 2.2's age.
So do something about it .. your server is Full of Holes .. you may as well not bother.
ratnix wrote:
Thu May 17, 2018 2:03 pm
My question is more about where 2.4 would have lost interop in such a low-key manner
I assure you, it was not a low key manner
ratnix wrote:
Thu May 17, 2018 2:03 pm
vs how 2.3 is behaving, since cross-version interoperability is (while not guaranteed) at least a usual consideration in development
And no doubt the problem you are experiencing was known about and discussed .. but it's also 5 Years Old .. and unsupported .. so it could be anything since between then and now .. Heartbleed for example ..

More likely it is the SSL library than Openvpn.
ratnix wrote:
Thu May 17, 2018 2:03 pm
and there's nothing so far to indicate where the watershed of incompatibility came in.
When 2.2 was obsoleted. It's probably on the wiki somewhere :mrgreen:

ratnix
OpenVpn Newbie
Posts: 12
Joined: Wed Mar 07, 2018 11:06 pm

Re: 2.4 clients subtle-failing against a 2.2 server

Post by ratnix » Thu May 17, 2018 4:48 pm

I grabbed a spare VM and put 2.3.14 (I know 2.3.18 is latest, but a .14 RPM was handy) in server mode, same server and client configs as the initial item in this thread, same results. So I've taken the 2.2-is-deprecated punching bag is out of the equation for you.

TinCanTech
OpenVPN Protagonist
Posts: 11137
Joined: Fri Jun 03, 2016 1:17 pm

Re: 2.4 clients subtle-failing against a 2.2 server

Post by TinCanTech » Thu May 17, 2018 5:56 pm

ratnix wrote:
Thu May 17, 2018 4:48 pm
same results
where ?

ratnix
OpenVpn Newbie
Posts: 12
Joined: Wed Mar 07, 2018 11:06 pm

Re: 2.4 clients subtle-failing against a 2.2 server

Post by ratnix » Thu May 17, 2018 6:22 pm

Against the now-2.3 test server, a 2.3 client is rock solid: ssh sessions create and tear down without issue, 30 minutes of successful testing convinced me this is fine.
Server unchanged, replacing the 2.3 client with a 2.4 client, same configs, the 2.4 client will connect every time. ~25% of the time, the client cannot ssh to something over the tunnel right then. ~75% of the time, you can ssh after the tunnel is established. Within a few minutes (2-10?) the client will be unable to ssh to a new box ever again. I've not had the 2.4 client survive longer than 30 minutes before becoming inoperative. But, at no point does it go into ping-restart and drop the tunnel.
Going back to using 2.3 on the client, total success again.

ratnix
OpenVpn Newbie
Posts: 12
Joined: Wed Mar 07, 2018 11:06 pm

Re: 2.4 clients subtle-failing against a 2.2 server

Post by ratnix » Fri May 18, 2018 12:56 am

I think I have where things are wrong. I am a bad C person, so bear with my notes as I try to sound this out.

tl;dr 2.4 has a bug: any server (seen in 2.2-2.4) with some flavor of 'comp-lzo' active ('yes' or 'adaptive'), vs a 2.4 client with 'comp-lzo no', will fail when compressed frames start arriving.


openvpn-2.3.18/src/openvpn/options.c line 6376
"comp-lzo" and no argument (which is what my 2.3 server says) yields
options->lzo = LZO_SELECTED|LZO_ON|LZO_ADAPTIVE;
My 2.2 server has this config option too, and is probably 'the same' since comp-lzo was the only compression then.

openvpn-2.3.18/src/openvpn/options.c line 6378
"comp-lzo no" (what the 2.3 client config says) yields
options->lzo = LZO_SELECTED;

then in
openvpn-2.3.18/src/openvpn/init.c line 3487
if ((options->lzo & LZO_SELECTED) && (c->mode == CM_P2P || child))
lzo_compress_init (&c->c2.lzo_compwork, options->lzo);

if we have LZO_SELECTED on the client, we kick off lzo_compress_init, openvpn-2.3.18/src/openvpn/lzo.c line 113. And since ENABLE_LZO_STUB is not true in pretty much any sane place, LZO is 'active' (that is, we use the contents of lzo.c), even though the option says 'no'.

Now, here's the fun part:
lzo_compression_enabled (openvpn-2.3.18/src/openvpn/lzo.c line 148) returns false for obvious reasons, so we never compress any frame (openvpn-2.3.18/src/openvpn/lzo.c line 182), but down in lzo_decompress (openvpn-2.3.18/src/openvpn/lzo.c line 231) we'll decompress anything that comes our way.

And going back to the beginning: the server has "comp-lzo" (adaptive mode by default). The server may compress, and the client will accept compressed, even though the client will never send compressed.



Moving to 2.4:
comp-lzo is deprecated but still valid. So looking into what happens when we use it:

openvpn-2.4.6/src/openvpn/options.c line 7393
a 'comp-lzo no' yields
options->comp.alg = COMP_ALG_STUB;
options->comp.flags = 0;

Then, in openvpn-2.4.6/src/openvpn/comp.c line 46:
compctx->flags = opt->flags;
compctx->alg = comp_stub_alg;

openvpn-2.4.6/src/openvpn/compstub.c has all of comp_stub_alg's references, so that file is where I go next.

But the stub_decompress function has no such concept of being nice to you and accepting adaptively-compressed incoming lzo frames.
If I'd had 'comp-lzo yes' | 'comp-lzo adaptive' | 'comp-lzo' then options->comp.alg = COMP_ALG_LZO; would've kicked in back in options.c, and
openvpn-2.4.6/src/openvpn/lzo.c would become active just like in 2.3, and we'd accept compressed frames.


Why did it fail so irregularly for me? My guess is it took the server some variable amount of time before it compressed something.
Why didn't I see it in the logs? My guess is that the client I was using (Viscosity) is suppressing some errors.. but I haven't proven that.

I can replicate this with a 2.4 server that has 'comp-lzo' adaptive/yes, also.
If I change the server to 'comp-lzo yes', I can VPN-connect but can not ssh anywhere at all, from the very first second.
If I change the server to 'comp-lzo no', I can VPN-connect and can ssh.
Switching to 'compress' in the client config avoids the problem, but (while deprecated) 'comp-lzo' is still valid in 2.4 and its behavior is now different than in pre-2.4 releases.

ratnix
OpenVpn Newbie
Posts: 12
Joined: Wed Mar 07, 2018 11:06 pm

Re: 2.4 clients subtle-failing against a 2.2 server

Post by ratnix » Fri May 18, 2018 2:10 am

https://community.openvpn.net/openvpn/ticket/861 is very similar to this, though it was declared notabug. :(

ratnix
OpenVpn Newbie
Posts: 12
Joined: Wed Mar 07, 2018 11:06 pm

Re: 2.4 clients subtle-failing against a 2.2 server

Post by ratnix » Fri May 18, 2018 11:50 am

Found https://community.openvpn.net/openvpn/ticket/952 and have latched onto that bug. I believe it fully covers my issue.

Post Reply