This is potentially a rather odd problem that I've been wrestling with since the release of OpenVPN 2.3.3...
Problem: When users connect to the VPN, all traffic is routed through the VPN as expected and
external websites can be accessed without problems. However, Samba and web services hosted on the
same box as the VPN frequently hang. This occurs whether the connection is over TCP or UDP.
So, for example, clients can connect out to
https://www.google.com/ fine, and
http://www.whatismyip.com/ shows the IP address of the VPN server, as expected. However, connections to the
local web server at
http://172.16.0.1/ hang, as does Samba file share to that server. Note that this does not appear to be a problem with my firewall (the web pages begin to load, but then hang), nor is it a web server problem (if I connect to the server via an SSH tunnel, everything works fine).
Setup: My organization has a central Linux server that hosts two OpenVPN instances, one running on UDP 1194 (for normal use) and one running on TCP 443 (for use when behind restrictive firewalls). The server also hosts a local instance of Redmine and provides Samba file sharing; both of these services are only available over the VPN. Packets are forwarded between the two instances using iptables and the appropriate route directives.
Here's the server configuration for the UDP VPN:
proto udp
port 1194
dev tun0
ca /opt/openvpn/utah/easy-rsa/keys_clients/ca.crt
cert /opt/openvpn/utah/easy-rsa/keys_server/server.crt
key /opt/openvpn/utah/easy-rsa/keys_server/server.key
dh /opt/openvpn/utah/easy-rsa/keys_server/dh2048.pem
plugin /usr/lib/openvpn/plugins/openvpn-plugin-auth-pam.so login
tls-auth /opt/openvpn/utah/easy-rsa/keys_server/ta.key 0
replay-persist /opt/openvpn/utah/clients/states/udp_1194
cipher AES-256-CBC
server 172.16.0.0 255.255.0.0
dhcp-option DNS 172.16.0.1
push "route 172.17.0.0 255.255.0.0"
client-config-dir /opt/openvpn/utah/clients/udp_1194
client-to-client
keepalive 10 120
comp-lzo
max-clients 60
user nobody
group nobody
persist-key
persist-tun
status /run/openvpn_udp_1194.status
log-append /var/log/openvpn_udp_1194.log
verb 3
;crl-verify /opt/openvpn/utah/easy-rsa/keys/crl.pem
;mtu-test
fragment 1300
mssfix 1250
Here's an example client configuration:
client
dev tun
proto udp
remote not.my.real.server.org 1194
resolv-retry infinite
nobind
persist-key
persist-tun
<ca>
...STUFF...
</ca>
<cert>
...STUFF...
</cert>
<key>
...STUFF...
</key>
ns-cert-type server
verify-x509-name server name
key-direction 1
<tls-auth>
...STUFF...
</tls-auth>
cipher AES-256-CBC
comp-lzo
verb 3
fragment 1300
mssfix
auth-user-pass
And here's a code snippet showing VPN initialization and a single connection attempt:
Thu Apr 24 08:06:02 2014 OpenVPN 2.3.3 x86_64-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [MH] [IPv6] built on Apr 9 2014
Thu Apr 24 08:06:02 2014 PLUGIN_INIT: POST /usr/lib/openvpn/plugins/openvpn-plugin-auth-pam.so '[/usr/lib/openvpn/plugins/openvpn-plugin-auth-pam.so] [login]' intercepted=PLUGIN_AUTH_USER_PASS_VERIFY
Thu Apr 24 08:06:02 2014 Diffie-Hellman initialized with 2048 bit key
Thu Apr 24 08:06:02 2014 Control Channel Authentication: using '/opt/openvpn/utah/easy-rsa/keys_server/ta.key' as a OpenVPN static key file
Thu Apr 24 08:06:02 2014 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Apr 24 08:06:02 2014 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Apr 24 08:06:02 2014 Socket Buffers: R=[212992->131072] S=[212992->131072]
Thu Apr 24 08:06:02 2014 ROUTE_GATEWAY 10.188.233.1/255.255.255.0 IFACE=eth0 HWADDR=d0:67:e5:f3:8f:9d
Thu Apr 24 08:06:02 2014 TUN/TAP device tun0 opened
Thu Apr 24 08:06:02 2014 TUN/TAP TX queue length set to 100
Thu Apr 24 08:06:02 2014 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Thu Apr 24 08:06:02 2014 /usr/bin/ip link set dev tun0 up mtu 1500
Thu Apr 24 08:06:02 2014 /usr/bin/ip addr add dev tun0 local 172.16.0.1 peer 172.16.0.2
Thu Apr 24 08:06:02 2014 /usr/bin/ip route add 172.16.0.0/16 via 172.16.0.2
Thu Apr 24 08:06:02 2014 GID set to nobody
Thu Apr 24 08:06:02 2014 UID set to nobody
Thu Apr 24 08:06:02 2014 UDPv4 link local (bound): [undef]
Thu Apr 24 08:06:02 2014 UDPv4 link remote: [undef]
Thu Apr 24 08:06:02 2014 MULTI: multi_init called, r=256 v=256
Thu Apr 24 08:06:02 2014 IFCONFIG POOL: base=172.16.0.4 size=16382, ipv6=0
Thu Apr 24 08:06:02 2014 Initialization Sequence Completed
Thu Apr 24 08:06:16 2014 50.8.99.183:58258 TLS: Initial packet from [AF_INET]50.8.99.183:58258, sid=c59740fa bf758a6b
Thu Apr 24 08:06:18 2014 50.8.99.183:58258 VERIFY OK: depth=1, C=US, ST=CO, L=Denver, O=The Public Interest Network, OU=utah, CN=authority, name=OpenVPN, emailAddress=iohelpdesk@publicinterestnetwork.org
Thu Apr 24 08:06:18 2014 50.8.99.183:58258 VERIFY OK: depth=0, C=US, ST=CO, L=Denver, O=The Public Interest Network, OU=IO Team, CN=nacks, name=Nathan Acks, emailAddress=nacks@publicinterestnetwork.org
no talloc stackframe at ../source3/param/loadparm.c:4864, leaking memory
Thu Apr 24 08:06:19 2014 50.8.99.183:58258 PLUGIN_CALL: POST /usr/lib/openvpn/plugins/openvpn-plugin-auth-pam.so/PLUGIN_AUTH_USER_PASS_VERIFY status=0
Thu Apr 24 08:06:19 2014 50.8.99.183:58258 TLS: Username/Password authentication succeeded for username 'nacks'
Thu Apr 24 08:06:19 2014 50.8.99.183:58258 Data Channel Encrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
Thu Apr 24 08:06:19 2014 50.8.99.183:58258 Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Apr 24 08:06:19 2014 50.8.99.183:58258 Data Channel Decrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
Thu Apr 24 08:06:19 2014 50.8.99.183:58258 Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Apr 24 08:06:20 2014 50.8.99.183:58258 Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 2048 bit RSA
Thu Apr 24 08:06:20 2014 50.8.99.183:58258 [nacks] Peer Connection Initiated with [AF_INET]50.8.99.183:58258
Thu Apr 24 08:06:20 2014 nacks/50.8.99.183:58258 OPTIONS IMPORT: reading client specific options from: /opt/openvpn/utah/clients/udp_1194/nacks
Thu Apr 24 08:06:20 2014 nacks/50.8.99.183:58258 MULTI: Learn: 172.16.233.5 -> nacks/50.8.99.183:58258
Thu Apr 24 08:06:20 2014 nacks/50.8.99.183:58258 MULTI: primary virtual IP for nacks/50.8.99.183:58258: 172.16.233.5
Thu Apr 24 08:06:22 2014 nacks/50.8.99.183:58258 PUSH: Received control message: 'PUSH_REQUEST'
Thu Apr 24 08:06:22 2014 nacks/50.8.99.183:58258 send_push_reply(): safe_cap=940
Thu Apr 24 08:06:22 2014 nacks/50.8.99.183:58258 SENT CONTROL [nacks]: 'PUSH_REPLY,route 172.17.0.0 255.255.0.0,route 172.16.0.0 255.255.0.0,topology net30,ping 10,ping-restart 120,redirect-gateway def1,dhcp-option DNS 172.16.0.1,dhcp-option WINS 172.16.0.1,dhcp-option NBT 8,ifconfig 172.16.233.5 172.16.233.6' (status=1)
The TCP VPN setup is essentially identical; differences are the protocol and port (of course!), the internal IP addresses, what route I'm pushing, and the absence of the fragment and mssfix directives.
What I've Done So Far: I made several changes in the move to 2.3.3 post-Heartbleed:
- Deployed new client and server certificates.
- Upgraded my Diffie-Hellman key to 2048 bits.
- Added packet forwarding between OpenVPN instances.
- Added tls-auth and replay-persists directives.
- Added fragment and mssfix directives (in the course of troubleshooting the current problem).
The fragment and mssfix directives where added after running mtu-test, but don't actually seem to solve the problem. I've also tried reverting each one of these changes (except for the certificates), all without effect.
OpenVPN 2.3.2 worked without problems.
All
desktop clients (Windows using the official OpenVPN client, MacOS X using either Tunnelblick or Viscosity, and Linux systems using Network Manager or the OpenVPN command line application directly) have this problem.
However, iOS clients using OpenVPN Connect
work fine.
There is no difference in the logs either client- or server-side that I can find at any level of verbosity <= 5.
Help! I have no idea what else I can test or do to try to understand and solve this issue!
Thank you for any help/ideas you can provide. I'm really at the end of my rope with this one.
