Bad (sub-100Mbit/s) performance over 10Gb/s fiber

rtokarek

We have a (currently 1gigabit ethernet) network we want to bridge to another site. We have one fiber link and will soon have a backup point-to-point wireless link. We want to use a VPN tunnel over these longer-distance connections to bridge a portion of our internal network between the two sites. We are using two Linux Cent OS 7.3 systems with a 10Gb/s fiber connection between them. We have the bridge and tap interfaces setup. We get traffic across the VPN, but the performance is awful (40-600Mb/s level awful depending on the setup described below).

Can somebody suggest a place to start with improving this? I feel like we must be missing something fundamental for performance to be this bad.

(10.128.0.0/16) 1gb < -- > bridge - vpn-server 
   <-- 10gb fiber  (192.168.168.0/29) openvpn end points -->
vpn-client bridge < -- > 1gb (extended 10.128.0.0/16)

Running iperf3 between vpn-server and vpn-client on the 10Gfiber link, we get very near full speed (9.9Gb/s). The MTU is set to 9000. No other special configuration has been done with networking to get that.

Running iperf3 between anywhere on the 10.128net to anything on the other side of the VPN in that setup, we get right around 40Mb/s. (Directly from vpn-server to vpn-client, from some machine elsewhere on the "server" side to anything on the "client" side, it doesn't matter; if it crosses the VPN tunnel, it's that slow.) It seems reliable (no dropped pings, no connection troubles), just really slow.

By dropping the fiber link MTU to 1500, switching to TCP for OpenVPN instead of UDP, and by setting large sndbuf/rcvbuf, we can get that to about 600Mb/s. Setting cipher from AES-256-CBC to AES-128-CBC or even to "none" makes no performance difference so encryption overhead doesn't seem to be an issue. Various values and combinations of txqueuelen, link-mtu, tun-mtu, fragment, mssfix, mtu-test (BTW, picked 1538 over a 9000 MTU link), and mtu-disc fail to improve performance noticeably (certain combinations are worse). CPU usage is notable with traffic, but not full at between 10-20% of a Intel Xeon E3-1245 v5 @ 3.50GHz core (1CPU/4 cores per machine; hyper threading off, though no difference with HT on). We tried the stock 3.10.0 Cent OS Linux Kernel as well as 4.9.0 (no difference).

We've tried both openvpn 2.3.14 (from EPEL) and a build of 2.4.0 (identical versions on each side). This is the first time we've tried OpenVPN across a such a fast link. We have other working tunnels at up to 100Mb/s that are apparently only limited by our external network connections. I did try a routed tun (no tap-bridging) over the 10Gb fiber, and there was no substantial throughput difference from the bridge configuration.

I'd have expected at least 4Gbs and hoped for 8Gb/s; 40Mb/s over superficially good settings, and 600Mb/s under what would otherwise seem to be suboptimal settings is really tough to swallow. This has to be some sort of packet fragmentation, reassembly, MTU, queue length, buffer, polling something-something issue, but I don't know what.

br0 and tap0 are defined in network initialization scripts on each machine. The 10.128.0.0/16 IP (the network to be bridged) is bound to br0. tap0 and the relevant ethernet (1Gb/s) interface are joined to br0. The 10Gb/s fiber ethernet link has one of 192.168.168.[1 or 2] bound (.1 is "server", .2 is "client"). No firewalls are currently configured, and net.ipv4.ip_forward is 0.

Configs and logs:
local 192.168.168.1
port 1194
proto udp
dev tap0
ca CA.pem
cert server-cert.crt
key server-cert.key
dh dh2048.pem
server-bridge
keepalive 5 30
tls-auth ta.key 0
cipher AES-256-CBC
persist-key
persist-tun
status /var/log/openvpn/server-status.log
log-append /var/log/openvpn/server.log
verb 4
tls-server
#fast-io


/var/log/openvpn/server.log:

Wed Jan  4 23:23:56 2017 us=887590 Current Parameter Settings:
Wed Jan  4 23:23:56 2017 us=887622   config = '/etc/openvpn/server.conf'
Wed Jan  4 23:23:56 2017 us=887628   mode = 1
Wed Jan  4 23:23:56 2017 us=887632   persist_config = DISABLED
Wed Jan  4 23:23:56 2017 us=887636   persist_mode = 1
Wed Jan  4 23:23:56 2017 us=887639   show_ciphers = DISABLED
Wed Jan  4 23:23:56 2017 us=887643   show_digests = DISABLED
Wed Jan  4 23:23:56 2017 us=887646   show_engines = DISABLED
Wed Jan  4 23:23:56 2017 us=887650   genkey = DISABLED
Wed Jan  4 23:23:56 2017 us=887653   key_pass_file = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887657   show_tls_ciphers = DISABLED
Wed Jan  4 23:23:56 2017 us=887660   connect_retry_max = 0
Wed Jan  4 23:23:56 2017 us=887664 Connection profiles [0]:
Wed Jan  4 23:23:56 2017 us=887667   proto = udp
Wed Jan  4 23:23:56 2017 us=887671   local = '192.168.168.1'
Wed Jan  4 23:23:56 2017 us=887674   local_port = '1194'
Wed Jan  4 23:23:56 2017 us=887678   remote = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887681   remote_port = '1194'
Wed Jan  4 23:23:56 2017 us=887684   remote_float = DISABLED
Wed Jan  4 23:23:56 2017 us=887688   bind_defined = DISABLED
Wed Jan  4 23:23:56 2017 us=887691   bind_local = ENABLED
Wed Jan  4 23:23:56 2017 us=887694   bind_ipv6_only = DISABLED
Wed Jan  4 23:23:56 2017 us=887698   connect_retry_seconds = 5
Wed Jan  4 23:23:56 2017 us=887701   connect_timeout = 120
Wed Jan  4 23:23:56 2017 us=887704   socks_proxy_server = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887708   socks_proxy_port = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887711   tun_mtu = 1500
Wed Jan  4 23:23:56 2017 us=887715   tun_mtu_defined = ENABLED
Wed Jan  4 23:23:56 2017 us=887718   link_mtu = 1500
Wed Jan  4 23:23:56 2017 us=887721   link_mtu_defined = DISABLED
Wed Jan  4 23:23:56 2017 us=887725   tun_mtu_extra = 32
Wed Jan  4 23:23:56 2017 us=887728   tun_mtu_extra_defined = ENABLED
Wed Jan  4 23:23:56 2017 us=887731   mtu_discover_type = -1
Wed Jan  4 23:23:56 2017 us=887735   fragment = 0
Wed Jan  4 23:23:56 2017 us=887739   mssfix = 1450
Wed Jan  4 23:23:56 2017 us=887743   explicit_exit_notification = 0
Wed Jan  4 23:23:56 2017 us=887746 Connection profiles END
Wed Jan  4 23:23:56 2017 us=887750   remote_random = DISABLED
Wed Jan  4 23:23:56 2017 us=887754   ipchange = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887757   dev = 'tap0'
Wed Jan  4 23:23:56 2017 us=887760   dev_type = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887764   dev_node = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887767   lladdr = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887770   topology = 1
Wed Jan  4 23:23:56 2017 us=887774   ifconfig_local = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887777   ifconfig_remote_netmask = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887780   ifconfig_noexec = DISABLED
Wed Jan  4 23:23:56 2017 us=887784   ifconfig_nowarn = DISABLED
Wed Jan  4 23:23:56 2017 us=887787   ifconfig_ipv6_local = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887790   ifconfig_ipv6_netbits = 0
Wed Jan  4 23:23:56 2017 us=887794   ifconfig_ipv6_remote = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887797   shaper = 0
Wed Jan  4 23:23:56 2017 us=887800   mtu_test = 0
Wed Jan  4 23:23:56 2017 us=887804   mlock = DISABLED
Wed Jan  4 23:23:56 2017 us=887807   keepalive_ping = 5
Wed Jan  4 23:23:56 2017 us=887811   keepalive_timeout = 30
Wed Jan  4 23:23:56 2017 us=887814   inactivity_timeout = 0
Wed Jan  4 23:23:56 2017 us=887817   ping_send_timeout = 5
Wed Jan  4 23:23:56 2017 us=887821   ping_rec_timeout = 60
Wed Jan  4 23:23:56 2017 us=887824   ping_rec_timeout_action = 2
Wed Jan  4 23:23:56 2017 us=887828   ping_timer_remote = DISABLED
Wed Jan  4 23:23:56 2017 us=887831   remap_sigusr1 = 0
Wed Jan  4 23:23:56 2017 us=887834   persist_tun = ENABLED
Wed Jan  4 23:23:56 2017 us=887838   persist_local_ip = DISABLED
Wed Jan  4 23:23:56 2017 us=887841   persist_remote_ip = DISABLED
Wed Jan  4 23:23:56 2017 us=887844   persist_key = ENABLED
Wed Jan  4 23:23:56 2017 us=887848   passtos = DISABLED
Wed Jan  4 23:23:56 2017 us=887851   resolve_retry_seconds = 1000000000
Wed Jan  4 23:23:56 2017 us=887855   resolve_in_advance = DISABLED
Wed Jan  4 23:23:56 2017 us=887861   username = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887864   groupname = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887868   chroot_dir = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887871   cd_dir = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887874   writepid = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887878   up_script = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887881   down_script = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887885   down_pre = DISABLED
Wed Jan  4 23:23:56 2017 us=887888   up_restart = DISABLED
Wed Jan  4 23:23:56 2017 us=887891   up_delay = DISABLED
Wed Jan  4 23:23:56 2017 us=887894   daemon = DISABLED
Wed Jan  4 23:23:56 2017 us=887898   inetd = 0
Wed Jan  4 23:23:56 2017 us=887901   log = ENABLED
Wed Jan  4 23:23:56 2017 us=887905   suppress_timestamps = DISABLED
Wed Jan  4 23:23:56 2017 us=887908   machine_readable_output = DISABLED
Wed Jan  4 23:23:56 2017 us=887911   nice = 0
Wed Jan  4 23:23:56 2017 us=887915   verbosity = 4
Wed Jan  4 23:23:56 2017 us=887918   mute = 0
Wed Jan  4 23:23:56 2017 us=887921   gremlin = 0
Wed Jan  4 23:23:56 2017 us=887925   status_file = '/var/log/openvpn/server-status.log'
Wed Jan  4 23:23:56 2017 us=887929   status_file_version = 1
Wed Jan  4 23:23:56 2017 us=887932   status_file_update_freq = 60
Wed Jan  4 23:23:56 2017 us=887935   occ = ENABLED
Wed Jan  4 23:23:56 2017 us=887939   rcvbuf = 0
Wed Jan  4 23:23:56 2017 us=887942   sndbuf = 0
Wed Jan  4 23:23:56 2017 us=887945   mark = 0
Wed Jan  4 23:23:56 2017 us=887948   sockflags = 0
Wed Jan  4 23:23:56 2017 us=887952   fast_io = DISABLED
Wed Jan  4 23:23:56 2017 us=887955   comp.alg = 0
Wed Jan  4 23:23:56 2017 us=887958   comp.flags = 0
Wed Jan  4 23:23:56 2017 us=887962   route_script = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887965   route_default_gateway = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887969   route_default_metric = 0
Wed Jan  4 23:23:56 2017 us=887972   route_noexec = DISABLED
Wed Jan  4 23:23:56 2017 us=887975   route_delay = 0
Wed Jan  4 23:23:56 2017 us=887979   route_delay_window = 30
Wed Jan  4 23:23:56 2017 us=887982   route_delay_defined = DISABLED
Wed Jan  4 23:23:56 2017 us=887986   route_nopull = DISABLED
Wed Jan  4 23:23:56 2017 us=887989   route_gateway_via_dhcp = DISABLED
Wed Jan  4 23:23:56 2017 us=887992   allow_pull_fqdn = DISABLED
Wed Jan  4 23:23:56 2017 us=887996   management_addr = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=887999   management_port = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888003   management_user_pass = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888006   management_log_history_cache = 250
Wed Jan  4 23:23:56 2017 us=888009   management_echo_buffer_size = 100
Wed Jan  4 23:23:56 2017 us=888013   management_write_peer_info_file = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888016   management_client_user = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888020   management_client_group = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888023   management_flags = 0
Wed Jan  4 23:23:56 2017 us=888027   shared_secret_file = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888030   key_direction = 1
Wed Jan  4 23:23:56 2017 us=888034   ciphername = 'AES-256-CBC'
Wed Jan  4 23:23:56 2017 us=888037   ncp_enabled = ENABLED
Wed Jan  4 23:23:56 2017 us=888041   ncp_ciphers = 'AES-256-GCM:AES-128-GCM'
Wed Jan  4 23:23:56 2017 us=888044   authname = 'SHA1'
Wed Jan  4 23:23:56 2017 us=888047   prng_hash = 'SHA1'
Wed Jan  4 23:23:56 2017 us=888051   prng_nonce_secret_len = 16
Wed Jan  4 23:23:56 2017 us=888054   keysize = 0
Wed Jan  4 23:23:56 2017 us=888058   engine = DISABLED
Wed Jan  4 23:23:56 2017 us=888061   replay = ENABLED
Wed Jan  4 23:23:56 2017 us=888065   mute_replay_warnings = DISABLED
Wed Jan  4 23:23:56 2017 us=888068   replay_window = 64
Wed Jan  4 23:23:56 2017 us=888071   replay_time = 15
Wed Jan  4 23:23:56 2017 us=888075   packet_id_file = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888078   use_iv = ENABLED
Wed Jan  4 23:23:56 2017 us=888081   test_crypto = DISABLED
Wed Jan  4 23:23:56 2017 us=888085   tls_server = ENABLED
Wed Jan  4 23:23:56 2017 us=888088   tls_client = DISABLED
Wed Jan  4 23:23:56 2017 us=888094   key_method = 2
Wed Jan  4 23:23:56 2017 us=888097   ca_file = 'CA.pem'
Wed Jan  4 23:23:56 2017 us=888100   ca_path = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888104   dh_file = 'dh2048.pem'
Wed Jan  4 23:23:56 2017 us=888107   cert_file = 'server-cert.crt'
Wed Jan  4 23:23:56 2017 us=888111   extra_certs_file = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888115   priv_key_file = 'server-cert.key'
Wed Jan  4 23:23:56 2017 us=888118   pkcs12_file = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888122   cipher_list = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888125   tls_verify = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888128   tls_export_cert = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888132   verify_x509_type = 0
Wed Jan  4 23:23:56 2017 us=888135   verify_x509_name = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888138   crl_file = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888142   ns_cert_type = 0
Wed Jan  4 23:23:56 2017 us=888145   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888148   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888151   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888155   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888158   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888161   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888164   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888168   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888171   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888174   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888177   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888181   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888184   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888187   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888190   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888193   remote_cert_ku[i] = 0
Wed Jan  4 23:23:56 2017 us=888197   remote_cert_eku = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888200   ssl_flags = 0
Wed Jan  4 23:23:56 2017 us=888204   tls_timeout = 2
Wed Jan  4 23:23:56 2017 us=888207   renegotiate_bytes = -1
Wed Jan  4 23:23:56 2017 us=888210   renegotiate_packets = 0
Wed Jan  4 23:23:56 2017 us=888214   renegotiate_seconds = 3600
Wed Jan  4 23:23:56 2017 us=888217   handshake_window = 60
Wed Jan  4 23:23:56 2017 us=888220   transition_window = 3600
Wed Jan  4 23:23:56 2017 us=888224   single_session = DISABLED
Wed Jan  4 23:23:56 2017 us=888227   push_peer_info = DISABLED
Wed Jan  4 23:23:56 2017 us=888230   tls_exit = DISABLED
Wed Jan  4 23:23:56 2017 us=888234   tls_auth_file = 'ta.key'
Wed Jan  4 23:23:56 2017 us=888237   tls_crypt_file = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888241   pkcs11_protected_authentication = DISABLED
...
Wed Jan  4 23:23:56 2017 us=888294   pkcs11_private_mode = 00000000
...
Wed Jan  4 23:23:56 2017 us=888345   pkcs11_private_mode = 00000000
Wed Jan  4 23:23:56 2017 us=888348   pkcs11_cert_private = DISABLED
...
Wed Jan  4 23:23:56 2017 us=888401   pkcs11_pin_cache_period = -1
Wed Jan  4 23:23:56 2017 us=888404   pkcs11_id = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888414   pkcs11_id_management = DISABLED
Wed Jan  4 23:23:56 2017 us=888423   server_network = 0.0.0.0
Wed Jan  4 23:23:56 2017 us=888427   server_netmask = 0.0.0.0
Wed Jan  4 23:23:56 2017 us=888432   server_network_ipv6 = ::
Wed Jan  4 23:23:56 2017 us=888436   server_netbits_ipv6 = 0
Wed Jan  4 23:23:56 2017 us=888440   server_bridge_ip = 0.0.0.0
Wed Jan  4 23:23:56 2017 us=888444   server_bridge_netmask = 0.0.0.0
Wed Jan  4 23:23:56 2017 us=888448   server_bridge_pool_start = 0.0.0.0
Wed Jan  4 23:23:56 2017 us=888452   server_bridge_pool_end = 0.0.0.0
Wed Jan  4 23:23:56 2017 us=888455   push_entry = 'route-gateway dhcp'
Wed Jan  4 23:23:56 2017 us=888459   push_entry = 'ping 5'
Wed Jan  4 23:23:56 2017 us=888462   push_entry = 'ping-restart 30'
Wed Jan  4 23:23:56 2017 us=888466   ifconfig_pool_defined = DISABLED
Wed Jan  4 23:23:56 2017 us=888469   ifconfig_pool_start = 0.0.0.0
Wed Jan  4 23:23:56 2017 us=888473   ifconfig_pool_end = 0.0.0.0
Wed Jan  4 23:23:56 2017 us=888477   ifconfig_pool_netmask = 0.0.0.0
Wed Jan  4 23:23:56 2017 us=888481   ifconfig_pool_persist_filename = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888484   ifconfig_pool_persist_refresh_freq = 600
Wed Jan  4 23:23:56 2017 us=888487   ifconfig_ipv6_pool_defined = DISABLED
Wed Jan  4 23:23:56 2017 us=888491   ifconfig_ipv6_pool_base = ::
Wed Jan  4 23:23:56 2017 us=888495   ifconfig_ipv6_pool_netbits = 0
Wed Jan  4 23:23:56 2017 us=888498   n_bcast_buf = 256
Wed Jan  4 23:23:56 2017 us=888502   tcp_queue_limit = 64
Wed Jan  4 23:23:56 2017 us=888505   real_hash_size = 256
Wed Jan  4 23:23:56 2017 us=888508   virtual_hash_size = 256
Wed Jan  4 23:23:56 2017 us=888512   client_connect_script = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888515   learn_address_script = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888519   client_disconnect_script = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888522   client_config_dir = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888527   ccd_exclusive = DISABLED
Wed Jan  4 23:23:56 2017 us=888531   tmp_dir = '/tmp'
Wed Jan  4 23:23:56 2017 us=888534   push_ifconfig_defined = DISABLED
Wed Jan  4 23:23:56 2017 us=888538   push_ifconfig_local = 0.0.0.0
Wed Jan  4 23:23:56 2017 us=888542   push_ifconfig_remote_netmask = 0.0.0.0
Wed Jan  4 23:23:56 2017 us=888546   push_ifconfig_ipv6_defined = DISABLED
Wed Jan  4 23:23:56 2017 us=888549   push_ifconfig_ipv6_local = ::/0
Wed Jan  4 23:23:56 2017 us=888553   push_ifconfig_ipv6_remote = ::
Wed Jan  4 23:23:56 2017 us=888556   enable_c2c = DISABLED
Wed Jan  4 23:23:56 2017 us=888560   duplicate_cn = DISABLED
Wed Jan  4 23:23:56 2017 us=888563   cf_max = 0
Wed Jan  4 23:23:56 2017 us=888566   cf_per = 0
Wed Jan  4 23:23:56 2017 us=888570   max_clients = 1024
Wed Jan  4 23:23:56 2017 us=888573   max_routes_per_client = 256
Wed Jan  4 23:23:56 2017 us=888577   auth_user_pass_verify_script = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888580   auth_user_pass_verify_script_via_file = DISABLED
Wed Jan  4 23:23:56 2017 us=888584   auth_token_generate = DISABLED
Wed Jan  4 23:23:56 2017 us=888587   auth_token_lifetime = 0
Wed Jan  4 23:23:56 2017 us=888591   port_share_host = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888594   port_share_port = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888598   client = DISABLED
Wed Jan  4 23:23:56 2017 us=888601   pull = DISABLED
Wed Jan  4 23:23:56 2017 us=888604   auth_user_pass_file = '[UNDEF]'
Wed Jan  4 23:23:56 2017 us=888609 OpenVPN 2.4.0 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jan  4 2017
Wed Jan  4 23:23:56 2017 us=888615 library versions: OpenSSL 1.0.1e-fips 11 Feb 2013, LZO 2.06
Wed Jan  4 23:23:56 2017 us=888674 NOTE: when bridging your LAN adapter with the TAP adapter, note that the new bridge adapter will often take on its own IP address that is different from what the LAN adapter was previously set to
Wed Jan  4 23:23:56 2017 us=892737 Diffie-Hellman initialized with 2048 bit key
Wed Jan  4 23:23:56 2017 us=892883 Failed to extract curve from certificate (UNDEF), using secp384r1 instead.
Wed Jan  4 23:23:56 2017 us=892892 ECDH curve secp384r1 added
Wed Jan  4 23:23:56 2017 us=892967 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Wed Jan  4 23:23:56 2017 us=892972 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Wed Jan  4 23:23:56 2017 us=892978 TLS-Auth MTU parms [ L:1653 D:1184 EF:66 EB:0 ET:0 EL:3 ]
Wed Jan  4 23:23:56 2017 us=893541 TUN/TAP device tap0 opened
Wed Jan  4 23:23:56 2017 us=893556 TUN/TAP TX queue length set to 100
Wed Jan  4 23:23:56 2017 us=893573 Data Channel MTU parms [ L:1653 D:1450 EF:121 EB:411 ET:32 EL:3 ]
Wed Jan  4 23:23:56 2017 us=893577 Could not determine IPv4/IPv6 protocol. Using AF_INET
Wed Jan  4 23:23:56 2017 us=893582 Socket Buffers: R=[212992->212992] S=[212992->212992]
Wed Jan  4 23:23:56 2017 us=893602 UDPv4 link local (bound): [AF_INET]192.168.168.1:1194
Wed Jan  4 23:23:56 2017 us=893605 UDPv4 link remote: [AF_UNSPEC]
Wed Jan  4 23:23:56 2017 us=893610 MULTI: multi_init called, r=256 v=256
Wed Jan  4 23:23:56 2017 us=893625 Initialization Sequence Completed
Wed Jan  4 23:23:58 2017 us=136217 MULTI: multi_create_instance called
Wed Jan  4 23:23:58 2017 us=136259 192.168.168.2:1194 Re-using SSL/TLS context
Wed Jan  4 23:23:58 2017 us=136310 192.168.168.2:1194 Control Channel MTU parms [ L:1653 D:1184 EF:66 EB:0 ET:0 EL:3 ]
Wed Jan  4 23:23:58 2017 us=136317 192.168.168.2:1194 Data Channel MTU parms [ L:1653 D:1450 EF:121 EB:411 ET:32 EL:3 ]
Wed Jan  4 23:23:58 2017 us=136331 192.168.168.2:1194 Local Options String (VER=V4): 'V4,dev-type tap,link-mtu 1589,tun-mtu 1532,proto UDPv4,keydir 0,cipher AES-256-CBC,auth SHA1,keysize 256,tls-auth,key-method 2,tls-server'
Wed Jan  4 23:23:58 2017 us=136336 192.168.168.2:1194 Expected Remote Options String (VER=V4): 'V4,dev-type tap,link-mtu 1589,tun-mtu 1532,proto UDPv4,keydir 1,cipher AES-256-CBC,auth SHA1,keysize 256,tls-auth,key-method 2,tls-client'
Wed Jan  4 23:23:58 2017 us=136353 192.168.168.2:1194 TLS: Initial packet from [AF_INET]192.168.168.2:1194, sid=b3947754 cccb11bb
Wed Jan  4 23:23:58 2017 us=152877 192.168.168.2:1194 VERIFY OK: depth=1, CN=(CA NAME)
Wed Jan  4 23:23:58 2017 us=152982 192.168.168.2:1194 VERIFY OK: depth=0, CN=(vpn-server)
Wed Jan  4 23:23:58 2017 us=157256 192.168.168.2:1194 peer info: IV_VER=2.4.0
Wed Jan  4 23:23:58 2017 us=157268 192.168.168.2:1194 peer info: IV_PLAT=linux
Wed Jan  4 23:23:58 2017 us=157273 192.168.168.2:1194 peer info: IV_PROTO=2
Wed Jan  4 23:23:58 2017 us=157277 192.168.168.2:1194 peer info: IV_NCP=2
Wed Jan  4 23:23:58 2017 us=157280 192.168.168.2:1194 peer info: IV_LZ4=1
Wed Jan  4 23:23:58 2017 us=157284 192.168.168.2:1194 peer info: IV_LZ4v2=1
Wed Jan  4 23:23:58 2017 us=157287 192.168.168.2:1194 peer info: IV_LZO=1
Wed Jan  4 23:23:58 2017 us=157291 192.168.168.2:1194 peer info: IV_COMP_STUB=1
Wed Jan  4 23:23:58 2017 us=157294 192.168.168.2:1194 peer info: IV_COMP_STUBv2=1
Wed Jan  4 23:23:58 2017 us=157298 192.168.168.2:1194 peer info: IV_TCPNL=1
Wed Jan  4 23:23:58 2017 us=160735 192.168.168.2:1194 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 ECDHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
Wed Jan  4 23:23:58 2017 us=160775 192.168.168.2:1194 [vpn-server] Peer Connection Initiated with [AF_INET]192.168.168.2:1194
Wed Jan  4 23:23:58 2017 us=160799 client-vpn/192.168.168.2:1194 MULTI: no dynamic or static remote --ifconfig address is available for client-vpn/192.168.168.2:1194
Wed Jan  4 23:23:59 2017 us=358173 client-vpn/192.168.168.2:1194 PUSH: Received control message: 'PUSH_REQUEST'
Wed Jan  4 23:23:59 2017 us=358191 client-vpn/192.168.168.2:1194 SENT CONTROL [client-vpn]: 'PUSH_REPLY,route-gateway dhcp,ping 5,ping-restart 30,peer-id 0,cipher AES-256-GCM' (status=1)
Wed Jan  4 23:23:59 2017 us=358200 client-vpn/192.168.168.2:1194 Data Channel MTU parms [ L:1581 D:1450 EF:49 EB:411 ET:32 EL:3 ]
Wed Jan  4 23:23:59 2017 us=358249 client-vpn/192.168.168.2:1194 Data Channel Encrypt: Cipher 'AES-256-GCM' initialized with 256 bit key
Wed Jan  4 23:23:59 2017 us=358254 client-vpn/192.168.168.2:1194 Data Channel Decrypt: Cipher 'AES-256-GCM' initialized with 256 bit key
Wed Jan  4 23:24:00 2017 us=186584 client-vpn/192.168.168.2:1194 MULTI: Learn: 0c:c4:7a:90:d3:0d -> client-vpn/192.168.168.2:1194

client
remote 192.168.168.1 1194
remote-cert-tls server
proto udp
dev tap0
ca CA.pem
cert client-cert.crt
key client-key.key
dh dh2048.pem
keepalive 5 30
tls-auth ta.key 1
cipher AES-256-CBC
persist-key
persist-tun
status /var/log/openvpn/client-status.log
log-append /var/log/openvpn/client.log
verb 4
mute 20
resolv-retry infinite
tls-client
#fast-io


/var/log/openvpn/client.log:

Wed Jan  4 23:23:58 2017 us=125242 Current Parameter Settings:
Wed Jan  4 23:23:58 2017 us=125272   config = '/etc/openvpn/server.conf'
Wed Jan  4 23:23:58 2017 us=125278   mode = 0
Wed Jan  4 23:23:58 2017 us=125282   persist_config = DISABLED
Wed Jan  4 23:23:58 2017 us=125285   persist_mode = 1
Wed Jan  4 23:23:58 2017 us=125289   show_ciphers = DISABLED
Wed Jan  4 23:23:58 2017 us=125292   show_digests = DISABLED
Wed Jan  4 23:23:58 2017 us=125295   show_engines = DISABLED
Wed Jan  4 23:23:58 2017 us=125298   genkey = DISABLED
Wed Jan  4 23:23:58 2017 us=125302   key_pass_file = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125305   show_tls_ciphers = DISABLED
Wed Jan  4 23:23:58 2017 us=125309   connect_retry_max = 0
Wed Jan  4 23:23:58 2017 us=125312 Connection profiles [0]:
Wed Jan  4 23:23:58 2017 us=125316   proto = udp
Wed Jan  4 23:23:58 2017 us=125319   local = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125323   local_port = '1194'
Wed Jan  4 23:23:58 2017 us=125326   remote = '192.168.168.1'
Wed Jan  4 23:23:58 2017 us=125329   remote_port = '1194'
Wed Jan  4 23:23:58 2017 us=125332   remote_float = DISABLED
Wed Jan  4 23:23:58 2017 us=125335   bind_defined = DISABLED
Wed Jan  4 23:23:58 2017 us=125339   bind_local = ENABLED
Wed Jan  4 23:23:58 2017 us=125342   bind_ipv6_only = DISABLED
Wed Jan  4 23:23:58 2017 us=125345   connect_retry_seconds = 5
Wed Jan  4 23:23:58 2017 us=125348   connect_timeout = 120
Wed Jan  4 23:23:58 2017 us=125352   socks_proxy_server = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125355   socks_proxy_port = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125358   tun_mtu = 1500
Wed Jan  4 23:23:58 2017 us=125362   tun_mtu_defined = ENABLED
Wed Jan  4 23:23:58 2017 us=125366   link_mtu = 1500
Wed Jan  4 23:23:58 2017 us=125369   link_mtu_defined = DISABLED
Wed Jan  4 23:23:58 2017 us=125372   tun_mtu_extra = 32
Wed Jan  4 23:23:58 2017 us=125375   tun_mtu_extra_defined = ENABLED
Wed Jan  4 23:23:58 2017 us=125379   mtu_discover_type = -1
Wed Jan  4 23:23:58 2017 us=125382   fragment = 0
Wed Jan  4 23:23:58 2017 us=125385   mssfix = 1450
Wed Jan  4 23:23:58 2017 us=125388   explicit_exit_notification = 0
Wed Jan  4 23:23:58 2017 us=125391 Connection profiles END
Wed Jan  4 23:23:58 2017 us=125395   remote_random = DISABLED
Wed Jan  4 23:23:58 2017 us=125398   ipchange = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125401   dev = 'tap0'
Wed Jan  4 23:23:58 2017 us=125404   dev_type = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125407   dev_node = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125411   lladdr = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125414   topology = 1
Wed Jan  4 23:23:58 2017 us=125417   ifconfig_local = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125420   ifconfig_remote_netmask = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125424   ifconfig_noexec = DISABLED
Wed Jan  4 23:23:58 2017 us=125427   ifconfig_nowarn = DISABLED
Wed Jan  4 23:23:58 2017 us=125430   ifconfig_ipv6_local = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125433   ifconfig_ipv6_netbits = 0
Wed Jan  4 23:23:58 2017 us=125436   ifconfig_ipv6_remote = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125440   shaper = 0
Wed Jan  4 23:23:58 2017 us=125443   mtu_test = 0
Wed Jan  4 23:23:58 2017 us=125446   mlock = DISABLED
Wed Jan  4 23:23:58 2017 us=125449   keepalive_ping = 5
Wed Jan  4 23:23:58 2017 us=125452   keepalive_timeout = 30
Wed Jan  4 23:23:58 2017 us=125456   inactivity_timeout = 0
Wed Jan  4 23:23:58 2017 us=125459   ping_send_timeout = 5
Wed Jan  4 23:23:58 2017 us=125462   ping_rec_timeout = 30
Wed Jan  4 23:23:58 2017 us=125465   ping_rec_timeout_action = 2
Wed Jan  4 23:23:58 2017 us=125469   ping_timer_remote = DISABLED
Wed Jan  4 23:23:58 2017 us=125472   remap_sigusr1 = 0
Wed Jan  4 23:23:58 2017 us=125475   persist_tun = ENABLED
Wed Jan  4 23:23:58 2017 us=125478   persist_local_ip = DISABLED
Wed Jan  4 23:23:58 2017 us=125481   persist_remote_ip = DISABLED
Wed Jan  4 23:23:58 2017 us=125485   persist_key = ENABLED
Wed Jan  4 23:23:58 2017 us=125488   passtos = DISABLED
Wed Jan  4 23:23:58 2017 us=125491   resolve_retry_seconds = 1000000000
Wed Jan  4 23:23:58 2017 us=125494   resolve_in_advance = DISABLED
Wed Jan  4 23:23:58 2017 us=125500   username = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125503   groupname = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125506   chroot_dir = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125510   cd_dir = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125513   writepid = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125516   up_script = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125519   down_script = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125523   down_pre = DISABLED
Wed Jan  4 23:23:58 2017 us=125526   up_restart = DISABLED
Wed Jan  4 23:23:58 2017 us=125529   up_delay = DISABLED
Wed Jan  4 23:23:58 2017 us=125532   daemon = DISABLED
Wed Jan  4 23:23:58 2017 us=125535   inetd = 0
Wed Jan  4 23:23:58 2017 us=125539   log = ENABLED
Wed Jan  4 23:23:58 2017 us=125542   suppress_timestamps = DISABLED
Wed Jan  4 23:23:58 2017 us=125545   machine_readable_output = DISABLED
Wed Jan  4 23:23:58 2017 us=125548   nice = 0
Wed Jan  4 23:23:58 2017 us=125552   verbosity = 4
Wed Jan  4 23:23:58 2017 us=125555   mute = 0
Wed Jan  4 23:23:58 2017 us=125558   gremlin = 0
Wed Jan  4 23:23:58 2017 us=125561   status_file = '/var/log/openvpn/client-status.log'
Wed Jan  4 23:23:58 2017 us=125565   status_file_version = 1
Wed Jan  4 23:23:58 2017 us=125568   status_file_update_freq = 60
Wed Jan  4 23:23:58 2017 us=125571   occ = ENABLED
Wed Jan  4 23:23:58 2017 us=125574   rcvbuf = 0
Wed Jan  4 23:23:58 2017 us=125577   sndbuf = 0
Wed Jan  4 23:23:58 2017 us=125581   mark = 0
Wed Jan  4 23:23:58 2017 us=125584   sockflags = 0
Wed Jan  4 23:23:58 2017 us=125587   fast_io = DISABLED
Wed Jan  4 23:23:58 2017 us=125590   comp.alg = 0
Wed Jan  4 23:23:58 2017 us=125593   comp.flags = 0
Wed Jan  4 23:23:58 2017 us=125596   route_script = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125600   route_default_gateway = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125603   route_default_metric = 0
Wed Jan  4 23:23:58 2017 us=125606   route_noexec = DISABLED
Wed Jan  4 23:23:58 2017 us=125610   route_delay = 0
Wed Jan  4 23:23:58 2017 us=125613   route_delay_window = 30
Wed Jan  4 23:23:58 2017 us=125616   route_delay_defined = DISABLED
Wed Jan  4 23:23:58 2017 us=125620   route_nopull = DISABLED
Wed Jan  4 23:23:58 2017 us=125623   route_gateway_via_dhcp = DISABLED
Wed Jan  4 23:23:58 2017 us=125626   allow_pull_fqdn = DISABLED
Wed Jan  4 23:23:58 2017 us=125629   management_addr = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125633   management_port = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125636   management_user_pass = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125639   management_log_history_cache = 250
Wed Jan  4 23:23:58 2017 us=125643   management_echo_buffer_size = 100
Wed Jan  4 23:23:58 2017 us=125646   management_write_peer_info_file = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125649   management_client_user = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125653   management_client_group = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125656   management_flags = 0
Wed Jan  4 23:23:58 2017 us=125659   shared_secret_file = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125663   key_direction = 2
Wed Jan  4 23:23:58 2017 us=125666   ciphername = 'AES-256-CBC'
Wed Jan  4 23:23:58 2017 us=125669   ncp_enabled = ENABLED
Wed Jan  4 23:23:58 2017 us=125672   ncp_ciphers = 'AES-256-GCM:AES-128-GCM'
Wed Jan  4 23:23:58 2017 us=125676   authname = 'SHA1'
Wed Jan  4 23:23:58 2017 us=125679   prng_hash = 'SHA1'
Wed Jan  4 23:23:58 2017 us=125682   prng_nonce_secret_len = 16
Wed Jan  4 23:23:58 2017 us=125686   keysize = 0
Wed Jan  4 23:23:58 2017 us=125689   engine = DISABLED
Wed Jan  4 23:23:58 2017 us=125692   replay = ENABLED
Wed Jan  4 23:23:58 2017 us=125695   mute_replay_warnings = DISABLED
Wed Jan  4 23:23:58 2017 us=125699   replay_window = 64
Wed Jan  4 23:23:58 2017 us=125702   replay_time = 15
Wed Jan  4 23:23:58 2017 us=125705   packet_id_file = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125708   use_iv = ENABLED
Wed Jan  4 23:23:58 2017 us=125712   test_crypto = DISABLED
Wed Jan  4 23:23:58 2017 us=125715   tls_server = DISABLED
Wed Jan  4 23:23:58 2017 us=125718   tls_client = ENABLED
Wed Jan  4 23:23:58 2017 us=125723   key_method = 2
Wed Jan  4 23:23:58 2017 us=125727   ca_file = 'WolframCA.pem'
Wed Jan  4 23:23:58 2017 us=125730   ca_path = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125733   dh_file = 'dh2048.pem'
Wed Jan  4 23:23:58 2017 us=125737   cert_file = 'client-cert.crt'
Wed Jan  4 23:23:58 2017 us=125740   extra_certs_file = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125743   priv_key_file = 'client-cert.key'
Wed Jan  4 23:23:58 2017 us=125747   pkcs12_file = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125750   cipher_list = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125754   tls_verify = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125757   tls_export_cert = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125760   verify_x509_type = 0
Wed Jan  4 23:23:58 2017 us=125763   verify_x509_name = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125767   crl_file = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125770   ns_cert_type = 0
Wed Jan  4 23:23:58 2017 us=125773   remote_cert_ku[i] = 160
Wed Jan  4 23:23:58 2017 us=125776   remote_cert_ku[i] = 136
Wed Jan  4 23:23:58 2017 us=125779   remote_cert_ku[i] = 0
...
Wed Jan  4 23:23:58 2017 us=125823   remote_cert_eku = 'TLS Web Server Authentication'
Wed Jan  4 23:23:58 2017 us=125827   ssl_flags = 0
Wed Jan  4 23:23:58 2017 us=125830   tls_timeout = 2
Wed Jan  4 23:23:58 2017 us=125833   renegotiate_bytes = -1
Wed Jan  4 23:23:58 2017 us=125837   renegotiate_packets = 0
Wed Jan  4 23:23:58 2017 us=125840   renegotiate_seconds = 3600
Wed Jan  4 23:23:58 2017 us=125843   handshake_window = 60
Wed Jan  4 23:23:58 2017 us=125846   transition_window = 3600
Wed Jan  4 23:23:58 2017 us=125849   single_session = DISABLED
Wed Jan  4 23:23:58 2017 us=125853   push_peer_info = DISABLED
Wed Jan  4 23:23:58 2017 us=125856   tls_exit = DISABLED
Wed Jan  4 23:23:58 2017 us=125859   tls_auth_file = 'ta.key'
Wed Jan  4 23:23:58 2017 us=125863   tls_crypt_file = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=125866   pkcs11_protected_authentication = DISABLED
...
Wed Jan  4 23:23:58 2017 us=125917   pkcs11_private_mode = 00000000
...
Wed Jan  4 23:23:58 2017 us=125969   pkcs11_cert_private = DISABLED
...
Wed Jan  4 23:23:58 2017 us=126026   pkcs11_pin_cache_period = -1
Wed Jan  4 23:23:58 2017 us=126029   pkcs11_id = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=126032   pkcs11_id_management = DISABLED
Wed Jan  4 23:23:58 2017 us=126037   server_network = 0.0.0.0
Wed Jan  4 23:23:58 2017 us=126041   server_netmask = 0.0.0.0
Wed Jan  4 23:23:58 2017 us=126045   server_network_ipv6 = ::
Wed Jan  4 23:23:58 2017 us=126048   server_netbits_ipv6 = 0
Wed Jan  4 23:23:58 2017 us=126052   server_bridge_ip = 0.0.0.0
Wed Jan  4 23:23:58 2017 us=126056   server_bridge_netmask = 0.0.0.0
Wed Jan  4 23:23:58 2017 us=126060   server_bridge_pool_start = 0.0.0.0
Wed Jan  4 23:23:58 2017 us=126063   server_bridge_pool_end = 0.0.0.0
Wed Jan  4 23:23:58 2017 us=126067   ifconfig_pool_defined = DISABLED
Wed Jan  4 23:23:58 2017 us=126070   ifconfig_pool_start = 0.0.0.0
Wed Jan  4 23:23:58 2017 us=126074   ifconfig_pool_end = 0.0.0.0
Wed Jan  4 23:23:58 2017 us=126078   ifconfig_pool_netmask = 0.0.0.0
Wed Jan  4 23:23:58 2017 us=126081   ifconfig_pool_persist_filename = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=126084   ifconfig_pool_persist_refresh_freq = 600
Wed Jan  4 23:23:58 2017 us=126088   ifconfig_ipv6_pool_defined = DISABLED
Wed Jan  4 23:23:58 2017 us=126093   ifconfig_ipv6_pool_base = ::
Wed Jan  4 23:23:58 2017 us=126096   ifconfig_ipv6_pool_netbits = 0
Wed Jan  4 23:23:58 2017 us=126100   n_bcast_buf = 256
Wed Jan  4 23:23:58 2017 us=126103   tcp_queue_limit = 64
Wed Jan  4 23:23:58 2017 us=126106   real_hash_size = 256
Wed Jan  4 23:23:58 2017 us=126110   virtual_hash_size = 256
Wed Jan  4 23:23:58 2017 us=126113   client_connect_script = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=126116   learn_address_script = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=126120   client_disconnect_script = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=126123   client_config_dir = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=126127   ccd_exclusive = DISABLED
Wed Jan  4 23:23:58 2017 us=126130   tmp_dir = '/tmp'
Wed Jan  4 23:23:58 2017 us=126133   push_ifconfig_defined = DISABLED
Wed Jan  4 23:23:58 2017 us=126139   push_ifconfig_local = 0.0.0.0
Wed Jan  4 23:23:58 2017 us=126143   push_ifconfig_remote_netmask = 0.0.0.0
Wed Jan  4 23:23:58 2017 us=126146   push_ifconfig_ipv6_defined = DISABLED
Wed Jan  4 23:23:58 2017 us=126150   push_ifconfig_ipv6_local = ::/0
Wed Jan  4 23:23:58 2017 us=126153   push_ifconfig_ipv6_remote = ::
Wed Jan  4 23:23:58 2017 us=126157   enable_c2c = DISABLED
Wed Jan  4 23:23:58 2017 us=126160   duplicate_cn = DISABLED
Wed Jan  4 23:23:58 2017 us=126163   cf_max = 0
Wed Jan  4 23:23:58 2017 us=126166   cf_per = 0
Wed Jan  4 23:23:58 2017 us=126169   max_clients = 1024
Wed Jan  4 23:23:58 2017 us=126173   max_routes_per_client = 256
Wed Jan  4 23:23:58 2017 us=126176   auth_user_pass_verify_script = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=126180   auth_user_pass_verify_script_via_file = DISABLED
Wed Jan  4 23:23:58 2017 us=126183   auth_token_generate = DISABLED
Wed Jan  4 23:23:58 2017 us=126186   auth_token_lifetime = 0
Wed Jan  4 23:23:58 2017 us=126189   port_share_host = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=126193   port_share_port = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=126196   client = ENABLED
Wed Jan  4 23:23:58 2017 us=126199   pull = ENABLED
Wed Jan  4 23:23:58 2017 us=126202   auth_user_pass_file = '[UNDEF]'
Wed Jan  4 23:23:58 2017 us=126207 OpenVPN 2.4.0 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jan  4 2017
Wed Jan  4 23:23:58 2017 us=126214 library versions: OpenSSL 1.0.1e-fips 11 Feb 2013, LZO 2.06
Wed Jan  4 23:23:58 2017 us=126617 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Wed Jan  4 23:23:58 2017 us=126625 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Wed Jan  4 23:23:58 2017 us=126651 Control Channel MTU parms [ L:1653 D:1184 EF:66 EB:0 ET:0 EL:3 ]
Wed Jan  4 23:23:58 2017 us=126716 Data Channel MTU parms [ L:1653 D:1450 EF:121 EB:411 ET:32 EL:3 ]
Wed Jan  4 23:23:58 2017 us=126731 Local Options String (VER=V4): 'V4,dev-type tap,link-mtu 1589,tun-mtu 1532,proto UDPv4,keydir 1,cipher AES-256-CBC,auth SHA1,keysize 256,tls-auth,key-method 2,tls-client'
Wed Jan  4 23:23:58 2017 us=126734 Expected Remote Options String (VER=V4): 'V4,dev-type tap,link-mtu 1589,tun-mtu 1532,proto UDPv4,keydir 0,cipher AES-256-CBC,auth SHA1,keysize 256,tls-auth,key-method 2,tls-server'
Wed Jan  4 23:23:58 2017 us=126742 TCP/UDP: Preserving recently used remote address: [AF_INET]192.168.168.1:1194
Wed Jan  4 23:23:58 2017 us=126749 Socket Buffers: R=[212992->212992] S=[212992->212992]
Wed Jan  4 23:23:58 2017 us=126754 UDP link local (bound): [AF_INET][undef]:1194
Wed Jan  4 23:23:58 2017 us=126758 UDP link remote: [AF_INET]192.168.168.1:1194
Wed Jan  4 23:23:58 2017 us=133813 TLS: Initial packet from [AF_INET]192.168.168.1:1194, sid=5dd59e40 526e30f6
Wed Jan  4 23:23:58 2017 us=141892 VERIFY OK: depth=1, CN=(Our CA)
Wed Jan  4 23:23:58 2017 us=142031 Validating certificate key usage
Wed Jan  4 23:23:58 2017 us=142041 ++ Certificate has key usage  00a0, expects 00a0
Wed Jan  4 23:23:58 2017 us=142045 VERIFY KU OK
Wed Jan  4 23:23:58 2017 us=142049 Validating certificate extended key usage
Wed Jan  4 23:23:58 2017 us=142053 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Wed Jan  4 23:23:58 2017 us=142056 VERIFY EKU OK
Wed Jan  4 23:23:58 2017 us=142060 VERIFY OK: depth=0, CN=vpn-server
Wed Jan  4 23:23:58 2017 us=155094 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 ECDHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
Wed Jan  4 23:23:58 2017 us=155113 [vpn-server] Peer Connection Initiated with [AF_INET]192.168.168.1:1194
Wed Jan  4 23:23:59 2017 us=352345 SENT CONTROL [vpn-server]: 'PUSH_REQUEST' (status=1)
Wed Jan  4 23:23:59 2017 us=356079 PUSH: Received control message: 'PUSH_REPLY,route-gateway dhcp,ping 5,ping-restart 30,peer-id 0,cipher AES-256-GCM'
Wed Jan  4 23:23:59 2017 us=356123 OPTIONS IMPORT: timers and/or timeouts modified
Wed Jan  4 23:23:59 2017 us=356129 OPTIONS IMPORT: route-related options modified
Wed Jan  4 23:23:59 2017 us=356133 OPTIONS IMPORT: peer-id set
Wed Jan  4 23:23:59 2017 us=356136 OPTIONS IMPORT: adjusting link_mtu to 1656
Wed Jan  4 23:23:59 2017 us=356139 OPTIONS IMPORT: data channel crypto options modified
Wed Jan  4 23:23:59 2017 us=356147 Data Channel MTU parms [ L:1584 D:1450 EF:52 EB:411 ET:32 EL:3 ]
Wed Jan  4 23:23:59 2017 us=356196 Data Channel Encrypt: Cipher 'AES-256-GCM' initialized with 256 bit key
Wed Jan  4 23:23:59 2017 us=356202 Data Channel Decrypt: Cipher 'AES-256-GCM' initialized with 256 bit key
Wed Jan  4 23:23:59 2017 us=356218 TUN/TAP device tap0 opened
Wed Jan  4 23:23:59 2017 us=356225 TUN/TAP TX queue length set to 100
Wed Jan  4 23:23:59 2017 us=356244 Initialization Sequence Completed

TinCanTech

Do your machines have AES-NI ?

Edit: I expect they do considering what they are ..

See this:
https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux

rtokarek

I saw and read the https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux document. As I mentioned, encryption doesn't seem to be causing the overhead as setting cipher to "none" doesn't improve speed. Increasing the tun-mtu doesn't have much of an effect.

"openssl speed -evp aes-256-cbc" produces about 8Gb/s (operating on 1024byte blocks). aes-128-cbc tests at 10Gb/s. I'd anticipate that extra packet mangling would drop those substantially in an actual tunnel, but not to the sub-Gb level we're seeing.

TinCanTech

For testing purpose, try using --dev tun, --server not --dev tap, --server-bridge ..

rtokarek

I did try that just in case the bridge or tap interfaces might be causing an issue, but I get the same behavior over a non-bridged tun setup as with a bridged tap. It's totally mysterious.

TinCanTech

Your posts indicate that your network is reasonably complex .. I would be checking all routers / firewalls etc.

Pippin

Setting cipher from AES-256-CBC to AES-128-CBC or even to "none" makes no performance difference so encryption overhead doesn't seem to be an issue

Yeah, that`s right because

AES-256-CBC

is overridden by NCP with

AES-256-GCM

"openssl speed -evp aes-256-cbc"

I found these tests to be meaningless w.r.t. throughput speed.
It should be

"openssl speed -elapsed -evp aes-256-gcm"

by the way.
It only shows what a machine is capable of crypto wise.

Intel Xeon E3-1245 v5 @ 3.50GHz core (1CPU/4 cores per machine; hyper threading off, though no difference with HT on

OpenVPN is single threaded so single core speed and AES-NI support is more important.
NIC drivers can be issue too, or the settings of them, offload checksum, TSO, LRO...etc.

Not an expert but it`s hard to see your setup maybe draw a pic?
You`ve tried different settings and versions, I would start to think hardware I guess....

Saw a report this week on pfSense subreddit w.r.t. AES-GCM:
https://www.reddit.com/r/PFSENSE/comments/5l45jk/openvpn_240_is_now_available_on_pfsense_24/

Javi IIBM

Hello,

I am experimenting a similar situation:

I have a KVM virtual machine with debian Jessie running openvpn 2.3.4-5+deb8u1 (amd64).

if I run it on a 10Gbe capable hardware, all probes (nuttcp) are internally correct with near 10Gb bandwidth, but when I connect with an openvpn client, I have incredibly poor performance.
if I run the same VM on a 1Gb capable hardware (same vmhost OS versión and packages), the performance is as expected.

I still have another installation of an openvpn VM running on a XEN Dom0 but with Wheezy and openvpn 2.2.1-8 (i386) with exactly the same behaviour. When I run it on a 10Gbe hardware I can hardly even do an ssh. Running exactly same VM on a 1Gbe hardware, everything goes ok again.

Tomorrow I'll try to limit my bandwidth with 'tc' at the VM and run it on the 10Gbe hardware to test if I can solve the problem with a soft limit on the bw. Also I will test again both situations with nuttcp and will post the results.

Cheers.

Javi IIBM

Sorry because I am the opposite of a "TCP stack hacker", my english is not the best, and I feel as if I don't understand anything and I am groping in the dark.

Yesterday I described (not very well) my scenario: running a VM with 1Gb capable hw works while running same VM on 10Gb does not.

WS <-100Mb-> VPN <-1Gb-> Control running nuttcp -S : Does work
WS <-100Mb-> VPN <-10Gb-> Control running nuttcp -S : Does NOT work

The problem (when using openvpn on the 10Gbe machine) comes when receiving TCP packets:

WS -> Control (TCP)

$ nuttcp  -Ri50m  control
   58.9375 MB /  10.03 sec =   49.3092 Mbps 85 %TX 6 %RX 0 retrans 14.90 msRTT

WS <- Control (TCP)

$ nuttcp  -Ri50m  -r control
nuttcp-r: v6.1.2: Error: not receiving data from server

WS -> Control (UDP)

$ nuttcp  -Ri50m  -u control
   53.8125 MB /  10.00 sec =   45.1424 Mbps 89 %TX 3 %RX 1489 / 56593 drop/pkt 2.63 %loss

WS <- Control (UDP)

$ nuttcp  -Ri50m  -u -r control
   58.4990 MB /  10.00 sec =   49.0729 Mbps 99 %TX 4 %RX 1119 / 61022 drop/pkt 1.83 %loss

I noticed that when the problem did happen, there where a syslog warning on the VM (not on the host/dom0):

Apr  6 17:46:45 vpn kernel: [  374.148739] ------------[ cut here ]------------
Apr  6 17:46:45 vpn kernel: [  374.149146] WARNING: CPU: 0 PID: 0 at /build/linux-GU1w8g/linux-3.16.39/net/core/dev.c:2247 skb_warn_
bad_offload+0xc6/0xd1()
Apr  6 17:46:45 vpn kernel: [  374.150248] : caps=(0x00000080000048c1, 0x0000000000000000) len=1408 data_len=1294 gso_size=1356 gso_
type=5 ip_summed=1
Apr  6 17:46:45 vpn kernel: [  374.152083] Modules linked in: tun ip6table_filter ip6_tables xt_tcpudp nf_conntrack_ipv4 nf_defrag_i
pv4 ppdev xt_conntrack evdev nf_conntrack serio_raw pcspkr virtio_balloon i2c_piix4 i2c_core pvpanic parport_pc parport iptable_filt
er processor thermal_sys button ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 ata_generic virtio_blk virtio_net ata_piix psmous
e uhci_hcd ehci_hcd libata floppy scsi_mod virtio_pci virtio_ring virtio usbcore usb_common
Apr  6 17:46:45 vpn kernel: [  374.158033] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W     3.16.0-4-amd64 #1 Debian 3.16.39-1
Apr  6 17:46:45 vpn kernel: [  374.158033] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf
 04/01/2014
Apr  6 17:46:45 vpn kernel: [  374.158033]  0000000000000000 ffffffff81514c11 ffff88003fc03c08 0000000000000009
Apr  6 17:46:45 vpn kernel: [  374.158033]  ffffffff81068867 ffff880036db5300 ffff88003fc03c58 0000000000000005
Apr  6 17:46:45 vpn kernel: [  374.158033]  0000000000000001 ffff880036db5300 ffffffff810688cc ffffffff81776438
Apr  6 17:46:45 vpn kernel: [  374.158033] Call Trace:
Apr  6 17:46:45 vpn kernel: [  374.158033]  <IRQ>  [<ffffffff81514c11>] ? dump_stack+0x5d/0x78
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff81068867>] ? warn_slowpath_common+0x77/0x90
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff810688cc>] ? warn_slowpath_fmt+0x4c/0x50
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff8151637c>] ? skb_warn_bad_offload+0xc6/0xd1
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff814266c1>] ? __skb_gso_segment+0x71/0xc0
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff81426a02>] ? dev_hard_start_xmit+0x162/0x610
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff814470f9>] ? sch_direct_xmit+0xc9/0x1a0
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff814270a7>] ? __dev_queue_xmit+0x1f7/0x4d0
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff814608f2>] ? ip_finish_output+0x3e2/0x850
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff81424ef3>] ? __netif_receive_skb_core+0x563/0x770
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff8142517f>] ? netif_receive_skb_internal+0x1f/0x80
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffffa00465aa>] ? virtnet_poll+0x52a/0x880 [virtio_net]
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff81425509>] ? net_rx_action+0x129/0x250
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff8106d911>] ? __do_softirq+0xf1/0x2d0
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff8106dd25>] ? irq_exit+0x95/0xa0
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff8151dbe2>] ? do_IRQ+0x52/0xe0
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff8151ba2d>] ? common_interrupt+0x6d/0x6d
Apr  6 17:46:45 vpn kernel: [  374.158033]  <EOI>  [<ffffffff8101da50>] ? mwait_idle+0xa0/0xa0
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff81052c22>] ? native_safe_halt+0x2/0x10
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff8101da69>] ? default_idle+0x19/0xd0
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff810a9b74>] ? cpu_startup_entry+0x374/0x470
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff81903076>] ? start_kernel+0x497/0x4a2
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff81902a04>] ? set_init_arg+0x4e/0x4e
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff81902120>] ? early_idt_handler_array+0x120/0x120
Apr  6 17:46:45 vpn kernel: [  374.158033]  [<ffffffff8190271f>] ? x86_64_start_kernel+0x14d/0x15c
Apr  6 17:46:45 vpn kernel: [  374.185990] ---[ end trace 79c0bd816eca87bb ]---

The 10Gbe hardware on the host machine is an Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01), and I'm using the ixgbe kernel module.

After "googling", trying disabling "Generic Segmentation Offload" (gso), "TCP Segmentation Offload" (tso), "Scatter Gather" (sg), "UDP Fragmentation Offload", "Generic Receive Offload" (gro) on the bridge, physical interfaces, virtual interfaces.... (this is "groping in the dark", isn't it?) with no success, I have decided, by the moment, to limit the bandwith with 'tc'. Now my openvpn virtual machine can run on my 10Gb ethernet hardware, and no syslog warnings are rised.

On the VM:

tc qdisc add dev eth0 handle 1: root htb default 11
tc class add dev eth0 parent 1: classid 1:1 htb rate 10000Mbps
tc class add dev eth0 parent 1:1 classid 1:11 htb rate 100Mbps
tc qdisc add dev eth0 parent 1:11 handle 10: netem delay 50ms

It is not a solution, I know. Or just a crude one (I don't manage 'tc' neither)

I'll take a look at the different encryption performance to see if there is one better for me.

My current configuration is quite simple:

# cat /etc/openvpn/server-1194.conf | egrep -v '^[#;]' | egrep -v '^$'

port 1194
proto udp
dev tun
ca /etc/openvpn/ssl/cert-2017/ca.crt
cert /etc/openvpn/ssl/cert-2017/server-1194.crt
key /etc/openvpn/ssl/private-2017/server-1194.key
dh ssl/cert-2017/dh2048.pem
server 172.30.234.0 255.255.255.0
ifconfig-pool-persist ipp-1194.txt
push "redirect-gateway"
push "dhcp-option DNS 172.30.1.3"
keepalive 10 120
comp-lzo
persist-key
persist-tun
status openvpn-status-1194.log
verb 3
mute 20
management localhost 7505
multihome Thanks for reading. Any suggestion will be helpful.

Javi IIBM

Sorry. :-)

At the end I did what you said. I read more or less the documentation and tried other type of encryption taking a look before at the throughput "openssl" can perform in my hardware.

The best result was for:

openssl speed -elapsed -evp aes-128-cbc

So I changed my config to
port 1194
proto udp
dev tun
ca /etc/openvpn/ssl/cert-2017/ca.crt
cert /etc/openvpn/ssl/cert-2017/server-1194.crt
key /etc/openvpn/ssl/private-2017/server-1194.key # This file should be kept secret
dh ssl/cert-2017/dh2048.pem
server 172.30.234.0 255.255.255.0
ifconfig-pool-persist ipp-1194.txt
push "redirect-gateway"
push "dhcp-option DNS 172.30.1.3"
keepalive 10 120
cipher AES-128-CBC # AES
persist-key
persist-tun
status openvpn-status-1194.log
verb 3
mute 20
management localhost 7505
multihome And now I works fine and no warning is droped to syslog.

Thanks. I took me a while to understand and to manage with the config of the encryption.