Automatic fail over/load balancing fail in openvpn

[Tek Chand]

Hello Team,
I have created one openvpn server and create client key over this server. The client tunnel up is coming up and working fine. Then i created another server by coping the configuration of existing server over new server and change the virtual address IP pool according to below article:

https://openvpn.net/community-resources ... iguration/

In client.conf/ovpn file i used both server ip one by one and its working fine. Then i used both server ip together to achieve high availability and stop the openvpn service over one server. To test the automatic fail over. But client unable to connect with second server and below are the logs:

```
Sep 19 10:10:08 redismaster ovpn-rana[3361]: Connection reset, restarting [0]
Sep 19 10:10:08 redismaster ovpn-rana[3361]: SIGUSR1[soft,connection-reset] received, process restarting
Sep 19 10:10:08 redismaster ovpn-rana[3361]: Restart pause, 5 second(s)
Sep 19 10:10:13 redismaster ovpn-rana[3361]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sep 19 10:10:13 redismaster ovpn-rana[3361]: Socket Buffers: R=[87380->87380] S=[16384->16384]
Sep 19 10:10:13 redismaster ovpn-rana[3361]: Attempting to establish TCP connection with [AF_INET]first_server_ip:443 [nonblock]
Sep 19 10:10:14 redismaster ovpn-rana[3361]: TCP: connect to [AF_INET]first_server_ip:443 failed, will try again in 5 seconds: Connection refused
Sep 19 10:10:14 redismaster ovpn-rana[3361]: SIGUSR1[soft,init_instance] received, process restarting
Sep 19 10:10:14 redismaster ovpn-rana[3361]: Restart pause, 5 second(s)
Sep 19 10:10:19 redismaster ovpn-rana[3361]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sep 19 10:10:19 redismaster ovpn-rana[3361]: Socket Buffers: R=[87380->87380] S=[16384->16384]
Sep 19 10:10:19 redismaster ovpn-rana[3361]: Attempting to establish TCP connection with [AF_INET]second_server_ip:443 [nonblock]
Sep 19 10:10:20 redismaster ovpn-rana[3361]: TCP connection established with [AF_INET]second_server_ip:443
Sep 19 10:10:20 redismaster ovpn-rana[3361]: TCPv4_CLIENT link local: [undef]
Sep 19 10:10:20 redismaster ovpn-rana[3361]: TCPv4_CLIENT link remote: [AF_INET]second_server_ip:443
Sep 19 10:10:20 redismaster ovpn-rana[3361]: TLS: Initial packet from [AF_INET]second_server_ip, sid=ea7f416f 9a427c23
Sep 19 10:10:20 redismaster ovpn-rana[3361]: VERIFY OK: depth=1, C=IN, ST=MH, L=Pune, O=ABC Technology, OU=Community, CN=ABC Technology CA, name=Community, emailAddress=abc@xyz.com
Sep 19 10:10:20 redismaster ovpn-rana[3361]: Validating certificate key usage
Sep 19 10:10:20 redismaster ovpn-rana[3361]: ++ Certificate has key usage 00a0, expects 00a0
Sep 19 10:10:20 redismaster ovpn-rana[3361]: VERIFY KU OK
Sep 19 10:10:20 redismaster ovpn-rana[3361]: Validating certificate extended key usage
Sep 19 10:10:20 redismaster ovpn-rana[3361]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Sep 19 10:10:20 redismaster ovpn-rana[3361]: VERIFY EKU OK
Sep 19 10:10:20 redismaster ovpn-rana[3361]: VERIFY OK: depth=0, C=IN, ST=MH, L=Pune, O=Promobi Technology, OU=Community, CN=server, name=Community, emailAddress=abc@xyz.com
Sep 19 10:10:20 redismaster ovpn-rana[3361]: Data Channel Encrypt: Cipher 'AES-128-CBC' initialized with 128 bit key
Sep 19 10:10:20 redismaster ovpn-rana[3361]: Channel Encrypt: Using 256 bit message hash 'SHA256' for HMAC authentication
Sep 19 10:10:20 redismaster ovpn-rana[3361]: Data Channel Decrypt: Cipher 'AES-128-CBC' initialized with 128 bit key
Sep 19 10:10:20 redismaster ovpn-rana[3361]: Data Channel Decrypt: Using 256 bit message hash 'SHA256' for HMAC authentication
Sep 19 10:10:20 redismaster ovpn-rana[3361]: Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
Sep 19 10:10:20 redismaster ovpn-rana[3361]: [server] Peer Connection Initiated with [AF_INET]first_server_ip:443
Sep 19 10:10:23 redismaster ovpn-rana[3361]: SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)
Sep 19 10:10:23 redismaster ovpn-rana[3361]: PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1 bypass-dhcp,dhcp-option DNS 208.67.222.222,dhcp-option DNS 208.67.220.220,route 10.8.0.1,topology net30,ping 10,ping-restart 60,ifconfig 10.8.0.10 10.8.0.9'
Sep 19 10:10:23 redismaster ovpn-rana[3361]: OPTIONS IMPORT: timers and/or timeouts modified
Sep 19 10:10:23 redismaster ovpn-rana[3361]: OPTIONS IMPORT: --ifconfig/up options modified
Sep 19 10:10:23 redismaster ovpn-rana[3361]: OPTIONS IMPORT: route options modified
Sep 19 10:10:23 redismaster ovpn-rana[3361]: OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Sep 19 10:10:23 redismaster ovpn-rana[3361]: Preserving previous TUN/TAP instance: tun0
Sep 19 10:10:23 redismaster ovpn-rana[3361]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
Sep 19 10:10:23 redismaster ovpn-rana[3361]: /sbin/ip route del 10.8.1.1/32
Sep 19 10:10:23 redismaster ovpn-rana[3361]: ERROR: Linux route delete command failed: external program exited with error status: 2
Sep 19 10:10:23 redismaster ovpn-rana[3361]: /sbin/ip route del 68.183.181.138/32
Sep 19 10:10:23 redismaster ovpn-rana[3361]: ERROR: Linux route delete command failed: external program exited with error status: 2
Sep 19 10:10:23 redismaster ovpn-rana[3361]: /sbin/ip route del 0.0.0.0/1
Sep 19 10:10:23 redismaster ovpn-rana[3361]: ERROR: Linux route delete command failed: external program exited with error status: 2
Sep 19 10:10:23 redismaster ovpn-rana[3361]: /sbin/ip route del 128.0.0.0/1
Sep 19 10:10:23 redismaster ovpn-rana[3361]: ERROR: Linux route delete command failed: external program exited with error status: 2
Sep 19 10:10:23 redismaster ovpn-rana[3361]: Closing TUN/TAP interface
Sep 19 10:10:23 redismaster ovpn-rana[3361]: /sbin/ip addr del dev tun0 local 10.8.1.6 peer 10.8.1.5
Sep 19 10:10:23 redismaster ovpn-rana[3361]: Linux ip addr del failed: external program exited with error status: 2
Sep 19 10:10:23 redismaster ovpn-rana[3361]: /etc/openvpn/update-resolv-conf tun0 1500 1572 10.8.1.6 10.8.1.5 init
Sep 19 10:10:23 redismaster ovpn-rana[3361]: WARNING: Failed running command (--up/--down): external program exited with error status: 1
Sep 19 10:10:23 redismaster ovpn-rana[3361]: Exiting due to fatal error
Sep 19 10:10:23 redismaster systemd[1]: openvpn@rana.service: Main process exited, code=exited, status=1/FAILURE
Sep 19 10:10:23 redismaster systemd[1]: openvpn@rana.service: Unit entered failed state.
Sep 19 10:10:23 redismaster systemd[1]: openvpn@rana.service: Failed with result 'exit-code'.

```

Can you please help me to fix this issue. What configuration changes are required at client end or server end?

Any help will be appreciated.

[TinCanTech]

What configuration changes are required

What configuration files ?

[Tek Chand]

Hello Team,

Below are our config file for client and server:

For Client:

Code: Select all

client
dev tun
proto tcp
remote 1.2.3.4 443 tcp
remote 5.6.7.8 443 tcp
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
cipher AES-128-CBC
auth SHA256
key-direction 1
script-security 2 
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf
remote-cert-tls server
comp-lzo
verb 5
ca /etc/openvpn/rana/keys/ca.crt
cert /etc/openvpn/rana/keys/rana.crt
key /etc/openvpn/rana/keys/rana.key

For Server:

Code: Select all

port 443
proto tcp
dev tun
ca ca.crt
cert server.crt
key server.key  
dh dh2048.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "redirect-gateway def1 bypass-dhcp"
push "dhcp-option DNS 208.67.222.222"
push "dhcp-option DNS 208.67.220.220"
keepalive 10 60
tls-auth ta.key 0
key-direction 0
cipher AES-128-CBC  
auth SHA256
comp-lzo
user nobody
group nogroup
persist-key
persist-tun
status openvpn-status.log
log-append  /var/log/openvpn.log
verb 5

We are using same configuration over both server except virtual ip address group are different i.e 10.8.0.0 and 10.8.1.0.

When we are stopping the VPN service over one server then its trying to connect another as we can see in logs in first post. But unable to switch.
Please help me.

Thanks.

[TinCanTech]

user nobody
group nogroup

You must remove this from your client config.

[Tek Chand]

user nobody
group nogroup

You must remove this from your client config.

Once i removed above lines from client config i am able to switch one time from first server to second vpn server. But when I start openvpn service back on first server and stopped over second server its unable to switch back to first server. When i stopped openvpn service at second server the tunnel was showing up on client which have virtual IP address of server2, as per my knowledge it should be down if openvpn service is not running over server.

I am getting below error on client when trying to switch back to first server:

Code: Select all

Sep 25 11:02:05 redismaster ovpn-rana[6333]: TCP: connect to [AF_INET]first_server:443 failed, will try again in 5 seconds: Connection refused
Sep 25 11:02:05 redismaster ovpn-rana[6333]: SIGUSR1[soft,init_instance] received, process restarting
Sep 25 11:02:05 redismaster ovpn-rana[6333]: Restart pause, 5 second(s)
Sep 25 11:01:49 redismaster ovpn-rana[6333]: Attempting to establish TCP connection with [AF_INET]second_server:443 [nonblock]
Sep 25 11:01:59 redismaster ovpn-rana[6333]: TCP: connect to [AF_INET]second_server:443 failed, will try again in 5 seconds: Connection timed out
Sep 25 11:01:59 redismaster ovpn-rana[6333]: SIGUSR1[soft,init_instance] received, process restarting
Sep 25 11:01:59 redismaster ovpn-rana[6333]: Restart pause, 5 second(s)
Sep 25 11:02:04 redismaster ovpn-rana[6333]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sep 25 11:02:04 redismaster ovpn-rana[6333]: Re-using SSL/TLS context
Sep 25 11:02:04 redismaster ovpn-rana[6333]: LZO compression initialized
Sep 25 11:02:04 redismaster ovpn-rana[6333]: Control Channel MTU parms [ L:1572 D:1170 EF:80 EB:0 ET:0 EL:3 ]
Sep 25 11:02:04 redismaster ovpn-rana[6333]: Socket Buffers: R=[87380->87380] S=[16384->16384]
Sep 25 11:02:04 redismaster ovpn-rana[6333]: Data Channel MTU parms [ L:1572 D:1450 EF:72 EB:143 ET:0 EL:3 AF:3/1 ]
Sep 25 11:02:04 redismaster ovpn-rana[6333]: Local Options String: 'V4,dev-type tun,link-mtu 1572,tun-mtu 1500,proto TCPv4_CLIENT,comp-lzo,keydir 1,cipher AES-128-CBC,auth SHA256,keysize 128,tls-auth,key-method 2,tls-client'
Sep 25 11:02:04 redismaster ovpn-rana[6333]: Expected Remote Options String: 'V4,dev-type tun,link-mtu 1572,tun-mtu 1500,proto TCPv4_SERVER,comp-lzo,keydir 0,cipher AES-128-CBC,auth SHA256,keysize 128,tls-auth,key-method 2,tls-server'
Sep 25 11:02:04 redismaster ovpn-rana[6333]: Local Options hash (VER=V4): 'e022a93f'
Sep 25 11:02:04 redismaster ovpn-rana[6333]: Expected Remote Options hash (VER=V4): '57076ab0'
Sep 25 11:02:04 redismaster ovpn-rana[6333]: Attempting to establish TCP connection with [AF_INET]first_server:443 [nonblock]

I have below lines in my server config also:

Code: Select all

user nobody
group nogroup

Please help me. What other changes are required?

Thanks.

[Tek Chand]

Hello Team,

user nobody
group nogroup

You must remove this from your client config.

I have removed the above lines from my client configuration. Below are the things which happen and we are expected after this:

1. We have two OPENVPN server to achieve HA i.e server1 and server2
2. Initially client was connected with server1.
3. Then i stooped openvpn service at server1.
4. Client was able to switch server2 successfully.
5. Now i started openvpn service again on server1
6. Stopped openvpn service on server2.
7. Now i am expecting that client will switch back to server1 but its failed.

We are getting below error logs in client logs:

Code: Select all

Sep 25 11:01:44 redismaster ovpn-rana[6333]: TCP: connect to [AF_INET]server_two:443 failed, will try again in 5 seconds: Connection refused
Sep 25 11:01:44 redismaster ovpn-rana[6333]: SIGUSR1[soft,init_instance] received, process restarting
Sep 25 11:01:44 redismaster ovpn-rana[6333]: Restart pause, 5 second(s)
Sep 25 11:01:49 redismaster ovpn-rana[6333]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sep 25 11:01:49 redismaster ovpn-rana[6333]: Re-using SSL/TLS context
Sep 25 11:01:49 redismaster ovpn-rana[6333]: LZO compression initialized
Sep 25 11:01:49 redismaster ovpn-rana[6333]: Control Channel MTU parms [ L:1572 D:1170 EF:80 EB:0 ET:0 EL:3 ]
Sep 25 11:01:49 redismaster ovpn-rana[6333]: Socket Buffers: R=[87380->87380] S=[16384->16384]
Sep 25 11:01:49 redismaster ovpn-rana[6333]: Data Channel MTU parms [ L:1572 D:1450 EF:72 EB:143 ET:0 EL:3 AF:3/1 ]
Sep 25 11:01:49 redismaster ovpn-rana[6333]: Local Options String: 'V4,dev-type tun,link-mtu 1572,tun-mtu 1500,proto TCPv4_CLIENT,comp-lzo,keydir 1,cipher AES-128-CBC,auth SHA256,keysize 128,tls-auth,key-method 2,tls-client'
Sep 25 11:01:49 redismaster ovpn-rana[6333]: Expected Remote Options String: 'V4,dev-type tun,link-mtu 1572,tun-mtu 1500,proto TCPv4_SERVER,comp-lzo,keydir 0,cipher AES-128-CBC,auth SHA256,keysize 128,tls-auth,key-method 2,tls-server'
Sep 25 11:01:49 redismaster ovpn-rana[6333]: Local Options hash (VER=V4): 'e022a93f'
Sep 25 11:01:49 redismaster ovpn-rana[6333]: Expected Remote Options hash (VER=V4): '57076ab0'
Sep 25 11:01:49 redismaster ovpn-rana[6333]: Attempting to establish TCP connection with [AF_INET]server_one:443 [nonblock]
Sep 25 11:01:59 redismaster ovpn-rana[6333]: TCP: connect to [AF_INET]server_one:443 failed, will try again in 5 seconds: Connection timed out
Sep 25 11:01:59 redismaster ovpn-rana[6333]: SIGUSR1[soft,init_instance] received, process restarting
Sep 25 11:01:59 redismaster ovpn-rana[6333]: Restart pause, 5 second(s)

We have below lines in server config also:

Code: Select all

user nobody
group nogroup

Noticed one more thing if we repeat this procedure again initially connected with server1 then try to switch server2 its failed. Its strange. First time worked and second time failed.

What other configuration changes are required? Please help me.

Note: Server2 is configured using server1. Means ca.crt, server.key etc are copied from server1 to server2.

Thanks.

[TinCanTech]

Your log indicates that the server did not respond to your client, was the server actually running ?

What you are describing should work without any further issue.

You can leave the --user/--group lines in the server config, they do not interfere with the client.

[Tek Chand]

Hello Team,

Your log indicates that the server did not respond to your client, was the server actually running ?

Yes, i am sure both server were in running state and tunnel was up on both server when i am testing these things. But i noticed one thing i.e:

When client was connected with server1 and we stopped openvpn service at server1 the tun interface on server goes down but on client its still showing up. But as per my knowledge it should be goes down if openven service is down on server.

This may be cause client unable to switch over server2 and vice versa.

Can you please help me on this behaviour?

[TinCanTech]

The client will wait for a time out before it decides the server is unavailable.

[Tek Chand]

Hello Team,

The client will wait for a time out before it decides the server is unavailable.

is below entry in server configuration is used for time out:

Code: Select all

keepalive 10 60

I think keepalive is server config file directive. Are we need to set ping-restart directive in client config?

One more thing if client will wait for timeout then first time client was connected with server2 after trying one time with server1 (was down) in 5 seconds. That's why i did not think about this.

Thanks.

[Tek Chand]

Hello Team,

I am facing a strange issue. As i have posted that i have 2 openvpn server i.e server1 and server2. We want to achieve HA. I have provided my config files in earlier posts. We have defined both vpn server IP in client.conf to achieve HA. Below are the finding:

1. When we start openvpn at client it connect with server1 which is defined first in client.conf file.
2. Now we stop openvpn service at server1 and expecting client will switch over server2 we have wait till 5 minutes but no success.

Now we are getting the below logs in client log:

Code: Select all

Sep 27 09:52:15 redismaster ovpn-rana[11090]: Attempting to establish TCP connection with [AF_INET][b]server1:443 [/b][nonblock]
Sep 27 09:52:16 redismaster ovpn-rana[11090]: TCP: connect to [AF_INET][b]server1:443 [/b]failed, will try again in 5 seconds: Connection refused
Sep 27 09:52:16 redismaster ovpn-rana[11090]: SIGUSR1[soft,init_instance] received, process restarting
Sep 27 09:52:16 redismaster ovpn-rana[11090]: Restart pause, 5 second(s)
Sep 27 09:52:21 redismaster ovpn-rana[11090]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sep 27 09:52:21 redismaster ovpn-rana[11090]: Re-using SSL/TLS context
Sep 27 09:52:21 redismaster ovpn-rana[11090]: LZO compression initialized
Sep 27 09:52:21 redismaster ovpn-rana[11090]: Control Channel MTU parms [ L:1572 D:1170 EF:80 EB:0 ET:0 EL:3 ]
Sep 27 09:52:21 redismaster ovpn-rana[11090]: Socket Buffers: R=[87380->87380] S=[16384->16384]
Sep 27 09:52:21 redismaster ovpn-rana[11090]: Data Channel MTU parms [ L:1572 D:1450 EF:72 EB:143 ET:0 EL:3 AF:3/1 ]
Sep 27 09:52:21 redismaster ovpn-rana[11090]: Local Options String: 'V4,dev-type tun,link-mtu 1572,tun-mtu 1500,proto TCPv4_CLIENT,comp-lzo,keydir 1,cipher AES-128-CBC,auth SHA256,keysize 128,tls-auth,key-method 2,tls-client'
Sep 27 09:52:21 redismaster ovpn-rana[11090]: Expected Remote Options String: 'V4,dev-type tun,link-mtu 1572,tun-mtu 1500,proto TCPv4_SERVER,comp-lzo,keydir 0,cipher AES-128-CBC,auth SHA256,keysize 128,tls-auth,key-method 2,tls-server'
Sep 27 09:52:21 redismaster ovpn-rana[11090]: Local Options hash (VER=V4): 'e022a93f'
Sep 27 09:52:21 redismaster ovpn-rana[11090]: Expected Remote Options hash (VER=V4): '57076ab0'
Sep 27 09:52:21 redismaster ovpn-rana[11090]: Attempting to establish TCP connection with [AF_INET[b]]server2[/b]:443 [nonblock]
Sep 27 09:52:31 redismaster ovpn-rana[11090]: TCP: connect to [AF_INET][b]server2:443[/b] failed, will try again in 5 seconds: Connection timed out
Sep 27 09:52:31 redismaster ovpn-rana[11090]: SIGUSR1[soft,init_instance] received, process restarting
Sep 27 09:52:31 redismaster ovpn-rana[11090]: Restart pause, 5 second(s)

According to log we tried to troubleshoot the issue:

1. Initially when client was connected with server1 and srever2 was also up and running openvpn we were able to telnet both server on port 443 from client as well as from another base machine. Our openvpn is running over port 443.
2. Then we stop openvpn service at server1 to test HA.
3. Now we again try to telnet the openvpn server2 where openvpn service is running and client should be switch over it but unable to telnet on port 443 from client machine but we were able to telnet server2 from another machine which means openvpn service was running on server2.
4. Logs shows that client try to connect server2.
5 If i start openvpn service on server1 then client connected back to server1.
6. If we restart the openvpn service at client when server1 is down then it connect to server2.

Its really strange and unable to understand this behaviour. I have added the below ufw rule on both server to masquerade client connection:

Code: Select all

-A POSTROUTING -s 10.8.0.0/8 -o eth0 -j MASQUERADE
COMMIT

But i have define virtual ip address range in server2 is 10.8.1.0/24 but its should not make any impact because 10.8.1.0/24 is also covered under 10.8.0.0/24.

Can you please help me whats wrong with my configuration. This issue is become road block in my project.

Thanks.

[TinCanTech]

See your server logs for the information you need ...