[SOLVED] Can not connect to services behind VPN and routes are not pushed

[holo]

Hello all

I configured OpenVPN on top of Kubernetes cluster. Im connecting to it and i can ping OpenVPN IP but i can not connect to services behind VPN. What am i doing wrong? I think it is probably connected with my iptables rules:

Code: Select all

iptables -A FORWARD -i tun0 -o eth0 -s 172.16.0.0/24 -d 10.0.0.0/8 -m conntrack --ctstate NEW -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -s 172.16.0.0/24 -j MASQUERADE

Here is my VPN server config:

View OriginalServerConfig

##protocol port

port 1194

proto tcp

dev tun



##ip server client

server 172.16.0.0 255.255.0.0



##key

ca /etc/openvpn/ca.crt

cert /etc/openvpn/server.crt

key /etc/openvpn/server.key

dh /etc/openvpn/dh2048.pem



##option

persist-key

persist-tun

keepalive 5 60

reneg-sec 432000



##option authen.

comp-lzo

user nobody

#group nogroup

client-to-client

username-as-common-name

client-cert-not-required

auth-user-pass-verify /etc/openvpn/scripts/login.sh via-env



##push to client

max-clients 50

push "persist-key"

push "persist-tun"

#push "redirect-gateway local def1"

push "explicit-exit-notify 1"

push "route add 10.0.0.0 255.0.0.0"



##DNS-Server

push "dhcp-option DNS 10.96.0.10"



##script connect-disconnect

script-security 2

##client-connect /etc/openvpn/scripts/connect.sh

client-disconnect /etc/openvpn/scripts/disconnect.sh



##log-status

##status /etc/openvpn/log/tcp_443.log

##log-append /etc/openvpn/log/openvpn.log

verb 3

Interfaces in the container and routing:

Code: Select all

root@openvpn-8cf977bdb-xgpdr:/# ip r
default via 169.254.1.1 dev eth0 
169.254.1.1 dev eth0 scope link 
172.16.0.0/16 via 172.16.0.2 dev tun0 
172.16.0.2 dev tun0 proto kernel scope link src 172.16.0.1 
root@openvpn-8cf977bdb-xgpdr:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if44: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 8a:40:1c:e2:e7:86 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.1.55/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::8840:1cff:fee2:e786/64 scope link 
       valid_lft forever preferred_lft forever
5: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none 
    inet 172.16.0.1 peer 172.16.0.2/32 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::ed4a:c206:3b94:fb5d/64 scope link flags 800 
       valid_lft forever preferred_lft forever
root@openvpn-8cf977bdb-xgpdr:/# 

Routes on my local machine:

Code: Select all

[holo@holo-pc ~]$ ip r
default via 192.168.10.1 dev enp5s0 proto dhcp metric 100 
10.0.0.0/8 via 172.16.0.5 dev tun0 
172.16.0.0/16 via 172.16.0.5 dev tun0 
172.16.0.5 dev tun0 proto kernel scope link src 172.16.0.6 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.10.0/24 dev enp5s0 proto kernel scope link src 192.168.10.181 metric 100 
192.168.10.1 dev enp5s0 proto static scope link metric 100 
194.177.28.83 via 192.168.10.1 dev enp5s0 proto static metric 100 
[holo@holo-pc ~]$ 

Route to 10.0.0.0/8 i needed to add manually because:

push "route add 10.0.0.0 255.0.0.0"

is not working, why?

Client log:

Code: Select all

...
Fri Aug  3 13:00:41 2018 us=626403 [server] Peer Connection Initiated with [AF_INET]194.177.28.83:1194
Fri Aug  3 13:00:42 2018 us=818013 SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)
Fri Aug  3 13:00:42 2018 us=863985 PUSH: Received control message: 'PUSH_REPLY,persist-key,persist-tun,explicit-exit-notify 1,route add 10.0.0.0 255.0.0.0,dhcp-option DNS 10.96.0.10,route 172.16.0.0 255.255.0.0,topology net30,ping 5,ping-restart 60,ifconfig 172.16.0.6 172.16.0.5,peer-id 0,cipher AES-256-GCM'
Fri Aug  3 13:00:42 2018 us=864063 Options error: route parameter network/IP 'add' must be a valid address
Fri Aug  3 13:00:42 2018 us=864144 OPTIONS IMPORT: timers and/or timeouts modified
Fri Aug  3 13:00:42 2018 us=864158 OPTIONS IMPORT: --explicit-exit-notify can only be used with --proto udp
Fri Aug  3 13:00:42 2018 us=864164 OPTIONS IMPORT: --persist options modified
Fri Aug  3 13:00:42 2018 us=864171 OPTIONS IMPORT: --ifconfig/up options modified
Fri Aug  3 13:00:42 2018 us=864178 OPTIONS IMPORT: route options modified
Fri Aug  3 13:00:42 2018 us=864186 OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Fri Aug  3 13:00:42 2018 us=864193 OPTIONS IMPORT: peer-id set
Fri Aug  3 13:00:42 2018 us=864200 OPTIONS IMPORT: adjusting link_mtu to 1627
Fri Aug  3 13:00:42 2018 us=864206 OPTIONS IMPORT: data channel crypto options modified
Fri Aug  3 13:00:42 2018 us=864215 Data Channel: using negotiated cipher 'AES-256-GCM'
Fri Aug  3 13:00:42 2018 us=864232 Data Channel MTU parms [ L:1555 D:1450 EF:55 EB:406 ET:0 EL:3 ]
Fri Aug  3 13:00:42 2018 us=864311 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Fri Aug  3 13:00:42 2018 us=864323 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Fri Aug  3 13:00:42 2018 us=864454 ROUTE_GATEWAY 192.168.10.1/255.255.255.0 IFACE=enp5s0 HWADDR=e0:d5:5e:29:8e:36
Fri Aug  3 13:00:42 2018 us=864730 TUN/TAP device tun0 opened
Fri Aug  3 13:00:42 2018 us=864769 TUN/TAP TX queue length set to 100
Fri Aug  3 13:00:42 2018 us=864788 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Fri Aug  3 13:00:42 2018 us=864803 /usr/bin/ip link set dev tun0 up mtu 1500
Fri Aug  3 13:00:42 2018 us=866579 /usr/bin/ip addr add dev tun0 local 172.16.0.6 peer 172.16.0.5
Fri Aug  3 13:00:42 2018 us=868866 /usr/bin/ip route add 172.16.0.0/16 via 172.16.0.5
Fri Aug  3 13:00:42 2018 us=871198 Initialization Sequence Completed

[TinCanTech]

push "route add 10.0.0.0 255.0.0.0"

is not working, why?

Options error: route parameter network/IP 'add' must be a valid address

See --route in The Manual v24x

[holo]

I changed it to:

push "route add 10.0.0.0 255.0.0.0 172.16.0.5 100"

and still that same (i do not think it was syntax problem as it was taken from examples and was looking like before). According to information it is not valid address but why?

And what with the main problem - packets are not forwarded?

EDIT:

I fixed it syntax however syntax was wrong correct one is:

Code: Select all

push "route 10.0.0.0 255.0.0.0"

But still i can not connect to my kubernetes services which are in 10.0.0.0/8 network

[holo]

I find out solution. I thought:

Code: Select all

sysctl -w net.ipv4.ip_forward=1

is taken from node settings, but i find out pod have its own namespace so this setting is needed. But..

there was problem with setup it as they are read only in containers. So i try to set it up with https://kubernetes.io/docs/tasks/admini ... nnotations feature from kubernetes but it did not work for me so i used https://kubernetes.io/docs/concepts/wor ... containers container setting and set up sysctls from start script.

Thanks to it i can connect to my k8s services IPs directly from clients