ICMP Redirect using Tun0

georgebarker

Need some help here. Would be appreciated. I have openvpn serverA(10.0.0.0) and ServerB (10.10.0.0) ServerA is in on the west coast, serverB is in on the east coast. They can communicate to each other, and can ping servers behind them, but not healthily. When pinging from ServerA to a server within serverB's subnet I get the following:
# ping 10.0.1.87
PING 10.0.1.87 (10.0.1.87) 56(84) bytes of data.
From 169.254.255.1: icmp_seq=1 Redirect Host(New nexthop: 10.0.1.87)
From 169.254.255.1: icmp_seq=2 Redirect Host(New nexthop: 10.0.1.87)
From 169.254.255.1: icmp_seq=3 Redirect Host(New nexthop: 10.0.1.87)
From 169.254.255.1: icmp_seq=4 Redirect Host(New nexthop: 10.0.1.87)
--- 10.0.1.87 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3314ms

Traceroute Fails:
traceroute to 10.0.1.87 (10.0.1.87), 30 hops max, 60 byte packets
1 169.254.255.1 (169.254.255.1) 86.053 ms 85.951 ms 86.136 ms

Tcpdump only confirms that there is an ICMP redirect:
07:23:20.975509 IP 169.254.255.1.36963 > 10.0.0.2.domain: 50093+ PTR? 87.1.0.10. in-addr.arpa. (40)
07:23:21.061567 IP 169.254.255.2.36963 > 10.0.0.2.domain: 50093+ PTR? 87.1.0.10. in-addr.arpa. (40)
07:23:21.061586 IP 169.254.255.1 > 169.254.255.2: ICMP redirect 10.0.0.2 to host 10.0.0.2, length 76
07:23:21.061589 IP 169.254.255.2.36963 > 10.0.0.2.domain: 50093+ PTR? 87.1.0.10. in-addr.arpa. (40)
07:23:21.971195 IP 169.254.255.2 > 10.0.1.87: ICMP echo request, id 20826, seq 7 , length 64
07:23:21.971211 IP 169.254.255.2 > 10.0.1.87: ICMP echo request, id 20826, seq 7 , length 64

Here is my config:
port 1194
dev tun
remote X.X.X.X
proto udp
route 10.10.0.0 255.255.0.0 169.254.255.2
route 10.0.0.0 255.255.0.0 169.254.255.1
push "redirect-gateway def1"
persist-key
persist-tun
comp-lzo
verb 3
ifconfig 169.254.255.1 169.254.255.2
push "route 10.0.0.0 255.255.0.0"
push "route 10.10.0.0 255.255.0.0"

mwandelaar

Some things come in mind when viewing this setup:
According to IANA,the 169.254.0.0/16 network is designated to link-local traffic. This network, with a random host-ip-part, is associated to your local lan.
So, even without any VPN-service open, there's a fair change you already have a "route" to this network.
To prevent looping this routes, i'll advise you to use an RFC1918 private range.

Second, the config shown, looks a client-config. It's localy adding routes (see above) and pushing the routes to the other-end (is it a server-config?). It even pushes a redirect-gateway. Which actually allready includes the 169.254.0.0/16 network. So it has some double routes there. This can cause network-issues too.

Can you:

Show the config on the other end

Show the routing-tables

Maybe before this, allready change the ip-ranges

georgebarker

Thank you for your reply. I have changed the IP of the VPN Server, and if I'm using the wrong config for site-to-site VPN across VPC (aws) please let me know.

Here are my routes. What has me concerned are the double routes, but I'm not sure what they should be.

[serverw01 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.10.1.1 0.0.0.0 UG 0 0 0 eth0
10.0.0.0 172.16.1.1 255.255.0.0 UG 0 0 0 tun0
10.10.0.0 172.16.2.1 255.255.0.0 UG 0 0 0 tun0
10.10.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.169.254 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
172.16.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 tun0

[servere01]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.2.1 0.0.0.0 UG 0 0 0 eth0
10.0.0.0 172.16.1.1 255.255.0.0 UG 0 0 0 tun0
10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
10.10.0.0 172.16.2.1 255.255.0.0 UG 0 0 0 tun0
169.254.169.254 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
172.16.2.1 0.0.0.0 255.255.255.255 UH 0 0 0 tun0

Here are my configs... I've yet to set them up for SSL:
=====SERVER E======
port 1194
dev tun
remote X.X.X.X
proto udp
route 10.10.0.0 255.255.0.0 172.16.2.1
route 10.0.0.0 255.255.0.0 172.16.1.1

persist-key
persist-tun
comp-lzo
verb 3

ifconfig 172.16.1.1 172.16.2.1
push "route 10.10.0.0 255.255.0.0"
secret ovpn.key

======SERVER W======
port 1194
dev tun
remote X.X.X.X
proto udp
route 10.0.0.0 255.255.0.0 172.16.1.1
route 10.10.0.0 255.255.0.0 172.16.2.1
persist-key
persist-tun
comp-lzo
verb 3
ifconfig 172.16.2.1 172.16.1.1
push "route 10.10.0.0 255.255.0.0"
secret ovpn.key

Furthermore, I can ping successfully from E to W, so it appears its establishing a connection, and I can ping devices on the subnet of W, but since I changed the IP I get:
PING 10.10.2.67 (10.10.2.67) 56(84) bytes of data.
From 172.16.2.1 icmp_seq=1 Destination Port Unreachable

Now even on the from the West Server, I cannot reach a server on the same local subnet
PING 10.10.2.67 (10.10.2.67) 56(84) bytes of data.
From 172.16.1.1: icmp_seq=1 Redirect Host(New nexthop: 172.16.2.1)

My goal is to create a VPN tunnel to make all devices accessible on either coast(site to site). I've tried all tutorials I could find, with no success. Any tips?

Look forward to your reply.

mwandelaar

I'm partially rebuilding your setup so it takes a while before a have a 100% clear answer.

I could reproduce the ICMP-redirect-issue on a -not interconnected- vpn-server and the cause of it was not allowing client-to-client traffic.
Therefore packets between 2 clients needs to be routed via the host and the host sends it back into the tun-interface. This causes the ICMP-redirect.
I noticed in your configs you are not allowing client-to-client traffic. Which might be a good choise if you decided not to allow it.

This requires packetforwarding on the server to be enabled and it seems to me (although i cannot be 100% sure) on your server this is not the case. Can you check:

sysctl -a | grep forward

for enabled forwarding?

I'll get back on the other issues.

georgebarker

Thank you for your reply. Here is the information you requested:

# sysctl -a | grep forward
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.eth0.forwarding = 1
net.ipv4.conf.eth0.mc_forwarding = 0
net.ipv4.conf.tun0.forwarding = 1
net.ipv4.conf.tun0.mc_forwarding = 0
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.lo.forwarding = 0
net.ipv6.conf.lo.mc_forwarding = 0
net.ipv6.conf.eth0.forwarding = 0
net.ipv6.conf.eth0.mc_forwarding = 0
net.ipv6.conf.tun0.forwarding = 0
net.ipv6.conf.tun0.mc_forwarding = 0

maikcat

hi there,

ifconfig 172.16.1.1 172.16.2.1

this should be

ifconfig 172.16.1.1 172.16.1.2

and vice versa on other node f.e

ifconfig 172.16.1.2 172.16.1.1

Michael.

georgebarker

I appreciate the attempt at getting this resolved with me. Though, swapping them out will make no difference as those directives are set initially in the configuration anyhow. I did a packet trace, and am making progress in the sense that now that I setup a nat server, I get that port 22 is closed, though its not. I can verify this by sshing from East to to server in the east.... but when I try from West to east my route gets blocked with a packet redirect:

# nmap --packet-trace 10.0.1.9
Starting Nmap 5.51 ( http://nmap.org ) at 2013-03-23 04:11 UTC
SENT (0.0880s) ICMP 172.16.1.1 > 10.0.1.9 Echo request (type=8/code=0) ttl=39 id=47844 iplen=28
SENT (0.0880s) TCP 172.16.1.1:49586 > 10.0.1.9:443 S ttl=50 id=37434 iplen=44 seq=2702887222 win=3072 <mss 1460>
SENT (0.0890s) TCP 172.16.1.1:49586 > 10.0.1.9:80 A ttl=42 id=27327 iplen=40 seq=0 win=3072
SENT (0.0890s) ICMP 172.16.1.1 > 10.0.1.9 Timestamp request (type=13/code=0) ttl=44 id=65326 iplen=40
RCVD (0.1730s) ICMP 172.16.2.1 > 172.16.1.1 Host redirect (type=5/code=1) ttl=64 id=48340 iplen=56
SENT (2.0900s) ICMP 172.16.1.1 > 10.0.1.9 Timestamp request (type=13/code=0) ttl=37 id=50568 iplen=40
SENT (2.0910s) TCP 172.16.1.1:49587 > 10.0.1.9:80 A ttl=46 id=16997 iplen=40 seq=0 win=3072
SENT (2.0910s) TCP 172.16.1.1:49587 > 10.0.1.9:443 S ttl=52 id=15288 iplen=44 seq=2702952759 win=1024 <mss 1460>
SENT (2.0910s) ICMP 172.16.1.1 > 10.0.1.9 Echo request (type=8/code=0) ttl=40 id=9818 iplen=28
RCVD (2.1750s) ICMP 172.16.2.1 > 172.16.1.1 Host redirect (type=5/code=1) ttl=64 id=48341 iplen=68
Note: Host seems down. If it is really up, but blocking our ping probes, try -Pn
Nmap done: 1 IP address (0 hosts up) scanned in 3.09 seconds

mwandelaar

RCVD (0.1730s) ICMP 172.16.2.1 > 172.16.1.1 Host redirect (type=5/code=1) ttl=64 id=48340 iplen=56

and

RCVD (2.1750s) ICMP 172.16.2.1 > 172.16.1.1 Host redirect (type=5/code=1) ttl=64 id=48341 iplen=68

Makes me think there's still a subnet issue in your setup because the the 172.16.2.1 network isn't defined in the setup.
One other thing which cause you problems is a double routing-entry in your setup. Both server W and server E have in there setup the 10.10.1 network and the 10.10.0 network:

[serverw01 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.10.0.0 172.16.2.1 255.255.0.0 UG 0 0 0 tun0
10.10.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0

Both networks are the same! 10.10.0.0 255.255.0.0 INCLUDES 10.10.1.0 netmask 255.255.255.0
This means there are 2 routes on this server (serverw01) for the same network.
Depending on your setup maybe you can change either your local subnet (10.10.1.0 on eth0) or shrink the network on the VPN, i.e. use 10.10.0.0 255.255.255.0 on the VPN. This way you prevent the routing issue of packets being sent to the wrong interface.

Another issue which -maybe- cause you a routing-issue is the fact you are adding AND pushing the routes from both sides, which is not necessary. In this setup you better add them localy using "route" in the config .

Please let us know if this helps.

georgebarker

Now I get as far as this:
When I try to ssh from W to a server on the E sub net, I get the following:
[root@openvpnw01 ~]# ssh -i /root/test.pem ubuntu@10.0.2.163
ssh: connect to host 10.0.2.163 port 22: Connection timed out

and on the server 10.0.2.163, here are the contents of a TCP dump confirming arrival of packet:

01:59:24.413406 ARP, Request who-has 10.0.2.163 tell 10.0.2.1, length 28
01:59:24.413429 ARP, Reply 10.0.2.163 is-at 13:1d: f2:83:79:63 (oui Unknown), length 28
01:59:24.413688 IP 10.0.2.163.47130 > 10.0.0.2.domain: 189+ PTR? 1.2.0.10.in-addr.arpa. (39)
01:59:24.414113 IP 10.0.0.2.domain > 10.0.2.163.47130: 189 NXDomain* 0/1/0 (89)
01:59:43.624117 IP 172.16.2.1.46906 > 10.0.2.163.ssh: Flags [S], seq 2508956628, win 14600, options [mss 1366,sackOK,TS val 29906055 ecr 0,nop,wscale 5], length 0
01:59:43.624177 IP 10.0.2.163.ssh > 172.16.2.1.46906: Flags [S.], seq 1340493015, ack 2508956629, win 14480, options [mss 1460,sackOK,TS val 84091032 ecr 29906055,nop,wscale 5], length 0
01:59:43.624437 IP 10.0.2.163.46861 > 10.0.0.2.domain: 58437+ PTR? 1.2.16.172.in-addr.arpa. (41)
01:59:43.624846 IP 10.0.0.2.domain > 10.0.2.163.46861: 58437 NXDomain 0/1/0 (91)
01:59:44.623873 IP 172.16.2.1.46906 > 10.0.2.163.ssh: Flags [S], seq 2508956628, win 14600, options [mss 1366,sackOK,TS val 29906305 ecr 0,nop,wscale 5], length 0
01:59:44.623913 IP 10.0.2.163.ssh > 172.16.2.1.46906: Flags [S.], seq 1340493015, ack 2508956629, win 14480, options [mss 1460,sackOK,TS val 84091282 ecr 29906055,nop,wscale 5], length 0
01:59:44.821057 IP 10.0.2.163.ssh > 172.16.2.1.46906: Flags [S.], seq 1340493015, ack 2508956629, win 14480, options [mss 1460,sackOK,TS val 84091332 ecr 29906055,nop,wscale 5], length 0
01:59:46.627739 IP 172.16.2.1.46906 > 10.0.2.163.ssh: Flags [S], seq 2508956628, win 14600, options [mss 1366,sackOK,TS val 29906806 ecr 0,nop,wscale 5], length 0
01:59:46.627779 IP 10.0.2.163.ssh > 172.16.2.1.46906: Flags [S.], seq 1340493015, ack 2508956629, win 14480, options [mss 1460,sackOK,TS val 84091783 ecr 29906055,nop,wscale 5], length 0
01:59:46.820973 IP 10.0.2.163.ssh > 172.16.2.1.46906: Flags [S.], seq 1340493015, ack 2508956629, win 14480, options [mss 1460,sackOK,TS val 84091832 ecr 29906055,nop,wscale 5], length 0
01:59:48.628992 ARP, Request who-has 10.0.2.1 tell 10.0.2.163, length 28
01:59:48.629157 ARP, Reply 10.0.2.1 is-at 16:1a:e7:40:00:02 (oui Unknown), length 28
01:59:50.635861 IP 172.16.2.1.46906 > 10.0.2.163.ssh: Flags [S], seq 2508956628, win 14600, options [mss 1366,sackOK,TS val 29907808 ecr 0,nop,wscale 5], length 0
01:59:50.635897 IP 10.0.2.163.ssh > 172.16.2.1.46906: Flags [S.], seq 1340493015, ack 2508956629, win 14480, options [mss 1460,sackOK,TS val 84092785 ecr 29906055,nop,wscale 5], length 0
01:59:50.820986 IP 10.0.2.163.ssh > 172.16.2.1.46906: Flags [S.], seq 1340493015, ack 2508956629, win 14480, options [mss 1460,sackOK,TS val 84092832 ecr 29906055,nop,wscale 5], length 0
01:59:58.652466 IP 172.16.2.1.46906 > 10.0.2.163.ssh: Flags [S], seq 2508956628, win 14600, options [mss 1366,sackOK,TS val 29909812 ecr 0,nop,wscale 5], length 0
01:59:58.652492 IP 10.0.2.163.ssh > 172.16.2.1.46906: Flags [S.], seq 1340493015, ack 2508956629, win 14480, options [mss 1460,sackOK,TS val 84094789 ecr 29906055,nop,wscale 5], length 0
01:59:58.824986 IP 10.0.2.163.ssh > 172.16.2.1.46906: Flags [S.], seq 1340493015, ack 2508956629, win 14480, options [mss 1460,sackOK,TS val 84094833 ecr 29906055,nop,wscale 5], length 0
02:00:08.682358 ARP, Request who-has 10.0.2.163 tell 10.0.2.1, length 28
02:00:08.682384 ARP, Reply 10.0.2.163 is-at 16:1a:e7:75:96:74 (oui Unknown), length 28
02:00:14.700165 IP 172.16.2.1.46906 > 10.0.2.163.ssh: Flags [S], seq 2508956628, win 14600, options [mss 1366,sackOK,TS val 29913824 ecr 0,nop,wscale 5], length 0
02:00:14.700206 IP 10.0.2.163.ssh > 172.16.2.1.46906: Flags [S.], seq 1340493015, ack 2508956629, win 14480, options [mss 1460,sackOK,TS val 84098801 ecr 29906055,nop,wscale 5], length 0
02:00:14.824995 IP 10.0.2.163.ssh > 172.16.2.1.46906: Flags [S.], seq 1340493015, ack 2508956629, win 14480, options [mss 1460,sackOK,TS val 84098833 ecr 29906055,nop,wscale 5], length 0
02:00:19.701006 ARP, Request who-has 10.0.2.1 tell 10.0.2.163, length 28
02:00:19.701247 ARP, Reply 10.0.2.1 is-at 16:1a:e7:40:00:02 (oui Unknown), length 28

I think it may be stemming from the nat box, or the the vpn box. Any Ideas?

mwandelaar

First off all, excuse me for putting quite a lot of questions in my reply.

These 2 packets:

01:59:43.624117 IP 172.16.2.1.46906 > 10.0.2.163.ssh: Flags [S], seq 2508956628, win 14600, options [mss 1366,sackOK,TS val 29906055 ecr 0,nop,wscale 5], length 0
01:59:43.624177 IP 10.0.2.163.ssh > 172.16.2.1.46906: Flags [S.], seq 1340493015, ack 2508956629, win 14480, options [mss 1460,sackOK,TS val 84091032 ecr 29906055,nop,wscale 5], length 0

tell me the syn-packets arrives at the system on the east-side and that system is sending an answer. But it appears it never returns to the vpn-server "West".

Could it be that the router, which acts as a default-gateway for the systems on the east-side, doesn't know the VPN-network is behind the VPN-server "East"?
This looks like the SYN-ACK packet is sent the wrong way out.

On the network on East side, do you have 2 gateways out of the network, one being the normal "default gateway" and a separate VPN-server (actually VPN-router) for the interconnection between East and West?

Can you check where this packet is sent to? And i think you have to figure-out on layer 2 meaning Ethernet-level.

You are talking about NAT here, but on the system 10.0.2.163 the packets arrive with a source-address of server West, so i cannot see a NAT being in place here. Is there somewhere NAT involved here or are you referring to the normal internet-gateway as the NAT-box?