Hello,
This is my first posting here, gratitude for all the good posts.
I have a peer-to-peer setup which is functioning great.
I use the peer addresses as via addresses for routing to cross-network IPs.
For various reasons relative to correctly handling hairpin NAT, I have SNAT'd a packet into the tunnel so that it arrives at its
destination with a source address matching the IP address of another interface on the same source machine.
The peer receives the packet correctly, and responds into the tunnel, visible by tcpdump on the peer.
The tcpdump on the source side has yet to ever show a return packet.
This is a packet whose DST= matches the IP on another interface on the destination machine,
that is on the destination machine for the return packet, which is the original source machine.
192.168.56.157 -> 96.64.59.106:64008 original packet from internal source arriving at machine with vpn tunnel
nat prerouting dnat(96.64.159.106:64008 to 192.168.56.248:8024)
nat postrouting snat(192.168.56.157->96.64.159.105)
routes into tunnel due to packet mark and special routing table
96.64.159.105 -> 192.168.56.248 at peer arrives like this, forwards to internal destination
92.168.56.248 -> 96.64.159.105 response from internal arrives at peer
routes into tunnel due to packet mark and special routing table
tcpdump shows the reply packet entering the tunnel, heading back through to tunnel to the original source side
and where o where can I find evidence of this reply packet in the tcpdump on that original source side?
Here is the iptables TRACE for the source packet coming arriving from its internal source and being forwarded into the tunnel.
I also ran a full wildcard tcpdump on the receiving side and never see the packet.
This is a packet from 192.168.56.248 to 96.64.159.105, through a tunnel to a target where 96.64.159.105 is an IP on another interface.
Can anyone offer any insight as to whether this is the expected behavior of openvpn in this case?
Can anyone suggest how I can see my packet? I do have more trace output, tried to be minimal for this first posting.
I am willing to adjust a local copy of the C code, though I would prefer a config-based solution.
service openvpn@vpn-vs7s2vc7s1 start
iptables -t raw -A OUTPUT -d 10.250.60.130 -j TRACE
iptables -t mangle -A OUTPUT -p udp --dport 1194 -j RETURN
iptables -t mangle -A OUTPUT -d 10.250.60.130 -j MARK --set-mark 1
ip rule add priority 1 fwmark 1 lookup 1
ip route add table 1 default via 10.249.57.58 dev tunvs7s2vc7s1
tcpdump -i tunvs7s2vc7s1 # watch server side of tunnel
ping -I 10.249.58.57 10.249.57.58 # successful, shows in both tcpdump displays and both iptables traces
ping 10.250.60.130 # shows only in server tcpdump and iptables
This packet has src=10.250.60.121, dst=10.250.60.130, and that dst is an IP on a non-tunnel interface on the destination machine.
I am thinking openvpn is by default reluctant to source into a box a packet that would appear to come from one of its own other interfaces.
My question is how can I recover this packet on the peer?
Gratitude.
service openvpn@vpn-vs7s2vc7s1 start
iptables -t raw -A OUTPUT -d 10.250.60.130 -j TRACE
iptables -t mangle -A OUTPUT -p udp --dport 1194 -j RETURN
iptables -t mangle -A OUTPUT -d 10.250.60.130 -j MARK --set-mark 1
ip rule add priority 1 fwmark 1 lookup 1
ip route add table 1 default via 10.249.57.58 dev tunvs7s2vc7s1
tcpdump -i tunvs7s2vc7s1 # watch server side of tunnel
The routing rules are set such that non-port-1194 traffic between the two external endpoints is re-routed through the tunnel.
The reason is related to how I am handling hairpin nat within a larger connected network, which I could discuss if thought helpful.
I have isolated the case here down to two IP addresses in a reproducable case just to analyze the openvpn interaction.
Since the packet enters the tunnel, it must have had a route out of the server to the client.
I would expect to see the packet exit the tunnel on the client side, and for the packet to then enter netfilter on the client, like for other packets.
For example, if I have a machine with eth0=1.1.1.1/24 and eth1=2.2.2.2/24, and I send in a packet from another host 1.1.1.2/24 via 1.1.1.1,
and that packet has src=3.3.3.3/dst=2.2.2.2, then the packet still enters netfilter, the trace shows, and with proper rules the packet can be routed
and enters the INPUT or FORWARD chains.
I would expect the same behavior from a tunnel interface as for eth0 in the example.
In my experiment, I see no packet exiting the tunnel on the client side and no packet in the client iptables -t raw -j TRACE.
I run tcpdump on both sides and only see the packet on the source side, so it seems openvpn is not passing this packet.
*** Am I correct in thinking the client and server tcpdump streams on a p2p should be identical regardless of any routing rules on either side?
For reference, the routing rules are the default at-boot rules as augmented by starting openvpn and running the ip rule and ip route commands.
Here is the final setup for the experiment:
root@ahau1:/etc/openvpn# ip ru
0: from all lookup local
1: from all fwmark 0x1 lookup 1
32766: from all lookup main
32767: from all lookup default
root@ahau1:/etc/openvpn# ip r
default via 10.250.60.254 dev eth0 onlink
10.249.57.58 dev tunvs7s2vc7s1 proto kernel scope link src 10.249.58.57
10.250.60.0/24 dev eth0 proto kernel scope link src 10.250.60.121
root@ahau1:/etc/openvpn# ip r s t 1
default via 10.249.57.58 dev tunvs7s2vc7s1
root@ahau1:/etc/openvpn# iptables -t mangle -nvL OUTPUT
Chain OUTPUT (policy ACCEPT 7670 packets, 829K bytes)
pkts bytes target prot opt in out source destination
6412 680K RETURN udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:1194
75 6300 MARK all -- * * 0.0.0.0/0 10.250.60.130 MARK set 0x1
root@idspgw7-a:/etc/openvpn# ip ru
0: from all lookup local
1: from all fwmark 0x1 lookup 1
32766: from all lookup main
32767: from all lookup default
root@idspgw7-a:/etc/openvpn# ip r
default via 10.250.60.254 dev enp1s0 onlink
10.249.58.57 dev tunvc7s1vs7s2 proto kernel scope link src 10.249.57.58
10.250.60.0/24 dev enp1s0 proto kernel scope link src 10.250.60.130
169.254.0.0/16 dev enp1s0 scope link metric 1000
root@idspgw7-a:/etc/openvpn# ip r s t 1
default via 10.249.58.57 dev tunvc7s1vs7s2
root@idspgw7-a:/etc/openvpn# iptables -t mangle -nvL OUTPUT
Chain OUTPUT (policy ACCEPT 366K packets, 28M bytes)
pkts bytes target prot opt in out source destination
6405 677K RETURN udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:1194
0 0 MARK all -- * * 0.0.0.0/0 10.250.60.121 MARK set 0x1
root@idspgw7-a:/etc/openvpn# iptables -t raw -nvL # tracing to search for all ICMP packets
Chain PREROUTING (policy ACCEPT 334 packets, 26095 bytes)
pkts bytes target prot opt in out source destination
160 14080 TRACE icmp -- * * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 267 packets, 29993 bytes)
pkts bytes target prot opt in out source destination
80 7040 TRACE icmp -- * * 0.0.0.0/0 0.0.0.0/0
Since the packet enters the tunnel, it must have had a route out of the server to the client.
I would expect to see the packet exit the tunnel on the client side, and for the packet to then enter netfilter on the client, like for other packets
You have --tls-server/client mixed with --topology p2p .. I forgot to mention that you will probably need an --iroute. (The 'i' is deliberate)
All code snippets from previous posts can be ignored. I repost all here, simplified to just a few settings, free of TLS, no scrollbars required.
Once I find a solution, I would be happy to re-phrase the question one more time for helping the bulletin board with a good topic name....
*** I continued to verify the symmetry of my test config and found that it is only server-to-client packets that do not emerge,
*** so my initial somewhat dubious theory about the situation having to do with addresses on other interfaces is hereby debunked.
Client-to-server rides just fine through the tunnel and is received by the server even with that dst address on another intrerface.
On your suggestion, I thought I would try adding "iroute 0.0.0.0 0.0.0.0" to the server config. From the man page:
"This option must be specified either in a client instance config file using --client-config-dir or dynamically generated using a --client-connect script."
Both --client-config-dir and --client-connect caused a message saying the option requires --mode server, and I am using --mode p2p.
It makes sense the situation could be due to something similar to the iroute mechanism. I wonder what could affect p2p mode.
I am considering trying mode server with only one client, still I specifically selected p2p for its simplicity, so I would prefer to stay with p2p.
I really like using only two IP addresses and setting up the via routes directly.
Here is the symmetric simplified config:
SERVER
dev tuntestserver
local 10.250.60.121
port 1194
proto udp
remote 10.250.60.130 1194
topology p2p
ifconfig 10.249.58.57 10.249.57.58
route-noexec
CLIENT
dev tuntestclient
local 10.250.60.130
port 1194
proto udp
remote 10.250.60.121 1194
topology p2p
ifconfig 10.249.57.58 10.249.58.57
route-noexec
service openvpn@vpn-testserver restart
iptables -t raw -F ; iptables -t mangle -F
iptables -t raw -A OUTPUT -p icmp -d 10.250.60.0/24 -j TRACE
iptables -t raw -A PREROUTING -p icmp -d 10.250.60.0/24 -j TRACE
iptables -t mangle -A OUTPUT -p udp --dport 1194 -j RETURN
iptables -t mangle -A OUTPUT -d 10.250.60.130 -j MARK --set-mark 1
ip rule delete priority 1
ip rule add priority 1 fwmark 1 lookup 1
ip route add table 1 default via 10.249.57.58 dev tuntestserver
ip route show
ip route show table 1
ip rule show
tcpdump -i tuntestserver &
service openvpn@vpn-testserver restart
iptables -t raw -F ; iptables -t mangle -F
iptables -t raw -A OUTPUT -p icmp -d 10.250.60.0/24 -j TRACE
iptables -t raw -A PREROUTING -p icmp -d 10.250.60.0/24 -j TRACE
iptables -t mangle -A OUTPUT -p udp --dport 1194 -j RETURN
iptables -t mangle -A OUTPUT -d 10.250.60.130 -j MARK --set-mark 1
ip rule delete priority 1
ip rule add priority 1 fwmark 1 lookup 1
ip route add table 1 default via 10.249.57.58 dev tuntestserver
ip route show
ip route show table 1
ip rule show
tcpdump -i tuntestserver &
+ ip route show
default via 10.250.60.254 dev eth0 onlink
10.249.57.58 dev tuntestserver proto kernel scope link src 10.249.58.57
10.250.60.0/24 dev eth0 proto kernel scope link src 10.250.60.121
+ ip route show table 1
default via 10.249.57.58 dev tuntestserver
+ ip rule show
0: from all lookup local
1: from all fwmark 0x1 lookup 1
32766: from all lookup main
32767: from all lookup default
+ ip route show
default via 10.250.60.254 dev enp1s0 onlink
10.249.58.57 dev tuntestclient proto kernel scope link src 10.249.57.58
10.250.60.0/24 dev enp1s0 proto kernel scope link src 10.250.60.130
+ ip route show table 1
default via 10.249.58.57 dev tuntestclient
+ ip rule show
0: from all lookup local
1: from all fwmark 0x1 lookup 1
32766: from all lookup main
32767: from all lookup default
Ping 10.250.60.130 from server to client, tcpdump on the server:
18:06:19.113623 IP 10.250.60.130 > 10.250.60.121: ICMP echo request, id 3140, seq 1, length 64
18:06:19.113698 IP 10.250.60.121 > 10.250.60.130: ICMP echo reply, id 3140, seq 1, length 64
18:06:20.112836 IP 10.250.60.130 > 10.250.60.121: ICMP echo request, id 3140, seq 2, length 64
18:06:20.112894 IP 10.250.60.121 > 10.250.60.130: ICMP echo reply, id 3140, seq 2, length 64
18:06:21.112836 IP 10.250.60.130 > 10.250.60.121: ICMP echo request, id 3140, seq 3, length 64
18:06:21.112909 IP 10.250.60.121 > 10.250.60.130: ICMP echo reply, id 3140, seq 3, length 64
I am thinking in the client-to-server case, the replies never make it back to the client
likely for the same reason the server-to-client source packets don't make it.
iptables trace for ping 10.250.60.121 leaving the client:
To be complete, so this post can stand by itself, per your suggestion I have verified forwarding is enabled,
and I have also disabled rp filtering on both machines.
Hello,
The plot thickens.
While augmenting my test scenario on your request, I decided to switch computers in the lab, and the pings started happening. Hmmm, said I.
The bottom line is if I run openvpn-2.3.10-1ubuntu2.1 on both sides my pings succeed.
I had openvpn-2.4.0-6+deb9u2 on one side when I had the asymmetric behavior.
Real good news for me and our members is that on my production Debian 9 servers I copied over an amd64 openvpn-2.3.10-1ubuntu2.1 binary
onto both sides, and now my hairpin NAT setup is running smoothly.
So says I, what next?
I went back to my experiment machines and built openvpn-2.5_git on both machines, one i386, one amd64.
I can confirm that my experiment as described in my previous post shows the asymmetric tcpdump behavior in 2.5_git,
meaning I see certain packets only on 1-ot-of-2 tcpdumps.
I switch to 2.3.10 on both sides, restart the tunnel, and the pings happen.
I also noticed in my last post I sent the content of experiment-server.sh twice so I repeat all files below.
These files are now using 10.250.60.120 and 10.250.60.130 as the tunnel addresses which is slightly more symmetrical.
I also added 10.250.120.120 on a bridge on the client and 10.250.130.130 on a bridge on the server to prove general routing.
SERVER
dev tuntestserver
local 10.250.60.120
port 1194
proto udp
remote 10.250.60.130 1194
topology p2p
ifconfig 10.249.58.57 10.249.57.58
route-noexec
CLIENT
dev tuntestclient
local 10.250.60.130
port 1194
proto udp
remote 10.250.60.120 1194
topology p2p
ifconfig 10.249.57.58 10.249.58.57
route-noexec
auto eth0
iface eth0 inet static
address 10.250.60.120
netmask 255.255.255.0
gateway 10.250.60.254
post-up ip link add brserver type bridge
post-up ip link set brserver up
post-up ip address add 10.250.120.120/24 dev brserver
auto enp1s0
iface enp1s0 inet static
address 10.250.60.130
netmask 255.255.255.0
gateway 10.250.60.254
post-up ip link add brclient type bridge
post-up ip link set brclient up
post-up ip address add 10.250.130.130/24 dev brclient
service openvpn@vpn-testserver restart
iptables -t raw -F ; iptables -t mangle -F
iptables -t raw -A OUTPUT -p icmp -d 10.250.0.0/16 -j TRACE
iptables -t raw -A PREROUTING -p icmp -d 10.250.0.0/16 -j TRACE
iptables -t mangle -A OUTPUT -p udp --dport 1194 -j RETURN
iptables -t mangle -A OUTPUT -d 10.250.60.130 -j MARK --set-mark 1
ip rule delete priority 1 ; ip rule add priority 1 fwmark 1 lookup 1
ip route replace table 1 default via 10.249.57.58 dev tuntestserver
ip route replace 10.250.130.0/24 dev tuntestserver src 10.250.120.120
ip route show
ip route show table 1
ip rule show
tcpdump -i tuntestserver &
service openvpn@vpn-testclient restart
iptables -t raw -F ; iptables -t mangle -F
iptables -t raw -A OUTPUT -p icmp -d 10.250.60.0/24 -j TRACE
iptables -t raw -A PREROUTING -p icmp -d 10.250.60.0/24 -j TRACE
iptables -t mangle -A OUTPUT -p udp --dport 1194 -j RETURN
iptables -t mangle -A OUTPUT -d 10.250.60.120/31 -j MARK --set-mark 1
ip rule delete priority 1 ; ip rule add priority 1 fwmark 1 lookup 1
ip route replace table 1 default via 10.249.58.57 dev tuntestclient
ip route replace 10.250.120.0/24 dev tuntestclient src 10.250.130.130
ip route show
ip route show table 1
ip rule show
tcpdump -i tuntestclient &
+ ip route show
default via 10.250.60.254 dev eth0 onlink
10.249.57.58 dev tuntestserver proto kernel scope link src 10.249.58.57
10.250.60.0/24 dev eth0 proto kernel scope link src 10.250.60.120
10.250.120.0/24 dev brserver proto kernel scope link src 10.250.120.120
10.250.130.0/24 dev tuntestserver scope link src 10.250.120.120
+ ip route show table 1
default via 10.249.57.58 dev tuntestserver
+ ip rule show
0: from all lookup local
1: from all fwmark 0x1 lookup 1
32766: from all lookup main
32767: from all lookup default
+ ip route show
default via 10.250.60.254 dev enp1s0 onlink
10.249.58.57 dev tuntestclient proto kernel scope link src 10.249.57.58
10.250.60.0/24 dev enp1s0 proto kernel scope link src 10.250.60.130
10.250.120.0/24 dev tuntestclient scope link src 10.250.130.130
10.250.130.0/24 dev brclient proto kernel scope link src 10.250.130.130
+ ip route show table 1
default via 10.249.58.57 dev tuntestclient
+ ip rule show
0: from all lookup local
1: from all fwmark 0x1 lookup 1
32766: from all lookup main
32767: from all lookup default
Now the fun.
With 2.3.10 on both sides, prove correct ping from client to server bridge:
19:46:59.523940 IP 10.250.60.130 > 10.250.60.120: ICMP echo request, id 2020, seq 1, length 64
19:46:59.524013 IP 10.250.60.120 > 10.250.60.130: ICMP echo reply, id 2020, seq 1, length 64
19:47:00.524931 IP 10.250.60.130 > 10.250.60.120: ICMP echo request, id 2020, seq 2, length 64
19:47:00.524976 IP 10.250.60.120 > 10.250.60.130: ICMP echo reply, id 2020, seq 2, length 64
And now repeat the tests with openerp-2.5_git from the master branch,
meaning I replace the binaries and re-run the experiment script on both sides,
same exact addressing, same exact routing rules and policies:
ping from client to the server bridge to prove routing:
occasionally if restarting the server and client and re-running the test, I do see the a first reply
in the server tcpdump, never any indicator of seq 2 or seq 3:
20:02:18.246623 IP 10.250.60.130 > 10.250.60.120: ICMP echo request, id 2370, seq 1, length 64
20:02:18.246704 IP 10.250.60.120 > 10.250.60.130: ICMP echo reply, id 2370, seq 1, length 64
(occasionally I see like that, stops here, reply never makes it back to client, seq 2 never arrives at server)
so the bottom line is, I wonder what is new in 2.5_git from 2.3.10,
and perhaps is there a setting or patch I could make so I can move my live servers forward
from the 2.3.10-ubuntu16 patch I applied over debian 9. I am happy with the patch for now....
02:93:b8:44:86:2d:08:00 SRC=192.168.56.157 DST=96.64.159.106 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=1166 DF PROTO=TCP SPT=42252 DPT=64008 SEQ=1100496672 ACK=0 WINDOW=29200 SYN
