sending packets whose destination address matches an IP on another interface on the peer

jimays

Hello,
This is my first posting here, gratitude for all the good posts.

I have a peer-to-peer setup which is functioning great.
I use the peer addresses as via addresses for routing to cross-network IPs.

For various reasons relative to correctly handling hairpin NAT, I have SNAT'd a packet into the tunnel so that it arrives at its
destination with a source address matching the IP address of another interface on the same source machine.

The peer receives the packet correctly, and responds into the tunnel, visible by tcpdump on the peer.

The tcpdump on the source side has yet to ever show a return packet.
This is a packet whose DST= matches the IP on another interface on the destination machine,
that is on the destination machine for the return packet, which is the original source machine.

192.168.56.157 -> 96.64.59.106:64008 original packet from internal source arriving at machine with vpn tunnel
nat prerouting dnat(96.64.159.106:64008 to 192.168.56.248:8024)
nat postrouting snat(192.168.56.157->96.64.159.105)
routes into tunnel due to packet mark and special routing table
96.64.159.105 -> 192.168.56.248 at peer arrives like this, forwards to internal destination
92.168.56.248 -> 96.64.159.105 response from internal arrives at peer
routes into tunnel due to packet mark and special routing table
tcpdump shows the reply packet entering the tunnel, heading back through to tunnel to the original source side
and where o where can I find evidence of this reply packet in the tcpdump on that original source side?

Here is the iptables TRACE for the source packet coming arriving from its internal source and being forwarded into the tunnel.

0 6 - 05/Mar/2018:12:17:58 -0700 TRACE: raw:RW_GENERAL_TRACING:rule:8 IN=vzbrI56 MAC=fe:3f:44:89:ae:ab:02:93:b8:44:86:2d:08:00 SRC=192.168.56.157 DST=96.64.159.106 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=1166 DF PROTO=TCP SPT=42252 DPT=64008 SEQ=1100496672 ACK=0 WINDOW=29200 SYN
...
0 6 - 05/Mar/2018:12:17:58 -0700 TRACE: nat:SN_EPHEM_A_7_3:rule:2 OUT=tunvs7s2vc7s1 SRC=192.168.56.157 DST=192.168.56.248 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=1166 DF PROTO=TCP SPT=42252 DPT=8024 SEQ=1100496672 ACK=0 WINDOW=29200 SYN mark=1073743619

The nat rule changes the source address to 96.64.159.105, which is an IP owned by an ethernet interface on the source machine.

iptables -A SN_EPHEM_A_7_3 -s 192.168.56.0/23 -j SNAT --to-source 96.64.159.105

Here is the trace for the source packet emerging at the peer and being forwarded to its internal destination.

0 6 - 05/Mar/2018:12:17:58 -0700 TRACE: raw:RW_GENERAL_TRACING:rule:8 IN=tunvc7s1vs7s2 SRC=96.64.159.105 DST=192.168.56.248 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=1166 DF PROTO=TCP SPT=42252 DPT=8024 SEQ=1100496672 ACK=0 WINDOW=29200 SYN
...
0 6 - 05/Mar/2018:12:17:58 -0700 TRACE: nat:POSTROUTING:policy:6 OUT=vzbrI56 SRC=96.64.159.105 DST=192.168.56.248 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=1166 DF PROTO=TCP SPT=42252 DPT=8024 SEQ=1100496672 ACK=0 WINDOW=29200 SYN mark=1073741824

Here is the return packet coming from the internal destination back to the peer and being forwarded back to the source.

0 6 - 05/Mar/2018:12:17:58 -0700 TRACE: raw:RW_GENERAL_TRACING:rule:8 IN=vzbrI56 MAC=fe:d4:d2:c6:3f:6c:ae:08:42:f8:f8:26:08:00 SRC=192.168.56.248 DST=96.64.159.105 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=8024 DPT=42252 SEQ=4232934541 ACK=1100496673 WINDOW=28960 ACK SYN
...
0 6 - 05/Mar/2018:12:17:58 -0700 TRACE: mangle:POSTROUTING:policy:5 OUT=tunvc7s1vs7s2 SRC=192.168.56.248 DST=96.64.159.105 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=8024 DPT=42252 SEQ=4232934541 ACK=1100496673 WINDOW=28960 ACK SYN mark=18180

Here is the tcpdump of the return packet entering the tunnel.

17:19:00.508904 IP odoo7-248.idsp56.net.8024 > 96-64-159-105-static.hfc.comcastbusiness.net.40004: Flags [S.], seq 782673324, ack 2663188677, win 28960, options [mss 1460,sackOK,TS val 1538743048 ecr 1922512830,nop,wscale 7], length 0

I have the same iptables trace rule on both sides, so I expect to see the packet emerge, ready for dnat.

iptables -t raw -I RW_GENERAL_TRACING 4 -p tcp --sport 8024 -s 192.168.56.248 -j TRACE

I also ran a full wildcard tcpdump on the receiving side and never see the packet.
This is a packet from 192.168.56.248 to 96.64.159.105, through a tunnel to a target where 96.64.159.105 is an IP on another interface.

Can anyone offer any insight as to whether this is the expected behavior of openvpn in this case?
Can anyone suggest how I can see my packet? I do have more trace output, tried to be minimal for this first posting.
I am willing to adjust a local copy of the C code, though I would prefer a config-based solution.

Gratitude.

jimays

I restate my question more concisely:

Why would any TCP packet only appear on one side of a peer-to-peer in tcpdump and iptables trace?

I would be very grateful for some enlightenment.

TinCanTech

Please see:
HOWTO: Request Help !

jimays

Hello openvpn seers.
I am attempting to be more conformant to 'How to Post'...
and am posting a simpler case that illustrates the same situation.

SERVER@10.250.60.121 with tunnel endpoint 10.249.58.57
CLIENT@10.250.60.130 with tunnel endpoint 10.249.57.58

dev tunvs7s2vc7s1
local 10.250.60.121
port 1194
proto udp
remote 10.250.60.130 1194
float

topology p2p
ifconfig 10.249.58.57 10.249.57.58
script-security 2
route-noexec
#route-up "/tru/mnt/realm-net-tools/bin/realm-net-routing --run tunvs7s2vc7s1"

tls-server
ping-restart 120

ping 10
persist-key
persist-tun

ca /etc/openvpn/realm-certs/^inter-dimensional-space-port.vpn.crt
cert /etc/openvpn/realm-certs/vpn-link-server/^inter-dimensional-space-port.net^vpn-vs7s2vc7s1.inter-dimensional-space-port.net.crt
key /etc/openvpn/realm-certs/vpn-link-server/^inter-dimensional-space-port.net^vpn-vs7s2vc7s1.inter-dimensional-space-port.net.key
dh /etc/openvpn/realm-certs/dh2048.pem
cipher AES-256-CBC

verb 3
status /dev/null

;log openvpn.log
;log-append openvpn.log
;mute 20

user nobody
group nogroup

dev tunvc7s1vs7s2
local 10.250.60.130
port 1194
proto udp
remote 10.250.60.121 1194
float

topology p2p
ifconfig 10.249.57.58 10.249.58.57
script-security 2
route-noexec
#route-up "/etc/realm-net/moc7-41.route-up.sh"

tls-client
ping-restart 180

ping 10
persist-key
persist-tun

ca /etc/openvpn/realm-certs/^inter-dimensional-space-port.vpn.crt
cert /etc/openvpn/realm-certs/vpn-link-client/^inter-dimensional-space-port.net^vpn-vc7s1vs7s2.inter-dimensional-space-port.net.crt
key /etc/openvpn/realm-certs/vpn-link-client/^inter-dimensional-space-port.net^vpn-vc7s1vs7s2.inter-dimensional-space-port.net.key

cipher AES-256-CBC

verb 3
status /dev/null

;log openvpn.log
;log-append openvpn.log
;mute 20

user nobody
group nogroup

On the server:

service openvpn@vpn-vs7s2vc7s1 start
iptables -t raw -A OUTPUT -d 10.250.60.130 -j TRACE
iptables -t mangle -A OUTPUT -p udp --dport 1194 -j RETURN
iptables -t mangle -A OUTPUT -d 10.250.60.130 -j MARK --set-mark 1
ip rule add priority 1 fwmark 1 lookup 1
ip route add table 1 default via 10.249.57.58 dev tunvs7s2vc7s1
tcpdump -i tunvs7s2vc7s1 # watch server side of tunnel

On the client:

service openvpn@vpn-vc7s1vs7s2 start
iptables -t raw -A OUTPUT -d 10.250.60.121 -j TRACE
iptables -t mangle -A OUTPUT -p udp --dport 1194 -j RETURN
iptables -t mangle -A OUTPUT -d 10.250.60.121 -j MARK --set-mark 1
ip rule add priority 1 fwmark 1 lookup 1
ip route add table 1 default via 10.249.58.57 dev tunvs7s2vc7s1
tcpdump -i tunvs7s2vc7s1 # watch client side of tunnel

From another window on server:

ping -I 10.249.58.57 10.249.57.58 # successful, shows in both tcpdump displays and both iptables traces
ping 10.250.60.130 # shows only in server tcpdump and iptables

This packet has src=10.250.60.121, dst=10.250.60.130, and that dst is an IP on a non-tunnel interface on the destination machine.
I am thinking openvpn is by default reluctant to source into a box a packet that would appear to come from one of its own other interfaces.

My question is how can I recover this packet on the peer?
Gratitude.

TinCanTech

jimays wrote server
local 10.250.60.121

jimays wrote client
local 10.250.60.130

jimays wrote ping 10.250.60.130 # shows only in server tcpdump and iptables

Assuming /24 then such a packet will not be routed via the tunnel ...

jimays

Kindly excuse the typo in the tunnel names in my last post.
These commands here were pasted directly from my test scripts.

experiment-server.sh:

service openvpn@vpn-vs7s2vc7s1 start
iptables -t raw -A OUTPUT -d 10.250.60.130 -j TRACE
iptables -t mangle -A OUTPUT -p udp --dport 1194 -j RETURN
iptables -t mangle -A OUTPUT -d 10.250.60.130 -j MARK --set-mark 1
ip rule add priority 1 fwmark 1 lookup 1
ip route add table 1 default via 10.249.57.58 dev tunvs7s2vc7s1
tcpdump -i tunvs7s2vc7s1 # watch server side of tunnel

experiment-client.sh:

service openvpn@vpn-vc7s1vs7s2 start
iptables -t raw -A OUTPUT -d 10.250.60.121 -j TRACE
iptables -t mangle -A OUTPUT -p udp --dport 1194 -j RETURN
iptables -t mangle -A OUTPUT -d 10.250.60.121 -j MARK --set-mark 1
ip rule add priority 1 fwmark 1 lookup 1
ip route add table 1 default via 10.249.58.57 dev tunvc7s1vs7s2
tcpdump -i tunvc7s1vs7s2 # watch client side of tunnel

running on the server: ping -I 10.249.58.57 10.249.57.58

tcpdump on server:

11:29:43.333416 IP 10.249.58.57 > 10.249.57.58: ICMP echo request, id 7988, seq 1, length 64
11:29:43.334120 IP 10.249.57.58 > 10.249.58.57: ICMP echo reply, id 7988, seq 1, length 64
11:29:44.335522 IP 10.249.58.57 > 10.249.57.58: ICMP echo request, id 7988, seq 2, length 64
11:29:44.336075 IP 10.249.57.58 > 10.249.58.57: ICMP echo reply, id 7988, seq 2, length 64
11:29:45.359535 IP 10.249.58.57 > 10.249.57.58: ICMP echo request, id 7988, seq 3, length 64
11:29:45.360066 IP 10.249.57.58 > 10.249.58.57: ICMP echo reply, id 7988, seq 3, length 64
...

tcpdump on client:

11:29:42.965634 IP 10.249.58.57 > 10.249.57.58: ICMP echo request, id 7988, seq 1, length 64
11:29:42.965661 IP 10.249.57.58 > 10.249.58.57: ICMP echo reply, id 7988, seq 1, length 64
11:29:43.967520 IP 10.249.58.57 > 10.249.57.58: ICMP echo request, id 7988, seq 2, length 64
11:29:43.967541 IP 10.249.57.58 > 10.249.58.57: ICMP echo reply, id 7988, seq 2, length 64
11:29:44.991500 IP 10.249.58.57 > 10.249.57.58: ICMP echo request, id 7988, seq 3, length 64
11:29:44.991519 IP 10.249.57.58 > 10.249.58.57: ICMP echo reply, id 7988, seq 3, length 64
...

running on the server: ping 10.250.60.130

tcpdump on server, which I believe is showing packets actually entering the tunnel on the server side:

11:33:17.400671 IP 10.250.60.121 > 10.250.60.130: ICMP echo request, id 14193, seq 1, length 64
11:33:18.415545 IP 10.250.60.121 > 10.250.60.130: ICMP echo request, id 14193, seq 2, length 64
11:33:19.439537 IP 10.250.60.121 > 10.250.60.130: ICMP echo request, id 14193, seq 3, length 64
...

tcpdump on client, which I am expecting to show all packets entering from the server:

(empty, and why would that be, and how can we see these packets?)

Gratitude for the assistance.

TinCanTech

I can see what you are trying to do ..
I do not understand why you are trying to do it ..

For me this has nothing to do with openvpn ..

* Thread has been moved *

As a side note: I presume IP forwarding is enabled on both hosts.

EDIT: perhaps your routing tables will have the information you need ..

jimays

Good idea to check the packet forwarding. I also checked reverse-packet filtering.

root@idspgw7-a:/etc/openvpn# cat /proc/sys/net/ipv4/ip_forward
1
root@idspgw7-a:/etc/openvpn# cat /proc/sys/net/ipv4/conf/all/rp_filter
0
root@idspgw7-a:/etc/openvpn# cat /proc/sys/net/ipv4/conf/enp1s0/rp_filter
0
root@idspgw7-a:/etc/openvpn# cat /proc/sys/net/ipv4/conf/tunvc7s1vs7s2/rp_filter
0
root@ahau1:/etc/openvpn# cat /proc/sys/net/ipv4/ip_forward
1
root@ahau1:/etc/openvpn# cat /proc/sys/net/ipv4/conf/all/rp_filter
0
root@ahau1:/etc/openvpn# cat /proc/sys/net/ipv4/conf/eth0/rp_filter
0
root@ahau1:/etc/openvpn# cat /proc/sys/net/ipv4/conf/tunvs7s2vc7s1/rp_filter
0

The routing rules are set such that non-port-1194 traffic between the two external endpoints is re-routed through the tunnel.
The reason is related to how I am handling hairpin nat within a larger connected network, which I could discuss if thought helpful.
I have isolated the case here down to two IP addresses in a reproducable case just to analyze the openvpn interaction.

Since the packet enters the tunnel, it must have had a route out of the server to the client.
I would expect to see the packet exit the tunnel on the client side, and for the packet to then enter netfilter on the client, like for other packets.

For example, if I have a machine with eth0=1.1.1.1/24 and eth1=2.2.2.2/24, and I send in a packet from another host 1.1.1.2/24 via 1.1.1.1,
and that packet has src=3.3.3.3/dst=2.2.2.2, then the packet still enters netfilter, the trace shows, and with proper rules the packet can be routed
and enters the INPUT or FORWARD chains.

I would expect the same behavior from a tunnel interface as for eth0 in the example.

In my experiment, I see no packet exiting the tunnel on the client side and no packet in the client iptables -t raw -j TRACE.
I run tcpdump on both sides and only see the packet on the source side, so it seems openvpn is not passing this packet.

*** Am I correct in thinking the client and server tcpdump streams on a p2p should be identical regardless of any routing rules on either side?

For reference, the routing rules are the default at-boot rules as augmented by starting openvpn and running the ip rule and ip route commands.
Here is the final setup for the experiment:

On the server:

root@ahau1:/etc/openvpn# ip ru
0:	from all lookup local 
1:	from all fwmark 0x1 lookup 1 
32766:	from all lookup main 
32767:	from all lookup default 

root@ahau1:/etc/openvpn# ip r
default via 10.250.60.254 dev eth0 onlink 
10.249.57.58 dev tunvs7s2vc7s1 proto kernel scope link src 10.249.58.57 
10.250.60.0/24 dev eth0 proto kernel scope link src 10.250.60.121 

root@ahau1:/etc/openvpn# ip r s t 1
default via 10.249.57.58 dev tunvs7s2vc7s1

root@ahau1:/etc/openvpn# iptables -t mangle -nvL OUTPUT
Chain OUTPUT (policy ACCEPT 7670 packets, 829K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 6412  680K RETURN     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:1194
   75  6300 MARK       all  --  *      *       0.0.0.0/0            10.250.60.130        MARK set 0x1

On the client:

root@idspgw7-a:/etc/openvpn# ip ru
0:	from all lookup local 
1:	from all fwmark 0x1 lookup 1 
32766:	from all lookup main 
32767:	from all lookup default 

root@idspgw7-a:/etc/openvpn# ip r
default via 10.250.60.254 dev enp1s0 onlink 
10.249.58.57 dev tunvc7s1vs7s2  proto kernel  scope link  src 10.249.57.58 
10.250.60.0/24 dev enp1s0  proto kernel  scope link  src 10.250.60.130 
169.254.0.0/16 dev enp1s0  scope link  metric 1000 

root@idspgw7-a:/etc/openvpn# ip r s t 1
default via 10.249.58.57 dev tunvc7s1vs7s2

root@idspgw7-a:/etc/openvpn# iptables -t mangle -nvL OUTPUT
Chain OUTPUT (policy ACCEPT 366K packets, 28M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 6405  677K RETURN     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:1194
    0     0 MARK       all  --  *      *       0.0.0.0/0            10.250.60.121        MARK set 0x1

root@idspgw7-a:/etc/openvpn# iptables -t raw -nvL  # tracing to search for all ICMP packets
Chain PREROUTING (policy ACCEPT 334 packets, 26095 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  160 14080 TRACE      icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
Chain OUTPUT (policy ACCEPT 267 packets, 29993 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   80  7040 TRACE      icmp --  *      *       0.0.0.0/0            0.0.0.0/0

Gratitude.

TinCanTech

jimays wrote Since the packet enters the tunnel, it must have had a route out of the server to the client.
I would expect to see the packet exit the tunnel on the client side, and for the packet to then enter netfilter on the client, like for other packets

You have --tls-server/client mixed with --topology p2p .. I forgot to mention that you will probably need an --iroute. (The 'i' is deliberate)

See --iroute in The Manual v24x

Also, this explains in more detail:
HOWTO: Expanding the scope of the VPN to include additional machines

Hopefully, that should fix it :geek:

jimays

Sorry, we encountered an error while displaying this content. If you're a user, please try again later. If you're an administrator, take a look in your Flarum log files for more information.

TinCanTech

There is a simple solution .. 😉

https://openvpn.net/index.php/open-source/documentation/miscellaneous/78-static-key-mini-howto.html

Make sure it works .. then apply your firewall and routing.

jimays

Hello,
The plot thickens.
While augmenting my test scenario on your request, I decided to switch computers in the lab, and the pings started happening. Hmmm, said I.

The bottom line is if I run openvpn-2.3.10-1ubuntu2.1 on both sides my pings succeed.
I had openvpn-2.4.0-6+deb9u2 on one side when I had the asymmetric behavior.

Real good news for me and our members is that on my production Debian 9 servers I copied over an amd64 openvpn-2.3.10-1ubuntu2.1 binary
onto both sides, and now my hairpin NAT setup is running smoothly.

So says I, what next?

I went back to my experiment machines and built openvpn-2.5_git on both machines, one i386, one amd64.
I can confirm that my experiment as described in my previous post shows the asymmetric tcpdump behavior in 2.5_git,
meaning I see certain packets only on 1-ot-of-2 tcpdumps.

I switch to 2.3.10 on both sides, restart the tunnel, and the pings happen.

I also noticed in my last post I sent the content of experiment-server.sh twice so I repeat all files below.
These files are now using 10.250.60.120 and 10.250.60.130 as the tunnel addresses which is slightly more symmetrical.
I also added 10.250.120.120 on a bridge on the client and 10.250.130.130 on a bridge on the server to prove general routing.

dev tuntestserver
local 10.250.60.120
port 1194
proto udp
remote 10.250.60.130 1194
topology p2p
ifconfig 10.249.58.57 10.249.57.58
route-noexec

dev tuntestclient
local 10.250.60.130
port 1194
proto udp
remote 10.250.60.120 1194
topology p2p
ifconfig 10.249.57.58 10.249.58.57
route-noexec

/etc/network/interfaces server:

auto eth0
iface eth0 inet static
  address 10.250.60.120
  netmask 255.255.255.0
  gateway 10.250.60.254
  post-up ip link add brserver type bridge
  post-up ip link set brserver up
  post-up ip address add 10.250.120.120/24 dev brserver

/etc/network/interfaces client:

auto enp1s0
iface enp1s0 inet static
  address 10.250.60.130
  netmask 255.255.255.0
  gateway 10.250.60.254
  post-up ip link add brclient type bridge
  post-up ip link set brclient up
  post-up ip address add 10.250.130.130/24 dev brclient

experiment-server.sh

service openvpn@vpn-testserver restart
iptables -t raw -F ; iptables -t mangle -F
iptables -t raw -A OUTPUT -p icmp -d 10.250.0.0/16 -j TRACE
iptables -t raw -A PREROUTING -p icmp -d 10.250.0.0/16 -j TRACE
iptables -t mangle -A OUTPUT -p udp --dport 1194 -j RETURN
iptables -t mangle -A OUTPUT -d 10.250.60.130 -j MARK --set-mark 1
ip rule delete priority 1 ; ip rule add priority 1 fwmark 1 lookup 1
ip route replace table 1 default via 10.249.57.58 dev tuntestserver
ip route replace 10.250.130.0/24 dev tuntestserver src 10.250.120.120
ip route show
ip route show table 1
ip rule show
tcpdump -i tuntestserver &

experiment-client.sh

service openvpn@vpn-testclient restart
iptables -t raw -F ; iptables -t mangle -F
iptables -t raw -A OUTPUT -p icmp -d 10.250.60.0/24 -j TRACE
iptables -t raw -A PREROUTING -p icmp -d 10.250.60.0/24 -j TRACE
iptables -t mangle -A OUTPUT -p udp --dport 1194 -j RETURN
iptables -t mangle -A OUTPUT -d 10.250.60.120/31 -j MARK --set-mark 1
ip rule delete priority 1 ; ip rule add priority 1 fwmark 1 lookup 1
ip route replace table 1 default via 10.249.58.57 dev tuntestclient
ip route replace 10.250.120.0/24 dev tuntestclient src 10.250.130.130
ip route show
ip route show table 1
ip rule show
tcpdump -i tuntestclient &

bash -x experiment-server.sh # excerpted

+ ip route show
default via 10.250.60.254 dev eth0 onlink 
10.249.57.58 dev tuntestserver proto kernel scope link src 10.249.58.57 
10.250.60.0/24 dev eth0 proto kernel scope link src 10.250.60.120 
10.250.120.0/24 dev brserver proto kernel scope link src 10.250.120.120 
10.250.130.0/24 dev tuntestserver scope link src 10.250.120.120 
+ ip route show table 1
default via 10.249.57.58 dev tuntestserver 
+ ip rule show
0:	from all lookup local 
1:	from all fwmark 0x1 lookup 1 
32766:	from all lookup main 
32767:	from all lookup default

bash -x experiment-client.sh # excerpted

+ ip route show
default via 10.250.60.254 dev enp1s0 onlink 
10.249.58.57 dev tuntestclient  proto kernel  scope link  src 10.249.57.58 
10.250.60.0/24 dev enp1s0  proto kernel  scope link  src 10.250.60.130 
10.250.120.0/24 dev tuntestclient  scope link  src 10.250.130.130 
10.250.130.0/24 dev brclient  proto kernel  scope link  src 10.250.130.130 
+ ip route show table 1
default via 10.249.58.57 dev tuntestclient 
+ ip rule show
0:	from all lookup local 
1:	from all fwmark 0x1 lookup 1 
32766:	from all lookup main 
32767:	from all lookup default

Now the fun.

With 2.3.10 on both sides, prove correct ping from client to server bridge:

root@idspgw7-a:~# ping 10.250.120.120
PING 10.250.120.120 (10.250.120.120) 56(84) bytes of data.
64 bytes from 10.250.120.120: icmp_seq=1 ttl=64 time=0.965 ms
64 bytes from 10.250.120.120: icmp_seq=2 ttl=64 time=0.714 ms

tcpdump on client tunnel interface:

19:37:03.579608 IP 10.250.130.130 > 10.250.120.120: ICMP echo request, id 2013, seq 1, length 64
19:37:03.580549 IP 10.250.120.120 > 10.250.130.130: ICMP echo reply, id 2013, seq 1, length 64
19:37:04.580673 IP 10.250.130.130 > 10.250.120.120: ICMP echo request, id 2013, seq 2, length 64
19:37:04.581375 IP 10.250.120.120 > 10.250.130.130: ICMP echo reply, id 2013, seq 2, length 64

tcpdump on server tunnel interface:

19:37:03.611313 IP 10.250.130.130 > 10.250.120.120: ICMP echo request, id 2013, seq 1, length 64
19:37:03.611402 IP 10.250.120.120 > 10.250.130.130: ICMP echo reply, id 2013, seq 1, length 64
19:37:04.612448 IP 10.250.130.130 > 10.250.120.120: ICMP echo request, id 2013, seq 2, length 64
19:37:04.612530 IP 10.250.120.120 > 10.250.130.130: ICMP echo reply, id 2013, seq 2, length 64

With 2.3.10 on both sides, prove correct ping from server to client bridge:

root@ahau1:~# ping 10.250.130.130
PING 10.250.130.130 (10.250.130.130) 56(84) bytes of data.
64 bytes from 10.250.130.130: icmp_seq=1 ttl=64 time=0.909 ms
64 bytes from 10.250.130.130: icmp_seq=2 ttl=64 time=0.585 ms

tcpdump on server tunnel interface:

19:41:39.600735 IP 10.250.120.120 > 10.250.130.130: ICMP echo request, id 6922, seq 1, length 64
19:41:39.601578 IP 10.250.130.130 > 10.250.120.120: ICMP echo reply, id 6922, seq 1, length 64
19:41:40.601730 IP 10.250.120.120 > 10.250.130.130: ICMP echo request, id 6922, seq 2, length 64
19:41:40.602290 IP 10.250.130.130 > 10.250.120.120: ICMP echo reply, id 6922, seq 2, length 64

tcpdump on client tunnel interface:

19:41:39.570693 IP 10.250.120.120 > 10.250.130.130: ICMP echo request, id 6922, seq 1, length 64
19:41:39.570722 IP 10.250.130.130 > 10.250.120.120: ICMP echo reply, id 6922, seq 1, length 64
19:41:40.571449 IP 10.250.120.120 > 10.250.130.130: ICMP echo request, id 6922, seq 2, length 64
19:41:40.571467 IP 10.250.130.130 > 10.250.120.120: ICMP echo reply, id 6922, seq 2, length 64

Now run the test, first from server to client, with openvpn 2.3.10, succeeds:

root@ahau1:~# ping 10.250.60.130
PING 10.250.60.130 (10.250.60.130) 56(84) bytes of data.
64 bytes from 10.250.60.130: icmp_seq=1 ttl=64 time=0.700 ms
64 bytes from 10.250.60.130: icmp_seq=2 ttl=64 time=0.991 ms

tcpdump on the server, packets correctly routed through the tunnel:

19:44:52.104097 IP 10.250.60.120 > 10.250.60.130: ICMP echo request, id 11500, seq 1, length 64
19:44:52.104739 IP 10.250.60.130 > 10.250.60.120: ICMP echo reply, id 11500, seq 1, length 64
19:44:53.111335 IP 10.250.60.120 > 10.250.60.130: ICMP echo request, id 11500, seq 2, length 64
19:44:53.112249 IP 10.250.60.130 > 10.250.60.120: ICMP echo reply, id 11500, seq 2, length 64

tcpdump on the client:

19:44:52.066904 IP 10.250.60.120 > 10.250.60.130: ICMP echo request, id 11500, seq 1, length 64
19:44:52.066975 IP 10.250.60.130 > 10.250.60.120: ICMP echo reply, id 11500, seq 1, length 64
19:44:53.074138 IP 10.250.60.120 > 10.250.60.130: ICMP echo request, id 11500, seq 2, length 64
19:44:53.074189 IP 10.250.60.130 > 10.250.60.120: ICMP echo reply, id 11500, seq 2, length 64

And the test from client to server, with 2.3.10 succeeds:

root@idspgw7-a:~# ping 10.250.60.120
PING 10.250.60.120 (10.250.60.120) 56(84) bytes of data.
64 bytes from 10.250.60.120: icmp_seq=1 ttl=64 time=0.813 ms
64 bytes from 10.250.60.120: icmp_seq=2 ttl=64 time=0.670 ms

tcpdump on the client:

19:46:59.479822 IP 10.250.60.130 > 10.250.60.120: ICMP echo request, id 2020, seq 1, length 64
19:46:59.480576 IP 10.250.60.120 > 10.250.60.130: ICMP echo reply, id 2020, seq 1, length 64
19:47:00.480840 IP 10.250.60.130 > 10.250.60.120: ICMP echo request, id 2020, seq 2, length 64
19:47:00.481471 IP 10.250.60.120 > 10.250.60.130: ICMP echo reply, id 2020, seq 2, length 64

tcpdump on the server:

19:46:59.523940 IP 10.250.60.130 > 10.250.60.120: ICMP echo request, id 2020, seq 1, length 64
19:46:59.524013 IP 10.250.60.120 > 10.250.60.130: ICMP echo reply, id 2020, seq 1, length 64
19:47:00.524931 IP 10.250.60.130 > 10.250.60.120: ICMP echo request, id 2020, seq 2, length 64
19:47:00.524976 IP 10.250.60.120 > 10.250.60.130: ICMP echo reply, id 2020, seq 2, length 64

And now repeat the tests with openerp-2.5_git from the master branch,
meaning I replace the binaries and re-run the experiment script on both sides,
same exact addressing, same exact routing rules and policies:

ping from client to the server bridge to prove routing:

root@idspgw7-a:~# ping 10.250.120.120
PING 10.250.120.120 (10.250.120.120) 56(84) bytes of data.
64 bytes from 10.250.120.120: icmp_seq=1 ttl=64 time=0.574 ms
64 bytes from 10.250.120.120: icmp_seq=2 ttl=64 time=0.511 ms

tcpdump on client:

19:55:19.197598 IP 10.250.130.130 > 10.250.120.120: ICMP echo request, id 2239, seq 1, length 64
19:55:19.198138 IP 10.250.120.120 > 10.250.130.130: ICMP echo reply, id 2239, seq 1, length 64
19:55:20.197307 IP 10.250.130.130 > 10.250.120.120: ICMP echo request, id 2239, seq 2, length 64
19:55:20.197802 IP 10.250.120.120 > 10.250.130.130: ICMP echo reply, id 2239, seq 2, length 64

tcpdump on server:

19:55:19.232602 IP 10.250.130.130 > 10.250.120.120: ICMP echo request, id 2239, seq 1, length 64
19:55:19.232666 IP 10.250.120.120 > 10.250.130.130: ICMP echo reply, id 2239, seq 1, length 64
19:55:20.232226 IP 10.250.130.130 > 10.250.120.120: ICMP echo request, id 2239, seq 2, length 64
19:55:20.232256 IP 10.250.120.120 > 10.250.130.130: ICMP echo reply, id 2239, seq 2, length 64

and ping client bridge from server to prove routing:

[root@ahau1:~# ping 10.250.130.130
PING 10.250.130.130 (10.250.130.130) 56(84) bytes of data.
64 bytes from 10.250.130.130: icmp_seq=1 ttl=64 time=0.649 ms
64 bytes from 10.250.130.130: icmp_seq=2 ttl=64 time=0.375 ms

tcpdump on server:

19:57:19.511575 IP 10.250.120.120 > 10.250.130.130: ICMP echo request, id 30698, seq 1, length 64
19:57:19.512177 IP 10.250.130.130 > 10.250.120.120: ICMP echo reply, id 30698, seq 1, length 64
19:57:20.535237 IP 10.250.120.120 > 10.250.130.130: ICMP echo request, id 30698, seq 2, length 64
19:57:20.535590 IP 10.250.130.130 > 10.250.120.120: ICMP echo reply, id 30698, seq 2, length 64

tcpdump on client:

19:57:19.483475 IP 10.250.120.120 > 10.250.130.130: ICMP echo request, id 30698, seq 1, length 64
19:57:19.483504 IP 10.250.130.130 > 10.250.120.120: ICMP echo reply, id 30698, seq 1, length 64
19:57:20.507030 IP 10.250.120.120 > 10.250.130.130: ICMP echo request, id 30698, seq 2, length 64
19:57:20.507048 IP 10.250.130.130 > 10.250.120.120: ICMP echo reply, id 30698, seq 2, length 64

And here is the testcase, from server to client, with openvpn 2.5_git

root@ahau1:~# ping 10.250.60.130
PING 10.250.60.130 (10.250.60.130) 56(84) bytes of data.
^C
--- 10.250.60.130 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2044ms

tcpdump on server:

19:59:12.730433 IP 10.250.60.120 > 10.250.60.130: ICMP echo request, id 1072, seq 1, length 64
19:59:13.751246 IP 10.250.60.120 > 10.250.60.130: ICMP echo request, id 1072, seq 2, length 64

tcpdump on client:

(empty)

And here is the testcase from client to server, openvpn-2.5_git:

root@idspgw7-a:~# ping 10.250.60.120
PING 10.250.60.120 (10.250.60.120) 56(84) bytes of data.
^C
--- 10.250.60.120 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms

tcpdump on client:

20:00:45.679662 IP 10.250.60.130 > 10.250.60.120: ICMP echo request, id 2241, seq 1, length 64
20:00:46.679376 IP 10.250.60.130 > 10.250.60.120: ICMP echo request, id 2241, seq 2, length 64

tcpdump on server

(empty)

occasionally if restarting the server and client and re-running the test, I do see the a first reply
in the server tcpdump, never any indicator of seq 2 or seq 3:

20:02:18.246623 IP 10.250.60.130 > 10.250.60.120: ICMP echo request, id 2370, seq 1, length 64
20:02:18.246704 IP 10.250.60.120 > 10.250.60.130: ICMP echo reply, id 2370, seq 1, length 64
(occasionally I see like that, stops here, reply never makes it back to client, seq 2 never arrives at server)

so the bottom line is, I wonder what is new in 2.5_git from 2.3.10,
and perhaps is there a setting or patch I could make so I can move my live servers forward
from the 2.3.10-ubuntu16 patch I applied over debian 9. I am happy with the patch for now....

forwarding config on server:

root@ahau1:/etc/openvpn# for f in $(find /proc/sys -name rp_filter) ; do echo $f ; cat $f ; done
/proc/sys/net/ipv4/conf/all/rp_filter
0
/proc/sys/net/ipv4/conf/brserver/rp_filter
0
/proc/sys/net/ipv4/conf/default/rp_filter
0
/proc/sys/net/ipv4/conf/eth0/rp_filter
0
/proc/sys/net/ipv4/conf/eth1/rp_filter
0
/proc/sys/net/ipv4/conf/lo/rp_filter
0
/proc/sys/net/ipv4/conf/tuntestserver/rp_filter
0
root@ahau1:/etc/openvpn# cat /proc/sys/net/ipv4/ip_forward
1

forwarding config on client:

root@idspgw7-a:/etc/openvpn# for f in $(find /proc/sys -name rp_filter) ; do echo $f ; cat $f ; done
/proc/sys/net/ipv4/conf/all/rp_filter
0
/proc/sys/net/ipv4/conf/brclient/rp_filter
0
/proc/sys/net/ipv4/conf/default/rp_filter
0
/proc/sys/net/ipv4/conf/enp0s25/rp_filter
0
/proc/sys/net/ipv4/conf/enp1s0/rp_filter
0
/proc/sys/net/ipv4/conf/enp2s0/rp_filter
0
/proc/sys/net/ipv4/conf/lo/rp_filter
0
/proc/sys/net/ipv4/conf/tuntestclient/rp_filter
0
root@idspgw7-a:/etc/openvpn# cat /proc/sys/net/ipv4/ip_forward
1

Gratitude.

TinCanTech

Because you have documented this so clearly (thank you), when I find some time, I will try to replicate this.