jan.u.thomsen@gmail.com
Pippin wrote
Sorry, no relevant experience with VM and bridging.
But you added more info which should be added from the start.....😉
Just for the record, VMs was not involved in the problem. Only mentioned because it forced me to create a bridging interface so the VMs could be networked. There were no VMs active during the tests, but even if they were the symptoms are the same and the br0 interface act like an ethernet interfaces for the OpenVPN host (not a VM) for all practical purposes.
jan.u.thomsen@gmail.com
"I think you have misunderstood the problem .. what is much more likely is that the phone is sending packets to your LAN via it's local connection before the VPN has reconnected."
I beg to differ. The phone does not have a local connection. Wifi connection was disabled during the tests !
Also, please note that the offending packets are TCP packets having a source IP address from the tun0 interface on the phone with IP address 10.193.169.3, with a destination address somewhere on the internet. As far as I can see, this can only occur if the packets have travelled through the OpenVPN connection, to enter the tun0 interface on the OpenVPN server, which have IP address 10.193.169.1, from where it enters the linux firewall PREROUTING chain, and here a routing decision is made to send it through the FORWARD chain, because destination is on the internet, and subsequently through the POSTROUTING chain where the a NAT masquerade rule change the source address from 10.193.169.3 to the br0 interface IP address, and subsequently the default route for internet packets send the NATed packet to the firewall. During normal operation, the linux firewall does connection tracking. But I suspect that when the OpenVPN connection is paused by putting the phone to sleep, the firewall detect that some of the TCP connections have gone dead and tear down the associated connection tracking resources, freeing up system resources. When the OpenVPN connection resume, it sends FIN/ACK, RST/ACK and RST TCP packets for some of the TCP connections which no longer exist in linux firewall connection tracking data structure. But since there are no conneciton tracking info available for them anymore, they are considered INVALID. All this has been confirmed through wireshark capture (you proposed tcpdump but I used wireshark). Of course this does not prove that INVALID packets dont get NATed - that is a theory I cannot prove. The rest is not !! Wireshark packet capture of traffic from 10.193.169.3 on br0 prove this to be the case, in combination with the firewall rule I used which blocks INVALID packets with source address 10.193.169.3 destined for br0 . Thus, when I activate the firewall rule mentioned, I can no longer provoke any martian packet error messages, and also wireshark can no longer detect packets from 10.193.169.3 on br0.
"If iptables is misbehaving then there is a serious issue with iptables which has been missed by the iptables developers and users and the maintainers of iptables for Ubuntu .. this is extremely unlikely .."
I don't think it is necessarily so serious. As I documented above, the problem only occurs on INVALID TCP packets for which there is no connection tracking info - i.e. supposedly dead or stale connections, which do not receive the expected source nating - but maybe that is by design ? and if it is a bug, then it has very limited consequences since it only occurs on INVALID TCP packets anyway. Maye it is even by design ? I cannot se why they should choose to not do source nating on INVALID packets, but I could be missing some important cases wher that would be a good idea to do.
"I am not saying you have done this wrong but I don't believe iptables is doing what you claim."
As mentioned above, I have to disagree on this :-)
"Hint: Run tcpdump on your server."
I ran wireshark instead. I have an image but cant figure out how to attach it.
b.r.
Jan
Pippin
You have multiple physical interfaces on the same machine connected?
jan.u.thomsen@gmail.com
Pippin wrote
You have multiple physical interfaces on the same machine connected?
No. I understand your question. That would often be the case when bridging. But the sole purpose of setting up a bridge interface is to enable networking from my VMs - which did not run during testing.
TinCanTech
All you need do is:
- Delete the bridge
- Delete your iptables rules
- Use the prescribed masquerade rule
- Run tcpdump on your server
jan.u.thomsen@gmail.com
Ok. I can change the bridge to a real interface and repeat the testing. I am convinced the result will be the same.
We will see. Stay tuned for result sometime tomorrow.
jan.u.thomsen@gmail.com
I have now tested the problem again after changing the network configuration to use the native ethernet interface p2p1 rather than the bridge interface br0, and changed the MASQUERADE rule to use this interface. I also removed the firewall rule that blocked INVALID packets in the FORWARD chain with source address from the android tun0 interface.
And as I expected this change made no difference. Now I get martian packet warnings again in my router, and wireshark capture of traffic OpenVPN ethernet interface p2p1 with source IP address of Android tun0 interface (10.193.169.3) show the same result as in my previous posts using the br0 bridge interface.
Also, re-introducing the firewall rule that blocked INVALID packets in the FORWARD chain for packets with source address from the android tun0 interface, makes martian packet warnings disappear and wireshark stop showing invalid packets.
After this test my conclusion is therefore unchanged:
When OpenVPN connection automatically resume after Android phone is brought to live, there are broken TCP connections that result in INVALID TCP packets appearing in the Linux firewall of the OpenVPN server, originating from the Android tun0 interface, when they enter the OpenVPN servers tun0 interface. These incoming INVALID packets on tun0 server interface traverse the Linux firewall, and for some reason the Linux firewall only apply the MASQUERADE source NAT rule on valid packets and let the INVALID packets pass through without source NATing - which in turn makes the firewall complain of martian packets.
Now, I have no idea why the Linux firewall refuse to apply source NATing of invalid packets. This may be according to design, or a bug - who knows. In any case I don't think this is a big problem at all, since these invalid packets are destined for hosts on the internet, and caused by broken TCP connections over the tunnel interface provided by OpenVPN. They can simply be dropped in the Linux firewall FORWARD chain as I have shown. And even when they are not blocked and pass through to the router/firewall they are caught as martian packets and dropped. So one could argue why did I raise the issue at all ? Well, because I did not know then what I now know :-)
I also tried a slightly different Source NAT rule instead of the MASQUERADE rule, where I specifically specify the IP address of the br0 interface , instead of relying on the automatic IP source selection of the MASQUERADE rule :
-A POSTROUTING -s 10.193.169.0/24 -o br0 -j SNAT --to-source 10.192.168.100
The source NAT rule change makes no difference though. Same result.
Thankyou to those of you who have bothered to engage in this thread. It has been an interesting experience which have brought clarity to the underlying cause of a problem I have been wondering about for some time. I am now certain that the root cause is as described as above - the testing methodology clearly show that beyond any doubt I believe. If the described source NAT behaviour for INVALID packets is a bug in Linux firewall ? I don't really know.
I think a plausible explanation could be that the invalid packets no longer have a Linux firewall connection tracking entry (all non-INVALID connections certainly do), and if the Source NAT address was specified here, then of course there would be no source address to supplant with. Also, invalid packets do not result in the creation of a new connection tracking entry in the Linux firewall. It is a perfectly reasonable explanation and I believe it is the reason.
But even if it is considered a bug, it has no obvious ill effects as far as I can see, except for some martian packet warnings in my router/firewall. But I now block the invalid packets with the INVALID rule I posted yesterday, so those warnings have stopped. Maybe others can see possible exploits of this behaviour - personally I can't.
jan.u.thomsen@gmail.com
Just for completeness:
I opened this issue on Ask Ubuntu forum and someone there pointed me to following link, which confirmed my analysis, and clarified that this behaviour is actually known (by a few only, it seems) and it is indeed according to design of netfilter/iptables that INVALID packets could not receive source NAT processin:
https://bugzilla.netfilter.org/show_bug.cgi?id=693
Pippin
Thank for reporting back .
TinCanTech
jan.u.thomsen@gmail.com wrote
Just for completeness:
I opened this issue on Ask Ubuntu forum and someone there pointed me to following link, which confirmed my analysis, and clarified that this behaviour is actually known (by a few only, it seems) and it is indeed according to design of netfilter/iptables that INVALID packets could not receive source NAT processin:
https://bugzilla.netfilter.org/show_bug.cgi?id=693
Pippin wrote
Thank for reporting back .
@
jan .. I
also thank you .. nice work 8-)
jan.u.thomsen@gmail.com
I ended up introducing the following iptables rules to drop INVALID packets of any protocol (not just TCP) from public (RFC1918) and link-local (RFC3927) source IP addresses. This solves the problem in a more general way than the specific confguration I made for my own little OpenVPN environment. In effect it prevents "leakage" of any type of invalid private and link-local source addresses to reach internet or local router/firewall. I have monitored these rules with all I could throw at it and I have so far seen no problems at all.
-A FORWARD -s 10.0.0.0/8 -o br0 -m state --state INVALID -j DROP
-A FORWARD -s 172.16.0.0/12 -o br0 -m state --state INVALID -j DROP
-A FORWARD -s 192.168.0.0/16 -o br0 -m state --state INVALID -j DROP
-A FORWARD -s 169.254.0.0/16 -o br0 -m state --state INVALID -j DROP
It may be a little overkill, but it would work even if I change my OpenVPN tunnel subnet in the future.
