[Solved] Martian source error messages in firewall/router

jan.u.thomsen@gmail.com

Hi
I have successfully set up my home Ubuntu 16.04 Linux server with OpenVPN 2.4.7 to work with my Debian Stretch and Android Phone. To connect from the internet I have made a port forwarding rule for UDP port 1194, and both clients match to connect and is able to direct internet traffic through my home network. So from end-user perspective it seems to be working without any problems.

However, I have noticed that sometimes my Ubiquity Unifi USG-3P router sometimes complain about some "Martian Packets" in the syslog when I connect my clients to the OpenVPN server:

Mar 25 13:02:13 USG-3P kernel: IPv4: martian source 77.66.39.43 from 10.193.169.3, on dev eth1
Mar 25 13:02:16 USG-3P kernel: IPv4: martian source 154.57.158.50 from 10.193.169.3, on dev eth1
Mar 25 13:02:18 USG-3P kernel: IPv4: martian source 213.19.162.71 from 10.193.169.3, on dev eth1

In these log entries 10.193.169.3 is the IP address of the tun0 interface on my Android phone and device eth1 is the internal LAN interface of my USG-3P firewall/router, whereas the source addresses are various internet hosts. Messages like these occur on both Android and Debian hosts from time to time. It seems to me that for the Android client the problem seems to be worse after a period of ideling where the OpenVPN connection has been shut down and automatically reconnect.

I assume the router drop these packages ( I am investigating this with Ubiquity), yet it is puzzling to me how it occurs, but I fear it could be a security issue. The way I understand it OpenVPN protocol, packets from the tun0 interfaces (in this instance the Android Phone) should be encapsulated in openvpn protocol so the source address should be from my phones public IP address and destination address being my Ubuntu server.

So I would appreciate if there is anyone who can address the following questions:

1. What might be the cause of these error messages ?
2. Do they pose a security concern

I have included server.conf of my OpenVPN server:
mode server
proto udp
port 1194
server 10.193.169.0 255.255.255.0
dev tun
topology subnet
persist-key
persist-tun
keepalive 10 120

ca /etc/openvpn/easy-rsa/keys/ca.crt
cert /etc/openvpn/easy-rsa/keys/server.crt
key /etc/openvpn/easy-rsa/keys/server.key
dh /etc/openvpn/easy-rsa/keys/dh2048.pem

remote-cert-tls client

push "redirect-gateway def1"
push "dhcp-option DNS 10.192.168.1"
client-to-client
user nobody
group nogroup
tls-crypt /etc/openvpn/easy-rsa/keys/ta.key
tls-version-min 1.2

tls-cipher TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-DHE-RSA-WITH-AES-256-CBC-SHA

cipher AES-256-GCM
ncp-ciphers AES-256-GCM:AES-128-GCM
auth SHA256

compress lzo # Compression - must be turned on at both end

ifconfig-pool-persist ipp.txt
status log/openvpn-status.log
verb 3 # verbose mode

TinCanTech

jan.u.thomsen@gmail.com wrote Mar 25 13:02:13 USG-3P kernel: IPv4: martian source 77.66.39.43 from 10.193.169.3, on dev eth1
Mar 25 13:02:16 USG-3P kernel: IPv4: martian source 154.57.158.50 from 10.193.169.3, on dev eth1
Mar 25 13:02:18 USG-3P kernel: IPv4: martian source 213.19.162.71 from 10.193.169.3, on dev eth1

jan.u.thomsen@gmail.com wrote I assume the router drop these packages

Check your openvpn server log at verb 4 for more information. Post it here when the problem occurs again. Openvpn will probably have something in the log to help. Make sure to keep accurate time so the logs match the activity.

jan.u.thomsen@gmail.com

TinCanTech
thanks for your response. I follwed your recommendation. Please find the requested information below:

I had one of the martian error messages from my firewall router at 16:43:17 (I use NTP time sync on all equipm.) as follows:

Mar 25 16:43:17 USG-3P kernel: IPv4: martian source 52.218.37.147 from 10.193.169.3, on dev eth1

the log from openvpn at verb4 follows below. The martian message occurred immediately after i restarted the openvpn connection from my android phone after going to sleep (soft exit). So the log include the whole period after opening the android until it goes to sleep again.

I dont see a smoking gun. There is some warnings of various mismatches during conneciton establishing, some of which disappear at verb 3 but they also appear when there are no martians in the router logs.

Mar 25 16:42:28 xenon ovpn-server[19982]: mydroid/xx.yy.146.217:4950 SIGTERM[soft,remote-exit] received, client-instance exiting
Mar 25 16:43:17 xenon ovpn-server[19982]: MULTI: multi_create_instance called
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 Re-using SSL/TLS context
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 LZO compression initializing
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 Control Channel MTU parms [ L:1622 D:1156 EF:94 EB:0 ET:0 EL:3 ]
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 Data Channel MTU parms [ L:1622 D:1450 EF:122 EB:406 ET:0 EL:3 ]
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 Local Options String (VER=V4): 'V4,dev-type tun,link-mtu 1550,tun-mtu 1500,proto UDPv4,comp-lzo,cipher AES-256-GCM,auth [null-digest],keysize 256,key-method 2,tls-server'
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1550,tun-mtu 1500,proto UDPv4,comp-lzo,cipher AES-256-GCM,auth [null-digest],keysize 256,key-method 2,tls-client'
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 TLS: Initial packet from [AF_INET]xx.yy.146.217:24947, sid=1a3b2ef2 d6256455
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 VERIFY OK: depth=1, C=DK, ST=Capital Region, L=Copenhagen, O=Home, CN=Home, emailAddress=me@myhost.home
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 VERIFY KU OK
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 Validating certificate extended key usage
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 ++ Certificate has EKU (str) TLS Web Client Authentication, expects TLS Web Client Authentication
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 VERIFY EKU OK
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 VERIFY OK: depth=0, C=DK, ST=Capital Region, L=Copenhagen, O=Home, CN=mydroid, emailAddress=me@myhost.home
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 peer info: IV_GUI_VER=OC30Android
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 peer info: IV_VER=3.2
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 peer info: IV_PLAT=android
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 peer info: IV_NCP=2
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 peer info: IV_TCPNL=1
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 peer info: IV_PROTO=2
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 peer info: IV_LZO=1
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 peer info: IV_IPv6=0
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 peer info: IV_AUTO_SESS=1
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1550', remote='link-mtu 1522'
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 WARNING: 'auth' is used inconsistently, local='auth [null-digest]', remote='auth SHA256'
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
Mar 25 16:43:17 xenon ovpn-server[19982]: xx.yy.146.217:24947 [mydroid] Peer Connection Initiated with [AF_INET]xx.yy.146.217:24947
Mar 25 16:43:17 xenon ovpn-server[19982]: mydroid/xx.yy.146.217:24947 MULTI_sva: pool returned IPv4=10.193.169.3, IPv6=(Not enabled)
Mar 25 16:43:17 xenon ovpn-server[19982]: mydroid/xx.yy.146.217:24947 MULTI: Learn: 10.193.169.3 -> mydroid/xx.yy.146.217:24947
Mar 25 16:43:17 xenon ovpn-server[19982]: mydroid/xx.yy.146.217:24947 MULTI: primary virtual IP for mydroid/xx.yy.146.217:24947: 10.193.169.3
Mar 25 16:43:17 xenon ovpn-server[19982]: mydroid/xx.yy.146.217:24947 PUSH: Received control message: 'PUSH_REQUEST'
Mar 25 16:43:17 xenon ovpn-server[19982]: mydroid/xx.yy.146.217:24947 SENT CONTROL [mydroid]: 'PUSH_REPLY,redirect-gateway def1,dhcp-option DNS 10.192.168.1,route-gateway 10.193.169.1,topology subnet,ping 10,ping-restart 120,ifconfig 10.193.169.3 255.255.255.0,peer-id 0,cipher AES-256-GCM' (status=1)
Mar 25 16:43:17 xenon ovpn-server[19982]: mydroid/xx.yy.146.217:24947 Data Channel MTU parms [ L:1550 D:1450 EF:50 EB:406 ET:0 EL:3 ]
Mar 25 16:43:17 xenon ovpn-server[19982]: mydroid/xx.yy.146.217:24947 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Mar 25 16:43:17 xenon ovpn-server[19982]: mydroid/xx.yy.146.217:24947 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Mar 25 16:43:35 xenon ovpn-server[19982]: mydroid/xx.yy.146.217:24947 SIGTERM[soft,remote-exit] received, client-instance exiting

TinCanTech

Are you using your phone as some sort of hotspot ?

jan.u.thomsen@gmail.com

No. My phone is a simple OpenVPN client going through my home network.

I did a tcpdump on my router/firewall and copied into wireshark to see what the router believe is going on.
As it turns out the router see traffic in it's LAN interface eth1 from android phone 10.192.169.3 IP address trying to reach destination on the internet. Seems the traffic is some TCP retransmissions and TCP RST messages. But traffic from the android should go through the OpenVPN server interface which has a firewall masquerade rule that should mask the tun0 IP address ? Odd.
Same thing happens from time to time with the Debian client on a notebook PC.

Pippin

.....................

/me should learn to read English better 🙂

jan.u.thomsen@gmail.com

Did I write something which is not correct or understandable ?

Pippin

No, I not pay attention enough, my fault 😉

TinCanTech

jan.u.thomsen@gmail.com wrote the problem seems to be worse after a period of ideling where the OpenVPN connection has been shut down and automatically reconnect

This is the source of the problem .. sleep and wake cycles ..

jan.u.thomsen@gmail.com wrote the router see traffic in it's LAN interface eth1 from android phone 10.192.169.3 IP address trying to reach destination on the internet.

because openvpn has not fully restarted from sleep mode.

jan.u.thomsen@gmail.com wrote But traffic from the android should go through the OpenVPN server interface which has a firewall masquerade rule that should mask the tun0 IP address ? Odd.

because openvpn has not fully restarted from sleep mode.

jan.u.thomsen@gmail.com wrote Same thing happens from time to time with the Debian client on a notebook PC

because openvpn has not fully restarted from sleep mode.

Your phone and laptop are pausing the VPN for power saving .. at that time, packets which ought to be routed over the VPN only have on interface to exit the client and that is the LAN.

Your router is correct to warn you but there is no security leak with regard to openvpn.

While the VPN is fully functional this will not happen and Openvpn is secure ..
But Openvpn has always had some minor issues related to sleep and wake cycles, it is ubiquitous.
And it is also free .. so you get what you pay for.. :twisted:

jan.u.thomsen@gmail.com

Thanks a lot.

I was sort of leaning towards something along those lines after going through the process of describing the problem for this forum and doing the checks you asked. The important thing is that it does not pose a security risk since it was meant to increase privacy and security, so I will follow your recommendation and disregard the martian warnings.

Have a nice day.

TinCanTech

You put the time into documenting your problem so I take the time to help .. everybody wins 8-)

And you had an interesting problem which has even taught me a little new information .. win for me 😃

And as a bonus point, if you ever see those martians on your router when using openvpn and you are not on your home LAN then be sure to let us know because that would be very interesting. :ugeek:

Enjoy

jan.u.thomsen@gmail.com

TinCanTech wrote You put the time into documenting your problem so I take the time to help .. everybody wins 8-)

And you had an interesting problem which has even taught me a little new information .. win for me 😃

And as a bonus point, if you ever see those martians on your router when using openvpn and you are not on your home LAN then be sure to let us know because that would be very interesting. :ugeek:

Enjoy

But this was indeed the case ! During the testing I did in my posts today, my phone was not on my home Wifi network. It was using my phone connection. Let me know if you need any further information to troubleshoot this.

TinCanTech

You will need to carry out more thorough testing..

So that you can reliably reproduce the problem and document it clearly.

jan.u.thomsen@gmail.com

If you could be more specific on what you would need, then I will do the necessary testing and documentation.

TinCanTech

Let me put this another way:

TinCanTech wrote And as a bonus point, if you ever see those martians on your router when using openvpn and you are not on your home LAN then be sure to let us know because that would be very interesting. :ugeek:

jan.u.thomsen@gmail.com wrote But this was indeed the case ! During the testing I did in my posts today, my phone was not on my home Wifi network. It was using my phone connection. Let me know if you need any further information to troubleshoot this.

You will most likely find that your testing was not done correctly/thoroughly.

If you were to find martian packets on you home LAN while you were out at F.E. a local library, while your server is masquerading your VPN traffic then you are claiming that iptables masquerade is failing ..

That would not be an openvpn bug but would be quite interesting to the iptables developers and probably the maintainers of the distro that you are using for your server.

If, after exhausting all known avenues with that trail, you can reliably reproduce the problem above then please let us know ..

jan.u.thomsen@gmail.com wrote If you could be more specific on what you would need, then I will do the necessary testing and documentation.

The problem is clear: Reliable reproduction ..

The burden of proof is now upon you.

Pippin

Having doubts...

You could post output of:

iptables -t nat -S

Was your WiFi turned off on Android?
Puzzling also, the Debian host...

jan.u.thomsen@gmail.com

output of "iptables -t nat -S"
-----
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
-A POSTROUTING -s 10.193.169.0/24 -o br0 -j MASQUERADE
----

Last rule is the important one here. As you can see I use a bridged interface due to use of libvirt for QEMU/Kvm virtual machine usage, and the other rules are created automatically by some libvirt configuration setting.

configuration of br0 bridged interface in /etc/network/interfaces is the following:
---
auto br0
iface br0 inet dhcp
bridge_ports p2p1
bridge_stp off
bridge_maxwait 0
---
where p2p1 is the ethernet interface, which is excluded from handling from Network Manager, to keep it free for the bridge interface.

It looks to me as if the iptables MASQUERADE rule listed above somehow is not executed to change the source address 10.193.169.3 to the IP address of the br0 interface.

I used wireshark to capture all traffic on br0 originating from 10.192.169.3 (android tun0 interface) and when I get martian packet error messages from my router/firewall there is also a flurry of captured packets that correclate with the martians. The packets seems all to be TCP RST or FIN,ACK and retransmissions of those. I tried to attach an image using the menu here but all I get is some link tags, so I dont know how fix that.

With regards to reproducibility - as was requested in earliger message - it is quite reproducible. I just have to close the display of my android phone, which cause openvpn client to go into sleep mode. Wait a couple of minutes and open it again and do some browsing. This will inevitably result in some martian messages in my router, and packets captured in wireshark on outgoing br0 interface of my openvpn server with source address 10.193.169.3 (android tun0).

I cant figure out how the traffic from the android somehow eludes the MASQUERADE rule - and only after reconnecting OpenVPN connection. Hmmm.

jan.u.thomsen@gmail.com

I have concluded that the problem is somehow related to the linux firewall not masquerading all TCP packets coming from the android phone tun0 interface 10.193.169.3 when the OpenVPN connection is re-established after the phone is activated. I think it may be related to the logic of the Linux firewall possibly considering the TCP connections from before the OpenVPN disconnect being dead and therefore decide not to do the MASQUERADE change of the source IP address to that of the OpenVPN servers br0 interfaces (could it maybe be caused by connection tracking entries for the stale/dead TCP connections being deleted or invalidated ?).

In any case, after I added the following rules to iptables, the martian messages from my router have disappeared.

-A FORWARD -s 10.193.169.3/32 -o br0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG RST,ACK -j DROP
-A FORWARD -s 10.193.169.3/32 -o br0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,ACK -j DROP
-A FORWARD -s 10.193.169.3/32 -o br0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG RST -j DROP

The rules were selected based on wireshark output from capture of traffic from 10.193.169.3 on OpenVPN servers br0 interface which showed the RST,ACK FIN,ACK and FIN messages being the ones appearing when martian packets occurred. After introducing these rules I have not seen any further martians in my router logs.

I also tried the following rule, and it seems to work too:

-A FORWARD -s 10.193.169.3 -o br0 -p tcp -m state --state INVALID -j DROP

Invalid state means the tcp connection is neither NEW, ESTABLISHED or RELATED, and it makes sense to me at least, that some of the TCP connections established prior to the OpenVPN connection going dormant, could end up in an invalid state It is puzzling though why iptables choose to route invalid packets to br0 without applying the source NAT MASQUERADE rule. Maybe this is expected iptables behaviour if one truly understand the inner logic of the linux firewall.

If anyone has a good explanation please let me know.

For now above workaround seems to work fine, but it may be just totally unnecessary :-) since it seems the martian packets are only related to closing broken TCP connections hanging around from before OpenVPN connection going dormant, and most likely have absolutely no bearing on security. And in any case the martians are dropped by the router/firewall, so there is probably no ill effects of dropping the packets (even if it is not all of them that result in martians)

Jan

Pippin

Sorry, no relevant experience with VM and bridging.
But you added more info which should be added from the start.....😉

TinCanTech

jan.u.thomsen@gmail.com wrote As you can see I use a bridged interface due to use of libvirt for QEMU/Kvm virtual machine usage

Until you work out what is happening, I would remove this bridge.

jan.u.thomsen@gmail.com wrote I have concluded that the problem is somehow related to the linux firewall not masquerading all TCP packets coming from the android phone tun0 interface 10.193.169.3 when the OpenVPN connection is re-established after the phone is activated

I think you have misunderstood the problem .. what is much more likely is that the phone is sending packets to your LAN via it's local connection before the VPN has reconnected.

If iptables is misbehaving then there is a serious issue with iptables which has been missed by the iptables developers and users and the maintainers of iptables for Ubuntu .. this is extremely unlikely ..

I am not saying you have done this wrong but I don't believe iptables is doing what you claim.

Hint: Run tcpdump on your server.