Inbound traffic dropping

Need help configuring your VPN? Just post here and you'll get that help.
Forum rules
Please use the [oconf] BB tag for openvpn Configurations. See viewtopic.php?f=30&t=21589 for an example.
Post Reply
vpnforward
OpenVpn Newbie
Posts: 7
Joined: Thu Aug 04, 2016 10:34 pm

Inbound traffic dropping

Post by vpnforward » Thu Aug 04, 2016 10:53 pm

I'm running into a very strange behavior and am hoping one of you may know what's going on. Of course, all the terms I wish to search on are "too common" so I can't tell if this has been asked before.

I have users connect to a pfSense system "in the cloud" running OpenVPN 2.3.11 and works fine - authentication is fine, traffic passes, routes propagate, all is good. Consider it a tunnel broker of sorts, it's for a security testing team. I'm using TAP rather than TUN right now to simplify troubleshooting but both have almost worked.

Where things break down is that I want to open ports to the VPN clients for various reasons (mostly penetration testing callbacks). This, too, works fine - I can forward arbitrary TCP and UDP ports to the VPN clients without a problem. Packet captures show, say, a SYN arriving at the VPN client's tap0 interface, all addressed correctly, but they are never delivered to the listening client (netcat, in this case).

That is, say I have a VPN subnet of 10.0.0.0/24. Client #1 connects and gets assigned 10.0.0.2. Client #1 sets up a port-forward for TCP port 50000, and sets up a netcat listener on that port (nc -vv -l -p 50000). "Somewhere else" on the interbutts, someone runs a netcat against that port (nc publicip.somehost.con 50000). A packet capture on client #1 sees SYNs arriving on tap0, but they are never delivered to the listening netcat. Another packet capture on client #1's main interface confirms that we're not [yet] seeing asymmetric routing.

Where this gets even more weird is that client-to-client works. In the above scenario, client #2 connects via VPN and gets assigned 10.0.0.3. If they run "netcat 10.0.0.2 50000" the connection works fine.

What all this suggests to me is that the openvpn client-side stack appears to be dropping inbound SYNs for which it does not yet have state, rather than acting like a simple transport. Has anyone else seen this, or can anyone confirm it? I've tested this across Windows, OS X, and Linux clients with varying levels of privilege, and the behavior is consistent on all platforms.

User avatar
TinCanTech
Forum Team
Posts: 10826
Joined: Fri Jun 03, 2016 1:17 pm

Re: Inbound traffic dropping

Post by TinCanTech » Thu Aug 04, 2016 11:10 pm

vpnforward wrote:all the terms I wish to search on are "too common" so
So:

On duckduckgo or google type

Code: Select all

site:forums.openvpn.net your search terms
Note: no space after site:

vpnforward
OpenVpn Newbie
Posts: 7
Joined: Thu Aug 04, 2016 10:34 pm

Re: Inbound traffic dropping

Post by vpnforward » Fri Aug 05, 2016 1:58 pm

TinCanTech wrote:So:

On duckduckgo or google type

Code: Select all

site:forums.openvpn.net your search terms
Note: no space after site:
Although the LMGTFY response can be gratifying, I wish it were more helpful. You see, I'd already done that and found nothing; some threads' search results suggested they might be something similar, but are apparently from an old indexing of an old version of the board, and no longer exist. I had hoped the internal search might provide better results.

Beyond search guidance, do you have any insight to the actual problem here?

User avatar
TinCanTech
Forum Team
Posts: 10826
Joined: Fri Jun 03, 2016 1:17 pm

Re: Inbound traffic dropping

Post by TinCanTech » Fri Aug 05, 2016 2:12 pm

duckduckgo works for me .. perhaps they finally resolved it ..
vpnforward wrote:do you have any insight to the actual problem here?
Reading your initial post but not having your current config, I will assume you have not read this:

vpnforward
OpenVpn Newbie
Posts: 7
Joined: Thu Aug 04, 2016 10:34 pm

Re: Inbound traffic dropping

Post by vpnforward » Fri Aug 05, 2016 4:14 pm

Speaking from the standpoint of a long time code-level OSS contributor and support provider, your meta-responses, while perhaps with good intentions, detract from the otherwise flawless quality of this project. Simply request my configuration if, after reading the post, you think it's relevant; don't be sarcastic about it. We can all afford to be polite.

I did not post my configuration (indeed, I attempted to post to "Server Administration" and was shunted to the configuration sub-forum) because it's not a configuration issue, per se. OpenVPN works as configured, but packets are transparently dropped immediately prior to being delivered to a listening application. The search from duckduckgo works fine, but returns many old results that, while potentially relevant, are no longer available on this forum (e.g., https://forums.openvpn.net/topic13555.html). Knowing the phpBB path patterns, it's obvious that particular URL is from an older version (if not a different BB software altogether).

I also have no issue pushing routes, but this isn't a question of that. If it were a routing issue, the listening application would at least receive the packet and send it out the wrong interface. That's not happening, and is what I explained above.

In interest of hopefully moving forward, here's a sample configuration.
server
dev ovpns1
verb 1
dev-type tap
dev-node /dev/tap1
writepid /var/run/openvpn_server1.pid
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-256-CFB
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
client-connect /usr/local/sbin/openvpn.attributes.sh
client-disconnect /usr/local/sbin/openvpn.attributes.sh
local 1.1.1.1
tls-server
server 10.0.0.0 255.255.255.0
client-config-dir /var/etc/openvpn-csc/server1
username-as-common-name
auth-user-pass-verify "/usr/local/sbin/ovpn_auth_verify user 'Local Database'
true server1" via-env
tls-verify "/usr/local/sbin/ovpn_auth_verify tls '1.1.1.1' 1"
lport 1194
management /var/etc/openvpn/server1.sock unix
push "route 2.2.2.0 255.255.255.0"
push "route 10.1.0.0 255.255.0.0"
client-to-client
duplicate-cn
ca /var/etc/openvpn/server1.ca
cert /var/etc/openvpn/server1.cert
key /var/etc/openvpn/server1.key
dh /etc/dh-parameters.2048
crl-verify /var/etc/openvpn/server1.crl-verify
tls-auth /var/etc/openvpn/server1.tls-auth 0
client
dev tap
persist-tun
persist-key
cipher AES-256-CFB
auth SHA256
tls-client
client
resolv-retry infinite
remote 1.1.1.1 1194 udp
lport 0
verify-x509-name "1.1.1.1" name
auth-user-pass
ns-cert-type server
key-direction 1
edit: oconf= wants a name, adding client config

User avatar
TinCanTech
Forum Team
Posts: 10826
Joined: Fri Jun 03, 2016 1:17 pm

Re: Inbound traffic dropping

Post by TinCanTech » Fri Aug 05, 2016 7:27 pm

vpnforward wrote:Speaking from the standpoint of a long time code-level OSS contributor and support provider, your meta-responses, while perhaps with good intentions, detract from the otherwise flawless quality of this project. Simply request my configuration if, after reading the post, you think it's relevant; don't be sarcastic about it. We can all afford to be polite.
If this were true .. you would understand why your configs were required in the first place and so by not posting them you have simply fallen at the very first hurdle. The second being your log files .. leaving me no choice but to hazard a guess ..

I am not prepared to discuss that issue any further.
vpnforward wrote:the openvpn client-side stack appears to be dropping inbound SYNs
I have never seen Openvpn drop packets on the client side. On the server, using --verb 4, Openvpn will log, with a reason, all packets which are dropped ..

vpnforward
OpenVpn Newbie
Posts: 7
Joined: Thu Aug 04, 2016 10:34 pm

Re: Inbound traffic dropping

Post by vpnforward » Fri Aug 05, 2016 7:50 pm

I have outlined precisely the information you need in the first post; insisting on form over content and continued sarcasm suggest you may not have grasped what I'm asking. Being civil is a critical component of supporting others in the OSS world; if you cannot be so, you poorly represent the project you've attached yourself to.

I'm well beyond debugging configurations or logs, since if there were a visible issue in those I wouldn't be here. I've tracked (and perfectly understand) the logs at --verb 9 on both sides and neither is reporting any error. I'm actually tracing the code right now and was hoping to find someone here sufficiently conversant with the inner operations of pf.c, route.c, and tun.c to be able to speak intelligently to what may be a bug. If you are not that person, I welcome you to bow out of the conversation.
TinCanTech wrote:I have never seen Openvpn drop packets on the client side. On the server, using --verb 4, Openvpn will log, with a reason, all packets which are dropped ..
They're not dropping on the server side. A tcpdump on the receiving client (as well as on the server) sees unrequited SYNs from the external IP to the client on the correct port. Those SYNs are never delivered to the listening application (netcat), even though they are arriving at the client interface (tap0, in this configuration case).

User avatar
TinCanTech
Forum Team
Posts: 10826
Joined: Fri Jun 03, 2016 1:17 pm

Re: Inbound traffic dropping

Post by TinCanTech » Fri Aug 05, 2016 8:07 pm

vpnforward wrote:I have outlined precisely the information you need
No you have not ..
vpnforward wrote:you poorly represent the project you've attached yourself to
Stick around and read the forum for a while and judge for yourself .. rather than jumping to your ill-informed conclusion.

If you feel you can do better, answer some posts for others or even apply for a moderator position yourself.
vpnforward wrote:hoping to find someone here sufficiently conversant with the inner operations of pf.c, route.c, and tun.c to be able to speak intelligently to what may be a bug
Raise a bug report in the appropriate place.
vpnforward wrote:They're not dropping on the server side
Read what I said again ..
vpnforward wrote: A tcpdump on the receiving client (as well as on the server) sees unrequited SYNs from the external IP to the client on the correct port. Those SYNs are never delivered to the listening application (netcat), even though they are arriving at the client interface (tap0
If tcpdump sees the unencrypted packet on the tun/tap interface then the packet has been delivered by openvpn .: It is no longer within the OpenVPN code stream ..
vpnforward wrote: welcome you to bow out of the conversation
Bowing out .. good luck.

vpnforward
OpenVpn Newbie
Posts: 7
Joined: Thu Aug 04, 2016 10:34 pm

Re: Inbound traffic dropping

Post by vpnforward » Fri Aug 05, 2016 8:14 pm

Thanks! Hopefully we can move forward here, or at least perhaps the mailing list will provide fruit.

vpnforward
OpenVpn Newbie
Posts: 7
Joined: Thu Aug 04, 2016 10:34 pm

Re: Inbound traffic dropping

Post by vpnforward » Mon Aug 08, 2016 4:45 pm

Thanks to Gert Doering and Selva Nair, the issue was uRPF. Essentially, by default the three main OSes will silently drop packets inbound on an interface that has no return route for the packet. They do so silently after a packet enters an interface, but before it arrives at a listening application, in the routing decision. This was not an OpenVPN bug, but its configuration flexibility allowed me to run into a routing edge case more quickly than without.

Linux is the most readily changed to diverge from that default drop behavior (/proc/sys/net/ipv4/conf/.../rp_filter), and by altering those settings appropriately I was able to see the expected asymmetric routing, which itself was a simpler problem to solve. The simplest answer is to advertise (and use) a default route over the tunnel if you wish to forward arbitrary ports from arbitrary clients.

MilitantDK
OpenVpn Newbie
Posts: 1
Joined: Mon Dec 27, 2021 7:59 pm

Re: Inbound traffic dropping

Post by MilitantDK » Tue Dec 28, 2021 8:23 am

Hi @vpnforward,

What was the appropriate settings you changes?

Sincerely
Max

Post Reply