Inbound traffic dropping

vpnforward

I'm running into a very strange behavior and am hoping one of you may know what's going on. Of course, all the terms I wish to search on are "too common" so I can't tell if this has been asked before.

I have users connect to a pfSense system "in the cloud" running OpenVPN 2.3.11 and works fine - authentication is fine, traffic passes, routes propagate, all is good. Consider it a tunnel broker of sorts, it's for a security testing team. I'm using TAP rather than TUN right now to simplify troubleshooting but both have almost worked.

Where things break down is that I want to open ports to the VPN clients for various reasons (mostly penetration testing callbacks). This, too, works fine - I can forward arbitrary TCP and UDP ports to the VPN clients without a problem. Packet captures show, say, a SYN arriving at the VPN client's tap0 interface, all addressed correctly, but they are never delivered to the listening client (netcat, in this case).

That is, say I have a VPN subnet of 10.0.0.0/24. Client #1 connects and gets assigned 10.0.0.2. Client #1 sets up a port-forward for TCP port 50000, and sets up a netcat listener on that port (nc -vv -l -p 50000). "Somewhere else" on the interbutts, someone runs a netcat against that port (nc publicip.somehost.con 50000). A packet capture on client #1 sees SYNs arriving on tap0, but they are never delivered to the listening netcat. Another packet capture on client #1's main interface confirms that we're not [yet] seeing asymmetric routing.

Where this gets even more weird is that client-to-client works. In the above scenario, client #2 connects via VPN and gets assigned 10.0.0.3. If they run "netcat 10.0.0.2 50000" the connection works fine.

What all this suggests to me is that the openvpn client-side stack appears to be dropping inbound SYNs for which it does not yet have state, rather than acting like a simple transport. Has anyone else seen this, or can anyone confirm it? I've tested this across Windows, OS X, and Linux clients with varying levels of privilege, and the behavior is consistent on all platforms.

TinCanTech

vpnforward wroteall the terms I wish to search on are "too common" so

So:

On duckduckgo or google type

site:forums.openvpn.net your search terms

Note: no space after site:

vpnforward

TinCanTech wroteSo:

On duckduckgo or google type

site:forums.openvpn.net your search terms

Note: no space after site:

Although the LMGTFY response can be gratifying, I wish it were more helpful. You see, I'd already done that and found nothing; some threads' search results suggested they might be something similar, but are apparently from an old indexing of an old version of the board, and no longer exist. I had hoped the internal search might provide better results.

Beyond search guidance, do you have any insight to the actual problem here?

TinCanTech

duckduckgo works for me .. perhaps they finally resolved it ..

vpnforward wrotedo you have any insight to the actual problem here?

Reading your initial post but not having your current config, I will assume you have not read this:

HOWTO: For OpenVPN Community Edition
This is specific to your problem:
HOWTO: Expanding the scope of the VPN to include additional machines

vpnforward

Speaking from the standpoint of a long time code-level OSS contributor and support provider, your meta-responses, while perhaps with good intentions, detract from the otherwise flawless quality of this project. Simply request my configuration if, after reading the post, you think it's relevant; don't be sarcastic about it. We can all afford to be polite.

I did not post my configuration (indeed, I attempted to post to "Server Administration" and was shunted to the configuration sub-forum) because it's not a configuration issue, per se. OpenVPN works as configured, but packets are transparently dropped immediately prior to being delivered to a listening application. The search from duckduckgo works fine, but returns many old results that, while potentially relevant, are no longer available on this forum (e.g., https://forums.openvpn.net/topic13555.html). Knowing the phpBB path patterns, it's obvious that particular URL is from an older version (if not a different BB software altogether).

I also have no issue pushing routes, but this isn't a question of that. If it were a routing issue, the listening application would at least receive the packet and send it out the wrong interface. That's not happening, and is what I explained above.

In interest of hopefully moving forward, here's a sample configuration.
dev ovpns1
verb 1
dev-type tap
dev-node /dev/tap1
writepid /var/run/openvpn_server1.pid
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-256-CFB
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
client-connect /usr/local/sbin/openvpn.attributes.sh
client-disconnect /usr/local/sbin/openvpn.attributes.sh
local 1.1.1.1
tls-server
server 10.0.0.0 255.255.255.0
client-config-dir /var/etc/openvpn-csc/server1
username-as-common-name
auth-user-pass-verify "/usr/local/sbin/ovpn_auth_verify user 'Local Database'
true server1" via-env
tls-verify "/usr/local/sbin/ovpn_auth_verify tls '1.1.1.1' 1"
lport 1194
management /var/etc/openvpn/server1.sock unix
push "route 2.2.2.0 255.255.255.0"
push "route 10.1.0.0 255.255.0.0"
client-to-client
duplicate-cn
ca /var/etc/openvpn/server1.ca
cert /var/etc/openvpn/server1.cert
key /var/etc/openvpn/server1.key
dh /etc/dh-parameters.2048
crl-verify /var/etc/openvpn/server1.crl-verify
tls-auth /var/etc/openvpn/server1.tls-auth 0 dev tap
persist-tun
persist-key
cipher AES-256-CFB
auth SHA256
tls-client
client
resolv-retry infinite
remote 1.1.1.1 1194 udp
lport 0
verify-x509-name "1.1.1.1" name
auth-user-pass
ns-cert-type server
key-direction 1 edit: oconf= wants a name, adding client config

TinCanTech

vpnforward wroteSpeaking from the standpoint of a long time code-level OSS contributor and support provider, your meta-responses, while perhaps with good intentions, detract from the otherwise flawless quality of this project. Simply request my configuration if, after reading the post, you think it's relevant; don't be sarcastic about it. We can all afford to be polite.

If this were true .. you would understand why your configs were required in the first place and so by not posting them you have simply fallen at the very first hurdle. The second being your log files .. leaving me no choice but to hazard a guess ..

I am not prepared to discuss that issue any further.

vpnforward wrotethe openvpn client-side stack appears to be dropping inbound SYNs

I have never seen Openvpn drop packets on the client side. On the server, using --verb 4, Openvpn will log, with a reason, all packets which are dropped ..

vpnforward

I have outlined precisely the information you need in the first post; insisting on form over content and continued sarcasm suggest you may not have grasped what I'm asking. Being civil is a critical component of supporting others in the OSS world; if you cannot be so, you poorly represent the project you've attached yourself to.

I'm well beyond debugging configurations or logs, since if there were a visible issue in those I wouldn't be here. I've tracked (and perfectly understand) the logs at --verb 9 on both sides and neither is reporting any error. I'm actually tracing the code right now and was hoping to find someone here sufficiently conversant with the inner operations of pf.c, route.c, and tun.c to be able to speak intelligently to what may be a bug. If you are not that person, I welcome you to bow out of the conversation.

TinCanTech wroteI have never seen Openvpn drop packets on the client side. On the server, using --verb 4, Openvpn will log, with a reason, all packets which are dropped ..

They're not dropping on the server side. A tcpdump on the receiving client (as well as on the server) sees unrequited SYNs from the external IP to the client on the correct port. Those SYNs are never delivered to the listening application (netcat), even though they are arriving at the client interface (tap0, in this configuration case).

TinCanTech

vpnforward wroteI have outlined precisely the information you need

No you have not ..

vpnforward wroteyou poorly represent the project you've attached yourself to

Stick around and read the forum for a while and judge for yourself .. rather than jumping to your ill-informed conclusion.

If you feel you can do better, answer some posts for others or even apply for a moderator position yourself.

vpnforward wrotehoping to find someone here sufficiently conversant with the inner operations of pf.c, route.c, and tun.c to be able to speak intelligently to what may be a bug

Raise a bug report in the appropriate place.

vpnforward wroteThey're not dropping on the server side

Read what I said again ..

vpnforward wrote A tcpdump on the receiving client (as well as on the server) sees unrequited SYNs from the external IP to the client on the correct port. Those SYNs are never delivered to the listening application (netcat), even though they are arriving at the client interface (tap0

If tcpdump sees the unencrypted packet on the tun/tap interface then the packet has been delivered by openvpn .: It is no longer within the OpenVPN code stream ..

vpnforward wrote welcome you to bow out of the conversation

Bowing out .. good luck.

vpnforward

Thanks! Hopefully we can move forward here, or at least perhaps the mailing list will provide fruit.

vpnforward

Thanks to Gert Doering and Selva Nair, the issue was uRPF. Essentially, by default the three main OSes will silently drop packets inbound on an interface that has no return route for the packet. They do so silently after a packet enters an interface, but before it arrives at a listening application, in the routing decision. This was not an OpenVPN bug, but its configuration flexibility allowed me to run into a routing edge case more quickly than without.

Linux is the most readily changed to diverge from that default drop behavior (/proc/sys/net/ipv4/conf/.../rp_filter), and by altering those settings appropriately I was able to see the expected asymmetric routing, which itself was a simpler problem to solve. The simplest answer is to advertise (and use) a default route over the tunnel if you wish to forward arbitrary ports from arbitrary clients.

MilitantDK

Hi @vpnforward,

What was the appropriate settings you changes?

Sincerely
Max