Slow upload caused by iptables port redirection.

vpnjumper

Hello,

I'm using OpenVPN 2.4.5 Server and currently experimenting with the following setup. Basically need to expose the same openvpn server on multiple ports:

Client -> (uses "remote Server:443" in ovpn config)
Server:443 -> (iptables redirection to :1194)
Server:1194 (OpenVPN Server)

As iptables Rule I use this (tried also nginx udp reverse proxy, socat... same result)

iptables -t nat -A PREROUTING -p udp --dport 443 -j REDIRECT --to-ports 1194

Connecting works fine, and download speed is unchanged. But when I run an upload speed test, speed is limited to about 4.5mbit/s. This does not happen if I connect to 1194 directly (then have 50mbit+). In server log I also get this when running upload speed test:

PID_ERR large diff [601] [SSL-0] [0000000000000000000000000000000000000000000000000000000000000000] 0:48861 0:48260 t=1625683565[0] r=[-4,64,15,982,1] sl=[41,64,64,528]
AEAD Decrypt error: bad packet ID (may be a replay): [ #48260 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
PID_ERR large diff [600] [SSL-0] [0000000000000000000000000000000000000000000000000000000000000000] 0:48861 0:48261 t=1625683565[0] r=[-4,64,15,982,1] sl=[41,64,64,528]
AEAD Decrypt error: bad packet ID (may be a replay): [ #48261 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

This seems very strange, I also verified with tcpdump that the ovpn server gets all the packets. What is the reason for this?

openvpn_inc

vpnjumper wrote I'm using OpenVPN 2.4.5 Server and currently experimenting with the following setup. Basically need to expose the same openvpn server on multiple ports:

Client -> (uses "remote Server:443" in ovpn config)
Server:443 -> (iptables redirection to :1194)
Server:1194 (OpenVPN Server)

Why? Our OpenVPN Access Server (proprietary software) product uses udp/1194 and tcp/443 by default (and in that order of preference, because OpenVPN performs best over UDP.) But I don't see much point in udp/443.

The reason for having tcp/443 is to get around certain "web-only" firewalls. Such firewalls are unlikely to allow any UDP ports.

vpnjumper wrote As iptables Rule I use this (tried also nginx udp reverse proxy, socat... same result)

iptables -t nat -A PREROUTING -p udp --dport 443 -j REDIRECT --to-ports 1194

PID_ERR large diff [601] [SSL-0] [0000000000000000000000000000000000000000000000000000000000000000] 0:48861 0:48260 t=1625683565[0] r=[-4,64,15,982,1] sl=[41,64,64,528]
AEAD Decrypt error: bad packet ID (may be a replay): [ #48260 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
PID_ERR large diff [600] [SSL-0] [0000000000000000000000000000000000000000000000000000000000000000] 0:48861 0:48261 t=1625683565[0] r=[-4,64,15,982,1] sl=[41,64,64,528]
AEAD Decrypt error: bad packet ID (may be a replay): [ #48261 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

This seems very strange, I also verified with tcpdump that the ovpn server gets all the packets. What is the reason for this?

Yes, it does seem strange. Best I can guess is that you're getting some replay packets, but I have no guess why udp/443 performance would differ from 1194. Show your work? How did you test this? Are you going through an asymmetric link? It's not uncommon for upload bandwidth to be a fraction of download.

Mystified, --rob0

P.S. I had this all written and then had an idea. 🙂 NAT rules generally need to be limited, yours is not. Your REDIRECT should only match packets arriving on the external interface. Supposing your external interface is eth0, try:

iptables -t nat -D PREROUTING -p udp --dport 443 -j REDIRECT --to-ports 1194
iptables -t nat -A PREROUTING -i eth0 -p udp --dport 443 -j REDIRECT --to-ports 1194

The first rule deletes your unrestricted NAT rule; the second adds one that's properly limited.

P.P.S. Don't use iptables like this. Use iptables-restore(8) to do a complete ruleset in one atomic operation. Your distro probably does this already.

vpnjumper

openvpn_inc wrote Why? Our OpenVPN Access Server (proprietary software) product uses udp/1194 and tcp/443 by default (and in that order of preference, because OpenVPN performs best over UDP.) But I don't see much point in udp/443.

QUIC or HTTP/3 seems to be using UDP/443. Allowing Port 443 Traffic regardless of protocol seems quite common in my experience.
Haven't looked at Access Server yet, thanks, might give it a try!

openvpn_inc wrote Yes, it does seem strange. Best I can guess is that you're getting some replay packets, but I have no guess why udp/443 performance would differ from 1194. Show your work? How did you test this? Are you going through an asymmetric link? It's not uncommon for upload bandwidth to be a fraction of download.

Nothing special, normal openvpn community server.

openvpn_inc wrote P.S. I had this all written and then had an idea. 🙂 NAT rules generally need to be limited, yours is not. Your REDIRECT should only match packets arriving on the external interface. Supposing your external interface is eth0, try:
iptables -t nat -D PREROUTING -p udp --dport 443 -j REDIRECT --to-ports 1194
iptables -t nat -A PREROUTING -i eth0 -p udp --dport 443 -j REDIRECT --to-ports 1194
The first rule deletes your unrestricted NAT rule; the second adds one that's properly limited.

Thanks, that solved it! 😃 😃
There was indeed an amplification happening, after resolving that upload speed is fine!

openvpn_inc

Great! We learned something! 🙂 --rob0

toptopweb

But on whether what she pursues is what she likes.
We evaluate the success of any person not on material conditions，and what she gives up is what prevents her from realizing her dream.
I very much agree with this sentence, life and work, we choose to work because of life, and work makes us neglect life.
When you want to get promoted, you have to sacrifice more of your private time to work.
voip phone system service provider
A very interesting website