Page 1 of 1

Slow upload caused by iptables port redirection.

Posted: Sat Jul 10, 2021 1:05 pm
by vpnjumper
Hello,

I'm using OpenVPN 2.4.5 Server and currently experimenting with the following setup. Basically need to expose the same openvpn server on multiple ports:

Code: Select all

Client -> (uses "remote Server:443" in ovpn config)
Server:443 -> (iptables redirection to :1194)
Server:1194 (OpenVPN Server)
As iptables Rule I use this (tried also nginx udp reverse proxy, socat... same result)

Code: Select all

iptables -t nat -A PREROUTING -p udp --dport 443 -j REDIRECT --to-ports 1194
Connecting works fine, and download speed is unchanged. But when I run an upload speed test, speed is limited to about 4.5mbit/s. This does not happen if I connect to 1194 directly (then have 50mbit+). In server log I also get this when running upload speed test:

Code: Select all

PID_ERR large diff [601] [SSL-0] [0000000000000000000000000000000000000000000000000000000000000000] 0:48861 0:48260 t=1625683565[0] r=[-4,64,15,982,1] sl=[41,64,64,528]
AEAD Decrypt error: bad packet ID (may be a replay): [ #48260 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
PID_ERR large diff [600] [SSL-0] [0000000000000000000000000000000000000000000000000000000000000000] 0:48861 0:48261 t=1625683565[0] r=[-4,64,15,982,1] sl=[41,64,64,528]
AEAD Decrypt error: bad packet ID (may be a replay): [ #48261 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
This seems very strange, I also verified with tcpdump that the ovpn server gets all the packets. What is the reason for this?

Re: Slow upload caused by iptables port redirection.

Posted: Sat Jul 10, 2021 2:14 pm
by openvpn_inc
vpnjumper wrote:
Sat Jul 10, 2021 1:05 pm
I'm using OpenVPN 2.4.5 Server and currently experimenting with the following setup. Basically need to expose the same openvpn server on multiple ports:

Code: Select all

Client -> (uses "remote Server:443" in ovpn config)
Server:443 -> (iptables redirection to :1194)
Server:1194 (OpenVPN Server)
Why? Our OpenVPN Access Server (proprietary software) product uses udp/1194 and tcp/443 by default (and in that order of preference, because OpenVPN performs best over UDP.) But I don't see much point in udp/443.

The reason for having tcp/443 is to get around certain "web-only" firewalls. Such firewalls are unlikely to allow any UDP ports.
vpnjumper wrote:
Sat Jul 10, 2021 1:05 pm
As iptables Rule I use this (tried also nginx udp reverse proxy, socat... same result)

Code: Select all

iptables -t nat -A PREROUTING -p udp --dport 443 -j REDIRECT --to-ports 1194
Connecting works fine, and download speed is unchanged. But when I run an upload speed test, speed is limited to about 4.5mbit/s. This does not happen if I connect to 1194 directly (then have 50mbit+). In server log I also get this when running upload speed test:

Code: Select all

PID_ERR large diff [601] [SSL-0] [0000000000000000000000000000000000000000000000000000000000000000] 0:48861 0:48260 t=1625683565[0] r=[-4,64,15,982,1] sl=[41,64,64,528]
AEAD Decrypt error: bad packet ID (may be a replay): [ #48260 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
PID_ERR large diff [600] [SSL-0] [0000000000000000000000000000000000000000000000000000000000000000] 0:48861 0:48261 t=1625683565[0] r=[-4,64,15,982,1] sl=[41,64,64,528]
AEAD Decrypt error: bad packet ID (may be a replay): [ #48261 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
This seems very strange, I also verified with tcpdump that the ovpn server gets all the packets. What is the reason for this?
Yes, it does seem strange. Best I can guess is that you're getting some replay packets, but I have no guess why udp/443 performance would differ from 1194. Show your work? How did you test this? Are you going through an asymmetric link? It's not uncommon for upload bandwidth to be a fraction of download.

Mystified, --rob0

P.S. I had this all written and then had an idea. :) NAT rules generally need to be limited, yours is not. Your REDIRECT should only match packets arriving on the external interface. Supposing your external interface is eth0, try:

Code: Select all

iptables -t nat -D PREROUTING -p udp --dport 443 -j REDIRECT --to-ports 1194
iptables -t nat -A PREROUTING -i eth0 -p udp --dport 443 -j REDIRECT --to-ports 1194
The first rule deletes your unrestricted NAT rule; the second adds one that's properly limited.

P.P.S. Don't use iptables like this. Use iptables-restore(8) to do a complete ruleset in one atomic operation. Your distro probably does this already.

Re: Slow upload caused by iptables port redirection.

Posted: Sat Jul 10, 2021 3:22 pm
by vpnjumper
openvpn_inc wrote:
Sat Jul 10, 2021 2:14 pm
Why? Our OpenVPN Access Server (proprietary software) product uses udp/1194 and tcp/443 by default (and in that order of preference, because OpenVPN performs best over UDP.) But I don't see much point in udp/443.
QUIC or HTTP/3 seems to be using UDP/443. Allowing Port 443 Traffic regardless of protocol seems quite common in my experience.
Haven't looked at Access Server yet, thanks, might give it a try!
openvpn_inc wrote:
Sat Jul 10, 2021 2:14 pm
Yes, it does seem strange. Best I can guess is that you're getting some replay packets, but I have no guess why udp/443 performance would differ from 1194. Show your work? How did you test this? Are you going through an asymmetric link? It's not uncommon for upload bandwidth to be a fraction of download.
Nothing special, normal openvpn community server.
openvpn_inc wrote:
Sat Jul 10, 2021 2:14 pm
P.S. I had this all written and then had an idea. :) NAT rules generally need to be limited, yours is not. Your REDIRECT should only match packets arriving on the external interface. Supposing your external interface is eth0, try:

Code: Select all

iptables -t nat -D PREROUTING -p udp --dport 443 -j REDIRECT --to-ports 1194
iptables -t nat -A PREROUTING -i eth0 -p udp --dport 443 -j REDIRECT --to-ports 1194
The first rule deletes your unrestricted NAT rule; the second adds one that's properly limited.
Thanks, that solved it! :D :D
There was indeed an amplification happening, after resolving that upload speed is fine!

Re: Slow upload caused by iptables port redirection.

Posted: Sat Jul 10, 2021 4:27 pm
by openvpn_inc
Great! We learned something! :) --rob0