Possible MTU issue on PPPoE link

rodtre

Hi,

I'm using OpenVPN for years and now I'm facing a problem that I can't figure out how to fix.

I have 4 Linux Fedora (14/15) servers, one is a OpenVPN server and 3 are clients.
All 4 are also the Internet gateway on each location, and I have a LAN behind them with NAT (iptables).
The setup is OpenVPN default.

Everything is working fine using RDP, SMB, NFS, SSH, HTTP, SFTP through the tunnel.

But I have a device connected to each OpenVPN client LAN, it's not a computer, it's a
data acquisiton device with a ethernet port and a IP stack. We use a Windows application
on the OpenVPN server LAN to manage, send and receive data from this device.

We can ping all 3 devices, and some operations work fine, like adjusting the device time and date.
However, when we make an operation that sends larger data, as a database table, the communication fails.
The apllication give me an error code meaning some network error.

Using this application on the same LAN as the device, everything works fine, the fail only happens via OpenVPN.
But it's not with all 3 devices, sometimes one or other stop working, the other 2 keep working.

Using iptraf and tcpdump we can see that actually all the data is really small, about 500B to 8KB,
all in TCP sessions. The device listens on TCP/51000. As far as we can see the data is binary.

In all locations, we have 2 Internet links, one in operation and one for
spare: Server: 2 dedicated FO links with MTU 1500. Clients:
A-Cable (MTU 1500) and FO with PPPoE (MTU 1492)
B-ADSL (MTU 1500) and radio with PPPoE (MTU 1480)
C-ADSL (MTU 1500) and radio (MTU 1500)

We are using no mtu-related parameters on server or clients, the MTU tunnels are default, 1500.

The question is, should I set [tun-mtu] for 1492 or less, or perhaps use [fragment] with [mssfix]?

We are not concerned about performance, we just need to get this application exchanging data
with those devices. Setting a very low MTU would be helpful?

I don't even know if is a MTU-related problem.

Thank you.

Mimiko

Check the devices for possibility to lowering packet size to be less than mtu used for the OpenVPN tunnel.

rodtre

Thank you for the answer, but the devices does'nt have any TCP configurations options, just IP address, netmask and gateway.

But I did a few more tests. Right now, this is the scenario:

Client A - connected to Internet with a radio link, PPPoE, MTU 1480 - Working
Client B - connected to Internet with FO link, PPPoE, MTU 1492 - NOT Working
Client C - connected to Internet with a radio link, MTU 1500 - NOT Working

I have changed the tun-mtu parameter in server and clients to 1492 with no effetct.

I realize now that even the operations that exchange very small amounts of data, and works in all clients, there is a difference.

In the middle of the operation, like getting the device serial number, on the working client (A), there is no
pause in the TCP traffic and the operation completes very fast.
In the other 2 clients (B and C), the operation completes sucessfully, but takes more time to complete and a pause of about 2 seconds happens ins the TCP traffic.

This operations sends 457 bytes and receives 544 bytes.

Here is the output of tcpdump on the cliente A LAN (Working):

10:38:28.785610 IP 10.8.0.1.57508 > 192.168.2.250.51000: Flags [S], seq 1681090528, win 8192, options [mss 1368,nop,wscale 8,nop,nop,sackOK], length 0
10:38:28.785670 IP 192.168.2.250.51000 > 10.8.0.1.57508: Flags [S.], seq 2840128231, ack 1681090529, win 4096, options [mss 1368], length 0

10:38:28.803160 IP 10.8.0.1.57508 > 192.168.2.250.51000: Flags [.], ack 1, win 64296, length 0
10:38:28.803179 IP 10.8.0.1.57508 > 192.168.2.250.51000: Flags [P.], seq 1:34, ack 1, win 64296, length 33

10:38:28.804740 IP 192.168.2.250.51000 > 10.8.0.1.57508: Flags [.], ack 34, win 4096, length 0
10:38:28.805854 IP 192.168.2.250.51000 > 10.8.0.1.57508: Flags [P.], seq 1:18, ack 34, win 4096, length 17
10:38:28.820140 IP 192.168.2.250.51000 > 10.8.0.1.57508: Flags [P.], seq 18:36, ack 34, win 4096, length 18

10:38:28.838225 IP 10.8.0.1.57508 > 192.168.2.250.51000: Flags [.], ack 36, win 64261, length 0
10:38:28.838246 IP 10.8.0.1.57508 > 192.168.2.250.51000: Flags [P.], seq 34:51, ack 36, win 64261, length 17

10:38:28.839258 IP 192.168.2.250.51000 > 10.8.0.1.57508: Flags [.], ack 51, win 4096, length 0

10:38:28.868112 IP 10.8.0.1.57508 > 192.168.2.250.51000: Flags [P.], seq 51:69, ack 36, win 64261, length 18

10:38:28.869131 IP 192.168.2.250.51000 > 10.8.0.1.57508: Flags [.], ack 69, win 4096, length 0
10:38:28.870206 IP 192.168.2.250.51000 > 10.8.0.1.57508: Flags [P.], seq 36:53, ack 69, win 4096, length 17
10:38:28.876211 IP 192.168.2.250.51000 > 10.8.0.1.57508: Flags [P.], seq 53:109, ack 69, win 4096, length 56

10:38:28.898227 IP 10.8.0.1.57508 > 192.168.2.250.51000: Flags [.], ack 109, win 64188, length 0
10:38:28.900641 IP 10.8.0.1.57508 > 192.168.2.250.51000: Flags [P.], seq 69:86, ack 109, win 64188, length 17

10:38:28.901678 IP 192.168.2.250.51000 > 10.8.0.1.57508: Flags [.], ack 86, win 4096, length 0

10:38:28.930580 IP 10.8.0.1.57508 > 192.168.2.250.51000: Flags [F.], seq 86, ack 109, win 64188, length 0

10:38:28.930639 IP 192.168.2.250.51000 > 10.8.0.1.57508: Flags [.], ack 87, win 4096, length 0

And this is the output of tcpdump on the LAN B client (NOT working):

10:46:57.537768 IP 10.8.0.1.57621 > 192.168.3.250.51000: Flags [S], seq 3161453309, win 8192, options [mss 1368,nop,wscale 8,nop,nop,sackOK], length 0
10:46:57.537823 IP 192.168.3.250.51000 > 10.8.0.1.57621: Flags [S.], seq 4221104935, ack 3161453310, win 4096, options [mss 1368], length 0

10:46:57.637767 IP 10.8.0.1.57621 > 192.168.3.250.51000: Flags [.], ack 1, win 64296, length 0
10:46:57.637787 IP 10.8.0.1.57621 > 192.168.3.250.51000: Flags [P.], seq 1:34, ack 1, win 64296, length 33

10:46:57.639341 IP 192.168.3.250.51000 > 10.8.0.1.57621: Flags [.], ack 34, win 4096, length 0
10:46:57.640447 IP 192.168.3.250.51000 > 10.8.0.1.57621: Flags [P.], seq 1:18, ack 34, win 4096, length 17
10:46:57.654745 IP 192.168.3.250.51000 > 10.8.0.1.57621: Flags [P.], seq 18:36, ack 34, win 4096, length 18

10:46:57.737797 IP 10.8.0.1.57621 > 192.168.3.250.51000: Flags [.], ack 36, win 64261, length 0
10:46:57.737817 IP 10.8.0.1.57621 > 192.168.3.250.51000: Flags [P.], seq 34:51, ack 36, win 64261, length 17
10:46:57.737837 IP 10.8.0.1.57621 > 192.168.3.250.51000: Flags [P.], seq 51:69, ack 36, win 64261, length 18

10:46:57.739350 IP 192.168.3.250.51000 > 10.8.0.1.57621: Flags [.], ack 69, win 4096, length 0

Here there is a 2 second pause.

10:47:01.237757 IP 10.8.0.1.57621 > 192.168.3.250.51000: Flags [P.], seq 69:87, ack 36, win 64261, length 18

10:47:01.238887 IP 192.168.3.250.51000 > 10.8.0.1.57621: Flags [.], ack 87, win 4096, length 0
10:47:01.239978 IP 192.168.3.250.51000 > 10.8.0.1.57621: Flags [P.], seq 36:53, ack 87, win 4096, length 17
10:47:01.246778 IP 192.168.3.250.51000 > 10.8.0.1.57621: Flags [P.], seq 53:109, ack 87, win 4096, length 56

10:47:01.337775 IP 10.8.0.1.57621 > 192.168.3.250.51000: Flags [.], ack 109, win 64188, length 0
10:47:01.337795 IP 10.8.0.1.57621 > 192.168.3.250.51000: Flags [P.], seq 87:104, ack 109, win 64188, length 17
10:47:01.337809 IP 10.8.0.1.57621 > 192.168.3.250.51000: Flags [F.], seq 104, ack 109, win 64188, length 0

10:47:01.337863 IP 192.168.3.250.51000 > 10.8.0.1.57621: Flags [.], ack 105, win 4079, length 0
10:47:01.338802 IP 192.168.3.250.51000 > 10.8.0.1.57621: Flags [.], ack 105, win 4096, length 0

Although, the operation completes sucessfully. The operation that sends a database table
fails, the ammount of data is about 8KB, and several pauses happens.

I guess that the application detect too many pauses when the data is larger and give up,
as a time out.

These are the ICMP delays pinging the server 10.8.0.1

Client A (Working): 20ms
Client B (NOT Working): 99ms
Client C (NOT Working): 99ms

Could be a delay problem and not a MTU problem, since the data is so small?

Mimiko

Ping delay is not high. Try to use lower MTU in configs. May be some device on the path to internet from clients are fragmenting packets.