No "Replay-window backtrack occurred [x]" messages

catkin

We would like to resolve an issue of commonly having "Authenticate/Decrypt packet error: bad packet ID (may be a replay)" messages.

We believe the underlying cause is that we have three load-balancing Internet-facing routers at the OpenVPN client site. Presumably the client's public IP address changes, the server somehow continues service but the UDP packet stream arriving at the client is disrupted.

Research suggests the solution is to configure the client's replay-window to suit the characteristics of the network and the value to use can be determined by running with verb to 4 to get "Replay-window backtrack occurred [x]" messages.

The client's config was changed and openvpn restarted using SIGUSR1 which seems to have worked. From the log:

SIGUSR1[soft,ping-restart] received, process restarting
Restart pause, 2 second(s)

After two days of running with verb 4, we still have plenty of "Authenticate/Decrypt packet error: bad packet ID (may be a replay)" messages but not a single "Replay-window backtrack occurred [x]" message.

Versions and configurations ...

Client running OpenVPN 2.1.3 on Debian 6.0.6 with this config:

client
dev tun
proto udp
remote XXX 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca /etc/openvpn/XXX/ca.crt
cert /etc/openvpn/XXX/XXX.crt
key /etc/openvpn/XXX/XXX.key
tls-auth /etc/openvpn/XXX/ta.key 1
ns-cert-type server
comp-lzo
verb 4
explicit-exit-notify 2
mssfix 1541

Server running OpenVPN 2.1.3 on Debian 6.0.6 with this config

port 1194                           #Use this port for OpenVPN
 
proto udp                           #Use UDP protocol
mssfix 1400
dev tun                             #Use network routing
ca /etc/openvpn/keys/ca.crt         #Full path to main certificate file 
cert /etc/openvpn/keys/server.crt   #Full path to server certificate
key /etc/openvpn/keys/server.key    #and key files
dh /etc/openvpn/keys/dh1024.pem     #Full path to Diffie Hellman file
tls-auth /etc/openvpn/keys/ta.key 0 #Full path to TLS key file
server 10.42.0.0 255.255.0.0        #Specify the virtual network address range and netmask
route 10.42.5.0 255.255.255.0
route 10.42.23.0 255.255.255.0
topology subnet
client-config-dir ccd               #Push specific IPs to clients;
route 10.42.0.0 255.255.255.252      #see section 14 below
push "dhcp-option WINS 10.42.0.137"
push "dhcp-option NBT 8"
client-to-client                    #Enable the VPN clients to see each other
keepalive 10 120                   
comp-lzo                            #Enable compression
user nobody                         #On UNIX systems, that will strip the
group nogroup                       #OpenVPN daemon from root access after launch for improved security
persist-key
persist-tun
ifconfig-pool-linear 
status openvpn-status.log
log    /var/log/openvpn.log
verb 1

janjust

can you try adding

float

to the client config to see if that helps?

catkin

janjust wrotecan you try adding

float

to the client config to see if that helps?

Thanks Jan 🙂

I've done that and will report the effect.

I'm not hopeful because it is the client that has changing public IPs but let's see.

catkin

Now 2 days and 2 hours since adding float to the client's config and no "Authenticate/Decrypt packet error: bad packet ID (may be a replay)" messages have been logged.

root:~# mkdir /tmp/tmp && cp -p /var/log/syslog* /tmp/tmp && cd /tmp/tmp && gunzip *gz 
root:/tmp/tmp# grep -h 'may be a replay' syslog* | grep -E 'Apr 0[1-9]|Apr 10' | wc -l 
13779
root:/tmp/tmp# grep -h 'may be a replay' syslog* | grep -E 'Apr 11|Apr 12' | wc -l 
0

Will continue to monitor the situation, especially as we saw "Authenticate/Decrypt packet error: bad packet ID (may be a replay)" messages on another OpenVPN client yesterday but could not reproduce it this morning.

Testing yesterday, we tail/followed the other OpenVPN client's syslog in one terminal and pinged the client by OpenVPN address from the OpenVPN server. For each ping there were "Authenticate/Decrypt packet error: bad packet ID (may be a replay)" messages. This casts doubt on whether the underlying cause is the load balancing router because the ping would generate a small number of packets.

When we can reproduce this behaviour, we ping a single packet, at least on the OpenVPN network (the number of packets that then travel over the underlying UDP network is not yet known).

In testing yesterday we noticed the number in brackets (the 281 in "bad packet ID (may be a replay): [ #281 ]") monotonically increasing by one whereas other reports of this message show discontinuous number sequences with large gaps. This may be associated with why we do not get the "Replay-window backtrack occurred [x]" messages ... ?

janjust

this sounds more and more like a dodgy router/switch somewhere, which is mangling network traffic ... your setup is not extraordinary, this works for others without any glitches.
Can you remove or switch routers/switches? can you switch to 'mode tcp' to see if it makes any difference?

catkin

Yes -- we will try one or both of those but it will take time.

catkin

Still planning to re-connect one of the two affected OpenVPN clients directly to one of our ISP connections, thus bypassing most of our internal network.

Meanwhile a pattern has been observed in when the "Replay-window backtrack occurred [x]" messages start to happen and stop happening. It is always immediately after the "Local Options String" and "Expected Remote Options String" pair of messages which I assume indicate a (re-)initialisation.

The Replay-window messages don't always start after the Options String messages or stop after them messages but every time (? -- there's a lot of archived messages to inspect) they do start or stop it is immediately after them.

catkin

janjust wrotethis sounds more and more like a dodgy router/switch somewhere, which is mangling network traffic ... your setup is not extraordinary, this works for others without any glitches.
Can you remove or switch routers/switches? can you switch to 'mode tcp' to see if it makes any difference?

Bang on the nail! We replaced one of our three WAN ADSL modem/routers three days ago and have not seen the reported messages since 😛