Hello All!
My Issue:
Every night around 7:00-8:00 pm PST the VPN speed for all users connected drop from around 90mb/s to 3mb/s or so. I have been working on trying to diagnose this issue for weeks now and had zero luck. It also occasionally happens for a few minutes during the day around 2:00PM PST.
After doing many packet captures, reading logs and tracerroutes from my home pc during an on going issue it seems around 8:00PM PST every night the VPN download speed goes from a stable 90mb/s or so to 3mb/s. This is happening for all users connected to the VPN, so its not an isolated issue.
How do I know this?
I setup a iperf script to run and upload/download every 3 minutes to a local machine. Where it uploads to a local machine on the domain, then does a reverse and downloads immediately after.
Here is an example of the start of the drop:
https://imgur.com/a/ZsrpMXM
As you can see in the images at the iperf at 5:06 everything is fine with 15mb/s upload and 90mb/s download which is expected for my home speeds. But then at 5:10 the upload remains fine at 15mb/s, but the download has dropped to 3mb/s and remained that low for roughly 2 hours. It seems the traffic is going into the VPN tunnel fine, but then is having issues going out. During this time of slow speed I have captured packet loss when pining machines within the local network and laggy RDP.
My Question:
What would cause something like this to happen? I have tried researching, going through Netgate support, going though ISP support. But no where can I find out why this is happening. I have spent roughly 3 weeks trying to figure this issue out looking at every possible area (That I can think of) and am at a loss now.
Notes on my setup/The Issue:
- Netgate XG-7100 1u, running pfsense 2.4.5
UDP openvpn setup
Default MTU: 1500. (I have run MTU tests during an issue and all matches up)
CPU never really goes over 20%. At the time of the issue on 4/8/2020 it was at 4%.
System RAM never goes above 15%.
I have OpenVPN logs set to verbosity: 4. But nothing alarming shows in the OpenVPN, Gateway, or system logs.
I have verified with our ISP during a live issue that the line is working as expected with no loss/latency was happening.
Issue is not isolated to one user. All connected users are affected.
OpenVPN .conf file:
dev tun
persist-tun
persist-key
cipher AES-256-CBC
ncp-ciphers AES-128-GCM
auth SHA256
tls-client
client
resolv-retry infinite
remote xxx.xxx.xxx.xx 1194 udp
auth-user-pass
ca xxxxx-xxxx-xxxx-ca.crt
tls-auth xxxxxx-xxxx-tls.key 1
remote-cert-tls server
route-nopull
route x.x.x.x xxx xxx.xxx.xxx.xxx
route xxx.xx.x.xxx xxx.xxx.xxx.xxx
route xx.xx.x.x
