[edit: title]
Hi all,
I hope somebody can help out with a seriously vexing problem we have been struggling with for some time.
We have an openvpn server, and 18 networks connecting to this server, all networks can see all other networks.
A few months ago, out of the blue, we started seeing drops on the vpn, where all traffic just stops for anywhere between 1 to 10 seconds, but most of the time 5 seconds, with a 10 second drop being next common. This happens every 10.x minutes. We have tried many things, including switching off the key renegotiation for some time, to get to the root of this issue. As one of the traffic types on this network is SIP, this obviously leads to frustrating problems for the users.
Running tcpdump, and importing the dump into wireshark, shows a clear drop of all traffic for that period - i.e. no packets transmit or receive for that time on tun0. Nothing shows in the vpn logs at all, no reconnect, no renegotiation, nothing at all. dmesg is also clean, firewall works fine. This is a busy server, and we have increased txqueuelen and a bunch of other tcp parameters, all to no avail.
For all intents and purposes, the machine (a VMWare ESXi VM) looks fine, and connections not running over the VPN run without a hitch. The drop is regular - like I said, every 10 minutes and a tiny bit (the minutes will roll over once a day, yesterday it was on xx:x6 today it is on xx:x7) We see no increases in traffic or load on this machine at all during the drop-outs, except for an increase in cpu wait. Here is a dstat -af for the drop
-------cpu0-usage--------------cpu1-usage------ --dsk/sda-- --net/eth0----net/eth1----net/eth2----net/tun0----net/tun1- ---paging-- ---system--
usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| recv send: recv send: recv send: recv send: recv send| in out | int csw
1 0 99 0 0 0: 1 1 97 0 0 1| 0 0 | 79k 117k: 109k 89k:1852B 0 : 33k 12k: 0 0 | 0 0 |1445 650
0 1 98 0 0 1: 0 0 98 0 0 2| 0 0 | 63k 122k: 129k 61k:2188B 0 : 18k 20k: 0 0 | 0 0 |1212 597
1 0 98 0 0 1: 0 0 99 0 0 1| 0 0 | 82k 211k: 235k 72k: 948B 42B: 33k 30k: 259B 1373B| 0 0 |1361 776
0 0 99 0 0 1: 0 1 99 0 0 0| 0 0 | 72k 171k: 144k 60k:1116B 42B: 33k 95k: 0 0 | 0 0 |1120 651
0 0 100 0 0 0: 0 0 100 0 0 0| 0 0 | 45k 89k: 68k 36k: 780B 42B: 11k 19k: 151B 40B| 0 0 | 984 530
0 0 100 0 0 0: 0 0 16 84 0 0| 0 0 | 37k 72k: 56k 30k: 840B 0 : 11k 20k: 0 0 | 0 0 | 850 444
0 0 100 0 0 0: 0 1 0 99 0 0| 0 0 | 48k 78k: 70k 32k:1796B 1256B:6328B 5858B: 93B 81B| 0 0 | 960 427
0 0 100 0 0 0: 0 0 0 100 0 0| 0 0 | 38k 30k: 45k 20k: 840B 0 : 0 0 : 0 0 | 0 0 | 589 105
0 0 100 0 0 0: 0 0 0 100 0 0| 0 0 | 38k 40k: 99k 25k: 780B 0 : 0 0 : 122B 0 | 0 0 | 735 193
0 0 99 0 0 1: 0 0 0 100 0 0| 0 0 | 41k 104k: 69k 22k: 840B 0 : 0 0 : 61B 0 | 0 0 | 711 177
0 0 99 0 0 1: 0 0 0 100 0 0| 0 0 | 46k 40k: 81k 37k: 60B 0 : 0 0 : 0 0 | 0 0 | 820 201
0 0 99 0 0 1: 0 0 0 100 0 0| 0 0 | 40k 46k: 47k 26k: 960B 0 : 0 0 : 0 0 | 0 0 | 702 177
0 0 100 0 0 0: 0 0 0 100 0 0| 0 0 | 50k 105k: 117k 38k:1176B 0 : 0 0 : 0 0 | 0 0 | 872 240
0 0 100 0 0 0: 1 0 0 99 0 0| 0 0 | 45k 25k: 33k 23k:1208B 0 : 0 0 : 0 0 | 0 0 | 743 14k
0 0 100 0 0 0: 0 0 0 100 0 0| 0 0 | 48k 169k: 200k 28k: 244B 0 : 0 0 : 0 0 | 0 0 | 811 240
0 0 99 0 0 1: 0 0 0 100 0 0| 0 0 | 44k 60k: 68k 25k: 840B 0 : 0 0 : 0 0 | 0 0 | 746 199
0 1 99 0 0 0: 1 0 0 98 0 1| 0 0 | 46k 64k: 92k 29k:1020B 216B: 0 0 : 0 0 | 0 0 | 814 265
0 0 100 0 0 0: 0 0 0 100 0 0| 0 0 | 43k 31k: 27k 20k: 840B 0 : 0 0 : 789B 0 | 0 0 | 599 123
0 0 99 0 0 1: 0 0 0 100 0 0| 0 0 | 43k 17k: 29k 18k: 60B 0 : 0 0 : 0 0 | 0 0 | 634 165
0 0 87 13 0 0: 0 0 0 100 0 0| 0 0 | 46k 19k: 32k 25k: 840B 0 : 0 0 : 41B 40B| 0 0 | 635 143
0 0 0 100 0 0: 1 0 0 96 0 3| 0 0 | 44k 95k: 116k 36k:1072B 240B: 0 0 : 52B 41B| 0 0 | 850 255
0 0 0 99 0 1: 0 0 0 100 0 0| 0 24k| 43k 76k: 124k 37k: 900B 42B: 0 0 : 0 0 | 0 0 | 904 242
0 0 0 100 0 0: 0 0 0 100 0 0| 0 0 | 48k 57k: 43k 24k: 60B 0 : 0 0 : 59B 0 | 0 0 | 789 188
0 0 0 100 0 0: 0 1 0 99 0 0| 0 0 | 41k 60k: 73k 24k: 840B 0 : 0 0 : 0 0 | 0 0 | 707 175
0 1 0 99 0 0: 0 1 77 22 0 0| 0 24k| 63k 331k: 71k 57k:2055B 689B: 23k 190k: 557B 2358B| 0 0 |1269 496
0 0 0 100 0 0: 0 0 100 0 0 0| 0 0 | 51k 86k: 116k 37k:1576B 0 : 0 0 : 0 0 | 0 0 | 863 255
0 0 11 89 0 0: 0 1 99 0 0 0| 0 32k| 68k 131k: 185k 62k: 754B 326B: 20k 32k: 0 0 | 0 0 |1150 453
0 0 99 0 0 1: 1 3 95 0 0 1| 0 0 | 115k 898k:1451k 114k:2878B 2192B: 29k 24k: 0 0 | 0 0 |2204 1370
0 0 99 0 0 1: 2 2 95 0 0 1| 0 0 | 223k 1202k: 585k 136k:1176B 396B: 19k 76k: 0 0 | 0 0 |3551 2180
0 1 99 0 0 0: 3 1 95 0 0 1| 0 16k| 125k 322k: 210k 75k: 840B 0 : 16k 19k: 0 0 | 0 0 |1926 1278
0 0 100 0 0 0: 1 0 99 0 0 0| 0 0 | 121k 176k: 81k 41k:2740B 1218B: 15k 14k: 0 0 | 0 0 |1419 1051
0 0 100 0 0 0: 2 1 96 0 0 1| 0 0 | 153k 239k: 138k 53k:1464B 624B: 25k 23k: 0 0 | 0 0 |1998 1537
and another one, somewhat shorter freeze:
0 0 100 0 0 0: 1 4 91 0 0 4| 0 0 | 290k 264k: 69k 135k: 60B 0 : 33k 14k: 0 0 | 0 0 |2677 2084
0 0 100 0 0 0: 1 0 98 0 0 1| 0 0 | 99k 132k: 83k 50k: 840B 0 : 19k 17k: 0 0 | 0 0 |1375 838
0 0 100 0 0 0: 2 0 95 0 0 3| 0 16k| 219k 317k: 179k 61k:3142B 2943B: 18k 14k: 0 0 | 0 0 |2597 2007
0 0 100 0 0 0: 1 0 98 0 0 1| 0 0 | 145k 174k: 64k 51k: 840B 0 : 22k 16k:1798B 971B| 0 0 |1751 1268
0 0 100 0 0 0: 1 3 96 0 0 0| 0 0 | 176k 269k: 148k 61k: 120B 0 : 18k 13k: 0 0 | 0 0 |2191 1564
0 0 100 0 0 0: 2 0 97 0 0 1| 0 0 | 119k 183k: 163k 100k:2096B 0 : 16k 14k:1802B 973B| 0 0 |1920 1025
0 0 100 0 0 0: 1 1 98 0 0 0| 0 0 | 196k 247k: 105k 47k:2160B 0 : 18k 12k: 0 0 | 0 0 |1632 1090
0 1 99 0 0 0: 1 0 98 0 0 1| 0 0 | 231k 384k: 221k 61k:2672B 630B: 21k 13k: 0 0 | 0 0 |1960 2389
0 0 100 0 0 0: 1 0 85 14 0 0| 0 0 | 212k 266k: 83k 50k: 146B 70B: 18k 12k: 0 0 | 0 0 |1756 1163
0 0 100 0 0 0: 2 2 0 96 0 0| 0 0 | 210k 225k: 54k 40k: 840B 0 : 14k 11k: 0 0 | 0 0 |1660 1175
0 0 99 0 0 1: 0 0 0 100 0 0| 0 0 | 201k 264k: 98k 39k: 780B 0 : 10k 13k: 0 0 | 0 0 |1607 1115
0 0 99 0 0 1: 0 0 0 100 0 0| 0 0 | 137k 161k: 99k 39k: 840B 0 : 10k 18k: 0 0 | 0 0 |1193 676
-------cpu0-usage--------------cpu1-usage------ --dsk/sda-- --net/eth0----net/eth1----net/eth2----net/tun0----net/tun1- ---paging-- ---system--
usr sys idl wai hiq siq:usr sys idl wai hiq siq| read writ| recv send: recv send: recv send: recv send: recv send| in out | int csw
0 0 100 0 0 0: 0 0 0 100 0 0| 0 0 | 50k 56k: 75k 23k: 60B 0 : 0 0 : 0 0 | 0 0 | 684 176
0 0 100 0 0 0: 0 0 0 100 0 0| 0 0 | 37k 42k: 45k 20k: 840B 0 : 0 0 : 0 0 | 0 0 | 636 128
0 0 100 0 0 0: 0 0 0 100 0 0| 0 0 | 38k 32k: 47k 21k: 780B 0 : 0 0 : 0 0 | 0 0 | 651 163
0 0 100 0 0 0: 0 0 0 100 0 0| 0 0 | 52k 17k: 43k 40k:1208B 0 : 0 0 : 0 0 | 0 0 | 812 140
0 0 100 0 0 0: 0 0 0 100 0 0| 0 0 | 42k 35k: 66k 30k: 704B 0 : 0 0 : 0 0 | 0 0 | 736 180
0 0 100 0 0 0: 0 1 0 99 0 0| 0 0 | 48k 42k: 46k 24k:1944B 0 : 0 0 : 0 0 | 0 0 | 724 142
0 0 100 0 0 0: 0 0 18 81 0 1| 0 32k| 77k 199k: 76k 59k:1531B 42B: 29k 97k: 52B 41B| 0 0 |1033 407
0 0 100 0 0 0: 2 1 97 0 0 0| 0 0 | 140k 185k: 119k 62k:1037B 131B: 30k 14k: 0 0 | 0 0 |1676 15k
0 0 99 0 0 1: 0 1 98 0 0 1| 0 0 | 178k 508k: 416k 66k: 278B 260B: 15k 14k: 0 0 | 0 0 |1916 1284
0 0 100 0 0 0: 2 1 97 0 0 0| 0 4096B| 209k 274k: 197k 59k:1275B 263B: 19k 12k: 52B 41B| 0 0 |1740 1095
0 1 99 0 0 0: 0 1 98 0 0 1| 0 8192B| 243k 600k: 334k 73k: 780B 0 : 43k 255k: 614B 5333B| 0 0 |2129 1478
0 0 99 0 0 1: 0 2 81 14 0 3| 0 4096B| 215k 311k: 147k 52k: 840B 0 : 25k 58k: 912B 6992B| 0 0 |1900 1330
0 0 100 0 0 0: 2 0 71 26 0 1| 0 20k| 219k 326k: 159k 53k: 60B 0 : 22k 83k: 52B 41B| 0 0 |2111 1526
0 0 100 0 0 0: 3 1 95 0 0 1| 0 0 | 74k 118k: 105k 45k: 840B 42B: 11k 11k: 0 0 | 0 0 |1376 818
0 0 100 0 0 0: 2 1 96 0 0 1| 0 0 | 211k 272k: 98k 48k: 780B 42B: 12k 11k: 0 0 | 0 0 |1680 1097
0 0 100 0 0 0: 5 0 93 0 0 2| 0 0 | 218k 298k: 134k 52k: 840B 42B: 11k 11k: 93B 81B| 0 0 |1928 1309
The drop is pretty obvious - cpu1 wait goes up to 100, tun0 drops like a stone. Any help or pointers at this time would be greatly appreciated.
Thanks
