OpenVPN tunnel for AWS VPC across regions / high dropped packet ratio

[nicolasg]

Hi everyone,

AWS documentation states VPC peering is only possible in a single region as per https://aws.amazon.com/answers/networki ... nectivity/ , in order to connect different VPC in different regions I followed this guide using EC2 Instances running OpenVPN https://aws.amazon.com/articles/0639686206802544 .

We have connectivity between the VPC in us-east-1 and the in us-west-1 but are having problems when transferring big amount of data through tunnel, the jobs that push the data runs only for 5 - 10 minutes and after that the connections is dropped due to high number of packet drops.

I thought this is due to the instance type limitation and have increased the size , we tried transferring data using the largest instance size m4.10xlarge to take full advantage of the 10 GB network card but unfortunately the results are the same. During that period we see a lot of network packet errors and packets dropped .

This is a hadoop cluster in us-east-1 trying to populate with data an ElasticCache MemcacheD cluster in us-west-1 , the size of data we need to transfer is 300 GB per day.

Here is some more details :

The OpenVPN instances :

CentOS Linux release 7.2.1511 (Core),
OpenVPN 2.3.14
latest network driver installed
Enhanced networking enabled

Ifconfig outputs :

us-east-1 sending the data :

Code: Select all

        tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 169.254.255.3  netmask 255.255.255.255  destination 169.254.255.22
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 100  (UNSPEC)
        RX packets 11419056  bytes 671725364 (640.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12221637  bytes 12380856060 (11.5 GiB)
        TX errors 0  dropped 4486568 overruns 0  carrier 0  collisions 0

us-west-1 receiving the data :

Code: Select all

        tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 169.254.255.22  netmask 255.255.255.255  destination 169.254.255.3
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 100  (UNSPEC)
        RX packets 11532352  bytes 11538601450 (10.7 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10265430  bytes 682782612 (651.1 MiB)
        TX errors 0  dropped 459409 overruns 0  carrier 0  collisions 0

openvpn server configuration :

user nobody
group nobody
port 1195
dev tun

remote <ip>
route 10.3.0.0 255.255.0.0

ifconfig 169.254.255.3 169.254.255.22
secret ovpn.key

status /var/log/openvpn/openvpn-status.log
log-append /var/log/openvpn/openvpn.log
verb 6
mute 20

Can you please advise ?

[nicolasg]

I just realised even with no significant traffic activity I still see a high number of dropped packets :

TX packets 11815936 bytes 11820781579 (11.0 GiB)
TX errors 0 dropped 2215793 overruns 0 carrier 0 collisions 0

any clues ?

[TinCanTech]

You are using Openvpn Community edition (the free one) correct ?

[nicolasg]

Yes TinCanTech .

I'v also just found out that the number of dropped packets it's actually I counter that you have to reset or it's cleared at reboot time so ignore my last post .

[nicolasg]

It seems the default txqueuelen OpenVPN sets for the TUN interface is 100 when by default CentOS set it to 1000 for eth0 ...

I'v increased the txqueuelen to 10000 and see lower number of packet drops and can transfer data for longer time but it still dies after some time...

I now see in the logs :

Code: Select all

Wed Jan 25 12:32:17 2017 us=311972 PID_ERR large diff [159] [STATIC-0] [0__0___0000__00____0000___000000__000_0_0_0_000000000___00____0_] 1485294859:31321014 1485294859:31320855 t=1485347537[0] r=[-3,64,15,217,1] sl=[35,64,64,528]
Wed Jan 25 12:32:17 2017 us=311979 Authenticate/Decrypt packet error: bad packet ID (may be a replay): [ #31320855 / time = (1485294859) Tue Jan 24 21:54:19 2017 ] – see the man pa

I tried increasing the txqueuelen to 20000 and get different error :

Wed Jan 25 13:42:35 2017 us=25545 PID_ERR replay-window backtrack occurred [50] [STATIC-0] [0__________________________________________________0000000000000] 1485351278:2803380 1485351278:2803330 t=1485351755[0] r=[-2,64,15,50,1] sl=[50,64,64,528]
Wed Jan 25 13:42:35 2017 us=26097 PID_ERR replay-window backtrack occurred [52] [STATIC-0] [00000000_____________________________________________00__0000000] 1485351278:2803387 1485351278:2803335 t=1485351755[0] r=[-2,64,15,52,1] sl=[43,64,64,528]

[TinCanTech]

I tried increasing the txqueuelen to 20000

Looks like you are poking the bear and making it mad ..

[nicolasg]

TinCanTech could you explain more ?

[TinCanTech]

Ask yourself: why am I increasing the queue length ?

I was trying to think of suitable analogy .. I came up with this:

Imagine you are using a hand cranked mincing machine to extrude sausage meat
into sausage skin. By increasing the queue length, you are making the hopper
(where you load the pork into the mincer) very much larger .. but the mincer still
only has the same capacity to mince and extrude the same amount of meat.

Does this data traverse the public internet ?

Finally, please see:
HOWTO: Request Help !