OpenVPN client floating write UDPv4: Operation not permitted

kubco2

Hello,

I have 2 wan with failover, and I want openvpn client which can hops between 2 wans.
This is my firewall:

filter:
-I OUTPUT -p udp --dport 1199 -j ACCEPT
-I INPUT -p udp --sport 1199 -j ACCEPT
mangle:
-A PREROUTING -j mwan3_hook
-A OUTPUT -j mwan3_hook
-A mwan3_hook -p udp -m udp --sport 1199 -j openvpn
-A mwan3_hook -p udp -m udp --dport 1199 -j openvpn

-A mwan3_iface_wan -s 10.1.5.0/24 -i eth0 -m mark --mark 0x0/0xff00 -m comment --comment default -j MARK --set-xmark 0xff00/0xff00
-A mwan3_iface_wan -i eth0 -m mark --mark 0x0/0xff00 -m comment --comment wan -j MARK --set-xmark 0x100/0xff00
-A mwan3_iface_wan3g -s 10.64.64.64/32 -i 3g-wan3g -m mark --mark 0x0/0xff00 -m comment --comment default -j MARK --set-xmark 0xff00/0xff00
-A mwan3_iface_wan3g -i 3g-wan3g -m mark --mark 0x0/0xff00 -m comment --comment wan3g -j MARK --set-xmark 0x200/0xff00

-A mwan3_ifaces -m mark --mark 0x0/0xff00 -j mwan3_iface_wan
-A mwan3_ifaces -m mark --mark 0x0/0xff00 -j mwan3_iface_wan3g

-A mwan3_policy_policy1 -m mark --mark 0x0/0xff00 -m comment --comment "wan3g 3 3" -j MARK --set-xmark 0x200/0xff00

-A mwan3_rules -m mark --mark 0x0/0xff00 -m comment --comment rule1 -j mwan3_policy_policy1

-A openvpn -j MARK --set-xmark 0x0/0xff00
-A openvpn -m mark --mark 0x0/0xff00 -j mwan3_ifaces
-A openvpn -m mark --mark 0x0/0xff00 -j mwan3_rules
-A openvpn -j CONNMARK --save-mark --nfmask 0xff00 --ctmask 0xff00
-A openvpn -j LOG
-A openvpn -j ACCEPT
-A openvpn -j LOG

Logs while failover from wan to 3gwan

Fri May 15 02:22:24 2015 user.notice mwan3track: Interface wan (eth0) is offline
Fri May 15 02:22:24 2015 user.notice mwan3: ifdown interface wan (eth0)
Fri May 15 02:22:26 2015 daemon.notice openvpn(client)[18493]: UDPv4 WRITE [113] to [AF_INET]OPENVPNSERVER:1199: P_DATA_V1 kid=0 DATA len=112
Fri May 15 02:22:26 2015 kern.warn kernel: [96743.530000] IN= OUT=eth0 SRC=10.1.5.246 DST=OPENVPNSERVER LEN=141 TOS=0x00 PREC=0x00 TTL=64 ID=15300 DF PROTO=UDP SPT=36338 DPT=1199 LEN=121 MARK=0x200 
Fri May 15 02:22:26 2015 daemon.err openvpn(client)[18493]: write UDPv4: Operation not permitted (code=1)
Fri May 15 02:22:34 2015 daemon.notice openvpn(client)[18493]: UDPv4 WRITE [113] to [AF_INET]OPENVPNSERVER:1199: P_DATA_V1 kid=0 DATA len=112
Fri May 15 02:22:34 2015 kern.warn kernel: [96751.560000] IN= OUT=eth0 SRC=10.1.5.246 DST=OPENVPNSERVER LEN=141 TOS=0x00 PREC=0x00 TTL=64 ID=15301 DF PROTO=UDP SPT=36338 DPT=1199 LEN=121 MARK=0x200 
Fri May 15 02:22:42 2015 daemon.notice openvpn(client)[18493]: UDPv4 WRITE [113] to [AF_INET]OPENVPNSERVER:1199: P_DATA_V1 kid=0 DATA len=112
Fri May 15 02:22:42 2015 kern.warn kernel: [96759.580000] IN= OUT=eth0 SRC=10.1.5.246 DST=OPENVPNSERVER LEN=141 TOS=0x00 PREC=0x00 TTL=64 ID=15302 DF PROTO=UDP SPT=36338 DPT=1199 LEN=121 MARK=0x200 
Fri May 15 02:22:45 2015 daemon.notice openvpn(client)[18493]: [server] Inactivity timeout (--ping-restart), restarting
Fri May 15 02:22:45 2015 daemon.notice openvpn(client)[18493]: TCP/UDP: Closing socket

As I can see packets got right mark 0x200 and should go out by 3gwan interface, but according to log they go by wan(eth0) interface

I want to ask what this error can be

Fri May 15 02:22:26 2015 daemon.err openvpn(client)[18493]: write UDPv4: Operation not permitted (code=1)

Thanks for help

Traffic

Please post your server and client config files

kubco2

server.conf

mode server
tls-server
port 1199
topology subnet
push "topology subnet"
push "route-gateway 10.200.0.1"
ifconfig 10.200.0.1 255.255.0.0
ifconfig-pool 10.200.1.50 10.200.255.254
proto udp
dev tun0
ca  /etc/openvpn/ca.crt
dh /etc/openvpn/dh2048.pem
cert /etc/openvpn/server.crt
key /etc/openvpn/server.key
cipher AES-256-CBC
auth SHA512
verb 3
keepalive 8 35
comp-lzo
push "comp-lzo yes"
persist-key
persist-tun
float
duplicate-cn
status /var/www/ovpnp-status.log 5
status-version 3
client-config-dir client_public
ccd-exclusive
log-append /var/www/ovpnp.log

client.conf

client
dev tun99
nobind
;float #tested with and without
remote OPENVPNSERVER 1199 udp
cipher AES-256-CBC
auth SHA512
comp-lzo adaptive
ns-cert-type server
+CERTS by cmd line

Traffic

I am not absolutely sure but .. I think this:

kubco2 wrote want to ask what this error can be
Code:
Fri May 15 02:22:26 2015 daemon.err openvpn(client)[18493]: write UDPv4: Operation not permitted (code=1)

may be caused by the local network interface going down ..

As you can see here:

kubco2 wroteFri May 15 02:22:45 2015 daemon.notice openvpn(client)[18493]: [server] Inactivity timeout (--ping-restart), restarting

OpenVPN restarts the client instance .. what happens after this restart .. do you get a new connection ?

BTW: --float is enabled by default on a server .. you only need to specify --float in a client config if the server is expected to change IP address quite frequently.

kubco2

Traffic wrotemay be caused by the local network interface going down ..

Wan interface is not truly down. Only mangle rules in firewall add mark 0x200 to direct packets over 3gwan interface, but they goes by wan instead 3gwan 🙁.

Traffic wroteOpenVPN restarts the client instance .. what happens after this restart .. do you get a new connection ?

After ping restart, connection is established fine over 3gwan.

If I remove openvpn chain, only new connections(by mangle rules) are redirected over 3gwan, so openvpn do first ping-restart and then connect over 3gwan.
I add openvpn chain to force openvpn packets to be redirected always to online interface.

Maybe openvpn client write directly to interface which was selected on connection initialization, so it cant change outgoing interface on the fly? I dont know very well how it works.

Traffic

kubco2 wroteWan interface is not truly down

OpenVPN thinks the interface is down:

kubco2 wroteLogs while failover from wan to 3gwan
Code:
Fri May 15 02:22:24 2015 user.notice mwan3track: Interface wan (eth0) is offline
Fri May 15 02:22:24 2015 user.notice mwan3: ifdown interface wan (eth0)

kubco2 wroteMaybe openvpn client write directly to interface which was selected on connection initialization, so it cant change outgoing interface on the fly?

Maybe ..

What version of openvpn does your client use ?

kubco2

OpenVPN thinks the interface is down

It is syslog and multiwan log entry

And I use OpenVPN 2.3.6

Traffic

kubco2 wroteserver.conf
Code:
client-config-dir client_public
ccd-exclusive

What directives are in your CCD file ?

kubco2

I use ccd-exclusive for disabling clients instead CRL. So only empty files.

Traffic

You could try making the --keepalive a longer period eg: 10 300

kubco2

No change with longer timeouts.