"could not execute external program" (chroot-ed environment)

[mr-4]

I am getting the above error in instances where my OpenVPN provider resets the connection and changes the ip address of the tun0 device. I am using OpenVPN 2.1.1 i686-redhat-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] built on Jan 5 2010.

When the connection is reset, but the ip address is not changed all is well.

I am also getting an error when I terminate OpenVPN (service openvpn stop). I need to mention that I am running OpenVPN in chrooted environment (uid:gid is _openvpn:_openvpn). The root (/) directory is in /var/lib/openvpn


I first thought that it may be a permission problem, but that does not appear to be the case (see below). Any ideas?

Here are my logs (xxx is my openvpn provider external ip address, yyy is my local network internal ip address, zzz is the ip address of the local tun device as assigned by my service provider):

-----------connection reset, ip (tun0) changed--------------------
Aug 19 04:44:51 test1 openvpn[6415]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
Aug 19 04:44:51 test1 openvpn[6415]: /sbin/ip route del 239.0.0.0/8
Aug 19 04:44:51 test1 openvpn[6415]: ERROR: Linux route delete command failed: could not execute external program
Aug 19 04:44:51 test1 openvpn[6415]: /sbin/ip route del xxx.xxx.xxx.xxx/32
Aug 19 04:44:51 test1 openvpn[6415]: ERROR: Linux route delete command failed: could not execute external program
Aug 19 04:44:51 test1 openvpn[6415]: /sbin/ip route del 0.0.0.0/1
Aug 19 04:44:51 test1 openvpn[6415]: ERROR: Linux route delete command failed: could not execute external program
Aug 19 04:44:51 test1 openvpn[6415]: /sbin/ip route del 128.0.0.0/1
Aug 19 04:44:51 test1 openvpn[6415]: ERROR: Linux route delete command failed: could not execute external program
Aug 19 04:44:51 test1 openvpn[6415]: Closing TUN/TAP interface
---------------------------------------------------------------------

-----------service openvpn stop executed-----------------------------
Aug 20 13:04:42 test1 openvpn[1589]: /sbin/ip route del 239.0.0.0/8
Aug 20 13:04:42 test1 openvpn[1589]: ERROR: Linux route delete command failed: could not execute external program
Aug 20 13:04:42 test1 openvpn[1589]: /sbin/ip route del yyy.yyy.yyy.yyy/29
Aug 20 13:04:42 test1 openvpn[1589]: ERROR: Linux route delete command failed: could not execute external program
Aug 20 13:04:42 test1 openvpn[1589]: /sbin/ip route del xxx.xxx.xxx.xxx/32
Aug 20 13:04:42 test1 openvpn[1589]: ERROR: Linux route delete command failed: could not execute external program
Aug 20 13:04:42 test1 openvpn[1589]: /sbin/ip route del 0.0.0.0/1
Aug 20 13:04:42 test1 openvpn[1589]: ERROR: Linux route delete command failed: could not execute external program
Aug 20 13:04:42 test1 openvpn[1589]: /sbin/ip route del 128.0.0.0/1
Aug 20 13:04:42 test1 openvpn[1589]: ERROR: Linux route delete command failed: could not execute external program
Aug 20 13:04:42 test1 openvpn[1589]: Closing TUN/TAP interface
Aug 20 13:04:42 test1 openvpn[1589]: /sbin/ip addr del dev tun0 zzz.zzz.zzz.zzz/24
Aug 20 13:04:42 test1 openvpn[1589]: Linux ip addr del failed: could not execute external program
---------------------------------------------------------------------

-----------connection reset, ip (tun0) NOT changed--------------------
Aug 18 03:18:23 test1 openvpn[6415]: OPTIONS IMPORT: timers and/or timeouts modified
Aug 18 03:18:23 test1 openvpn[6415]: OPTIONS IMPORT: --ifconfig/up options modified
Aug 18 03:18:23 test1 openvpn[6415]: OPTIONS IMPORT: route options modified
Aug 18 03:18:23 test1 openvpn[6415]: OPTIONS IMPORT: route-related options modified
Aug 18 03:18:23 test1 openvpn[6415]: OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Aug 18 03:18:23 test1 openvpn[6415]: Preserving previous TUN/TAP instance: tun0
Aug 18 03:18:23 test1 openvpn[6415]: Initialization Sequence Completed
---------------------------------------------------------------------

-bash-4.1# ls -las /var/lib/openvpn/sbin/
total 240
4 drwxr-x---. 2 _openvpn root 4096 Aug 18 01:22 .
4 drwxr-x---. 3 _openvpn root 4096 Aug 18 01:22 ..
232 -rwxr-x---. 1 root _openvpn 233244 Aug 18 01:22 ip

[ecrist]

The error seems pretty straight-forward to me, especially since you claim you're running OpenVPN in a chroot environment. It would appear that your route change commands aren't able to be executed. Part of this is likely due to the change in privileges due to the user and group change to _openvpn. Does the _openvpn user or group have access to the route command, and rights to change the routing table? Also, does the route command exist within the chroot environment?

[mr-4]

The error seems pretty straight-forward to me, especially since you claim you're running OpenVPN in a chroot environment. It would appear that your route change commands aren't able to be executed. Part of this is likely due to the change in privileges due to the user and group change to _openvpn. Does the _openvpn user or group have access to the route command, and rights to change the routing table? Also, does the route command exist within the chroot environment?

I think it has (see the last part of my initial post):

Code: Select all

-bash-4.1# ls -las /var/lib/openvpn/sbin/
total 240
4 drwxr-x---. 2 _openvpn root 4096 Aug 18 01:22 .
4 drwxr-x---. 3 _openvpn root 4096 Aug 18 01:22 ..
232 -rwxr-x---. 1 root _openvpn 233244 Aug 18 01:22 ip

The 'ip' program, which is used by OpenVPN to alter the parameters/routing table is present and available to the new root (/var/lib/openvpn) - see above - and it has the correct permissions (rwx as directory owner and rx on the 'ip' file as well), at least this is what I think (I stand to be corrected on this as I don't claim to be an expert in OpenVPN, hence my initial post in this thread).

[ecrist]

chroot yourself into that dir and try to execute the command. I'll bet you find you don't have access to change the routing tables.

[mr-4]

chroot yourself into that dir and try to execute the command. I'll bet you find you don't have access to change the routing tables.

How bizarre!

I did try to chroot and I couldn't, let alone execute ANY command. I tried 'chroot /var/lib/openvpn' and it failed with 'chroot: failed to run command `/bin/bash': No such file or directory'.

So, I created /var/lib/openvpn/bin and copied 'bash' in it, then altered ALL the necessary permissions (enabled o+r+x recursively on /var/lib/openvpn just to make sure) and still no go (note that with the above command I am trying to chroot as root!).

When I try 'chroot --userspec=_openvpn:_openvpn /var/lib/openvpn' I get 'chroot: invalid user'. I get the same message when try 'chroot --userspec=root:root /var/lib/openvpn', which I find rather bizarre!

I also tried the numerical values, i.e. 'chroot --userspec=499:499 /var/lib/openvpn' (499 = _openvpn in /etc/passwd) - no go: 'chroot: failed to run command `/bin/bash': No such file or directory' error message. I also tried 'chroot --userspec=0:0 /var/lib/openvpn' (0 = root) - same error message.

I did also try 'chroot --userspec=450:450 /var/lib/openvpn' (450 does NOT correspond to any user registered in /etc/passwd!) - same error message as above, so clearly the numerical values in --userspec are ignored. I also tried --userspec=root, --userspec=_openvpn - same effect: 'chroot: invalid user'!

As a last resort I tried 'chroot /var/lib/openvpn /sbin/ip' - that did not work either ('chroot: failed to run command `/sbin/ip': No such file or directory' error message). What am I missing here?!

All files seem to be present and the permissions were set to o+r+x on all files and directories below and including /var/lib/openvpn, so that should allow me access to anything in these directories, isn't that right?

When I try 'su - _openvpn' (I altered etc/passwd with 'usermod -d /var/lib/openvpn -s /bin/bash _openvpn' before that to make sure _openvpn has the right access) I get into /var/lib/openvpn and from there I could happily execute 'sbin/ip' - no problems!

So what seems to be wrong?

[ecrist]

The problem you're running in to right now is that your user shell isn't available inside the chroot. In addition, you need to execute the ip script with it's relative path once in the chroot. For example, if you've got a directory, /chroot, and a script, /chroot/ip.sh, when you chroot yourself, you need to run 'chroot /ip.sh' since your new root is within /chroot.

Also, I really don't think you're going to be able to change the system routing table from within the chroot.

[mr-4]

The problem you're running in to right now is that your user shell isn't available inside the chroot. In addition, you need to execute the ip script with it's relative path once in the chroot. For example, if you've got a directory, /chroot, and a script, /chroot/ip.sh, when you chroot yourself, you need to run 'chroot /ip.sh' since your new root is within /chroot.

Well, that is precisely what I am doing - executing /sbin/ip (this is a program, not a script!) as the directory structure is (using your example above) /chroot/sbin/ip, with the appropriate permissions. As for not having the shell (bin/bash or /bin/sh) available - read my previous post - they ARE available in the chrooted tree.

Also, I really don't think you're going to be able to change the system routing table from within the chroot.

I decided, as a test, to temporarily remove the chroot restrictions and run openvpn to be constrained just by its own uid:gid (which is _openvpn:_openvpn).

I am getting similar errors:

Code: Select all

Aug 26 04:44:58 test1 openvpn[1582]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
Aug 26 04:44:58 test1 openvpn[1582]: /sbin/ip route del 239.0.0.0/8
Aug 26 04:44:58 test1 openvpn[1582]: ERROR: Linux route delete command failed: external program exited with error status: 2
Aug 26 04:44:58 test1 openvpn[1582]: /sbin/ip route del xxx.xxx.xxx.xxx/32
Aug 26 04:44:58 test1 openvpn[1582]: ERROR: Linux route delete command failed: external program exited with error status: 2
Aug 26 04:44:58 test1 openvpn[1582]: /sbin/ip route del 0.0.0.0/1
Aug 26 04:44:58 test1 openvpn[1582]: ERROR: Linux route delete command failed: external program exited with error status: 2
Aug 26 04:44:58 test1 openvpn[1582]: /sbin/ip route del 128.0.0.0/1
Aug 26 04:44:58 test1 openvpn[1582]: ERROR: Linux route delete command failed: external program exited with error status: 2
Aug 26 04:44:58 test1 openvpn[1582]: Closing TUN/TAP interface
Aug 26 04:44:58 test1 openvpn[1582]: /sbin/ip addr del dev tun0 zzz.zzz.zzz.zzz/24
Aug 26 04:44:58 test1 openvpn[1582]: Linux ip addr del failed: external program exited with error status: 2
Aug 26 04:45:01 test1 openvpn[1582]: Note: Cannot ioctl TUNSETIFF tun0: Operation not permitted (errno=1)

As evident, the program executes this time, but it fails with 'error status: 2'. What does that mean?
Same question for the last error: 'Operation not permitted (errno=1)'?

[ecrist]

As evident, the program executes this time, but it fails with 'error status: 2'. What does that mean?
Same question for the last error: 'Operation not permitted (errno=1)'?

You'd have to look at the man page for the command. I'm guessing user _openvpn doesn't have permissions to change the routing table.