Why does my 2nd instance stop almost immediately?

rseiler

I've had a single instance TCP 443 server setup for a while. I wanted to add UDP as well, so I created a differently named .conf and in it changed the port, proto, a couple of the push lines, the server line, and the log location. Did I miss something?

Upon restarting openvpn, "top" shows the second openvpn disappearing in a second or two, leaving only the original TCP one. "netstat -lnp" shows the second openvpn briefly on 4430, listed with its own PID, but the line then disappears since the process is gone.

Why is the second instance stopping/crashing?

Here's my tcp conf file. The udp conf file is the same except for the changes noted in the comments (for space reasons I'm not showing both separately here).

Thanks!

dev tun
port 443    #port 4430
proto tcp   #proto udp
ifconfig-pool-persist ipp.txt
push "dhcp-option DNS 10.8.0.1"    #push "dhcp-option DNS 172.16.0.1"
push "dhcp-option DNS 208.67.222.222"
push "redirect-gateway def1"
push "route 10.8.0.0 255.255.255.0"   #push "route 172.16.0.0 255.255.255.0"
server 10.8.0.0 255.255.255.0   #server 172.16.0.0 255.255.255.0
ca ca.crt
cert server.crt
cipher AES-128-CBC
dh dh1024.pem
key server.key
comp-lzo
daemon
duplicate-cn
group daemon
keepalive 1800 3600
mute 20
ping-timer-rem
log /var/log/openvpn.log   #log /var/log/openvpn_udp.log
verb 3
persist-key
persist-tun

Here are the logs for both:

Tue Feb 26 10:11:42 2013 OpenVPN 2.1.3 x86_64-pc-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [MH] [PF_INET6] [eurephia] built on Feb 21 2012
Tue Feb 26 10:11:42 2013 WARNING: --ifconfig-pool-persist will not work with --duplicate-cn
Tue Feb 26 10:11:42 2013 NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
Tue Feb 26 10:11:42 2013 Diffie-Hellman initialized with 1024 bit key
Tue Feb 26 10:11:42 2013 /usr/bin/openssl-vulnkey -q -b 1024 -m <modulus omitted>
Tue Feb 26 10:11:42 2013 TLS-Auth MTU parms [ L:1560 D:140 EF:40 EB:0 ET:0 EL:0 ]
Tue Feb 26 10:11:42 2013 Socket Buffers: R=[87380->131072] S=[16384->131072]
Tue Feb 26 10:11:42 2013 ROUTE: default_gateway=UNDEF
Tue Feb 26 10:11:42 2013 TUN/TAP device tun0 opened
Tue Feb 26 10:11:42 2013 TUN/TAP TX queue length set to 100
Tue Feb 26 10:11:42 2013 /sbin/ifconfig tun0 10.8.0.1 pointopoint 10.8.0.2 mtu 1500
Tue Feb 26 10:11:42 2013 /sbin/route add -net 10.8.0.0 netmask 255.255.255.0 gw 10.8.0.2
Tue Feb 26 10:11:42 2013 Data Channel MTU parms [ L:1560 D:1450 EF:60 EB:135 ET:0 EL:0 AF:3/1 ]
Tue Feb 26 10:11:42 2013 GID set to daemon
Tue Feb 26 10:11:42 2013 Listening for incoming TCP connection on [undef]
Tue Feb 26 10:11:42 2013 TCPv4_SERVER link local (bound): [undef]
Tue Feb 26 10:11:42 2013 TCPv4_SERVER link remote: [undef]
Tue Feb 26 10:11:42 2013 MULTI: multi_init called, r=256 v=256
Tue Feb 26 10:11:42 2013 IFCONFIG POOL: base=10.8.0.4 size=62
Tue Feb 26 10:11:42 2013 IFCONFIG POOL LIST
Tue Feb 26 10:11:42 2013 MULTI: TCP INIT maxclients=1024 maxevents=1028
Tue Feb 26 10:11:42 2013 Initialization Sequence Completed

Tue Feb 26 10:11:42 2013 OpenVPN 2.1.3 x86_64-pc-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [MH] [PF_INET6] [eurephia] built on Feb 21 2012
Tue Feb 26 10:11:42 2013 WARNING: --ifconfig-pool-persist will not work with --duplicate-cn
Tue Feb 26 10:11:42 2013 NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
Tue Feb 26 10:11:42 2013 Diffie-Hellman initialized with 1024 bit key
Tue Feb 26 10:11:42 2013 /usr/bin/openssl-vulnkey -q -b 1024 -m <modulus omitted>
Tue Feb 26 10:11:42 2013 TLS-Auth MTU parms [ L:1558 D:138 EF:38 EB:0 ET:0 EL:0 ]
Tue Feb 26 10:11:42 2013 Socket Buffers: R=[245760->131072] S=[245760->131072]
Tue Feb 26 10:11:42 2013 ROUTE: default_gateway=UNDEF
Tue Feb 26 10:11:42 2013 TUN/TAP device tun1 opened
Tue Feb 26 10:11:42 2013 TUN/TAP TX queue length set to 100
Tue Feb 26 10:11:42 2013 /sbin/ifconfig tun1 172.16.0.1 pointopoint 172.16.0.2 mtu 1500
Tue Feb 26 10:11:42 2013 /sbin/route add -net 172.16.0.0 netmask 255.255.255.0 gw 172.16.0.2
Tue Feb 26 10:11:42 2013 Data Channel MTU parms [ L:1558 D:1450 EF:58 EB:135 ET:0 EL:0 AF:3/1 ]
Tue Feb 26 10:11:42 2013 GID set to daemon
Tue Feb 26 10:11:42 2013 UDPv4 link local (bound): [undef]
Tue Feb 26 10:11:42 2013 UDPv4 link remote: [undef]
Tue Feb 26 10:11:42 2013 MULTI: multi_init called, r=256 v=256
Tue Feb 26 10:11:42 2013 IFCONFIG POOL: base=172.16.0.4 size=62
Tue Feb 26 10:11:42 2013 IFCONFIG POOL LIST
Tue Feb 26 10:11:42 2013 Initialization Sequence Completed

maikcat

hi there,

Why is the second instance stopping/crashing?

when 2 instance crashes,logs are remaining the same???

which distro are you using?

Michael.

rseiler

maikcat wrotewhen 2 instance crashes,logs are remaining the same???

which distro are you using?

Well, they're different in the way that they should be, right? For a while, I thought that the UDP one not having a "Listening" line might be notable, but it's not, since that's normal for UDP processes in Linux.

I guess if the OpenVPN logs don't show anything off means that it's something about the system that's terminating it, but that's just a guess. I'm not sure in what log to look for such a thing.

This is Debian 6 64-bit (2.6.32-042stab072.10).

rseiler

I tried using just the udp conf file to see how it would work alone. Guess what? While it takes longer for the process to disappear (perhaps 10-15 seconds), it does disappear!

That's very interesting to me, since this isn't a two-process issue but rather specifically a udp issue. I even tried UDP 443, thinking that there might be some magic in the port.

Until I or someone figures this out, I've retreated to my previous TCP 443 configuration, which does stick.

maikcat

hi there,

if you monitor the processes by using ps or top , what do you see?

system logs say anything??

also try to start openvpn manually like

openvpn --config udpconfig.conf

Michael.

rseiler

I looked in /var/log for anything new that relates, but I didn't see a single thing related (I looked in xconsole.log, syslog, and auth.log).

Next, I tried stopping openvpn and running just the udp instance in the way that you described. It runs with no message in the console. Openvpn loads in memory, Netstat shows it as udp and on the right port. Then, as expected, it dies around 15 seconds later.

I have found, however, that once in a while when I'm only running a udp instance, that it will stick around for a minute or even several minutes. For a while, I was fooled into thinking that udp 443 (alone) actually worked for good, but I was wrong.

"netstat -lptu" shows no port conflicts.

I found in the manual that the port doesn't actually need to be different when one instance is tcp and one is udp. That's good to know, but it doesn't help me here.

So, in summary, I have absolutely no idea what's going on, and I've so far found no way to make udp last. tcp does last every single time.

If you're curious what else is in memory (openvpn was not running at the time):

  PID TTY          TIME CMD
    1 ?        00:00:00 init
    2 ?        00:00:00 kthreadd/10905
    3 ?        00:00:00 khelper/10905
  436 ?        00:00:00 syslogd
  460 ?        00:00:00 saslauthd
  462 ?        00:00:00 saslauthd
  476 ?        00:00:00 apache2
  479 ?        00:00:00 apache2
  504 ?        00:00:00 cron
  548 ?        00:00:00 sshd
  554 ?        00:00:00 xinetd
  590 ?        00:00:00 sshd
  592 ?        00:00:00 sftp-server
  593 pts/1    00:00:00 bash
  598 ?        00:00:00 sendmail-mta
 1163 pts/1    00:00:00 ps

maikcat

can you try adding the pid directive so openvpn configs create different pids...

Michael.

rseiler

I just realized when looking at one of my earlier posts here that I'm not running the latest version, or anywhere near, actually (v2.1.3). I found the page here explaining why that is and how to go about upgrading (it looks easy enough). Might that be a good idea? The more I play with this, the more it feels like a bug. The fact that I didn't have the latest version caught me by surprise, as I just installed earlier this month.
https://community.openvpn.net/openvpn/wiki/OpenvpnSoftwareRepos

This one?

--writepid file
Write OpenVPN's main process ID to file.

If so, I'll try dropping that into the conf, though I don't understand what writing it to a file would do. The processes would have to get separate pid's even without that, right? Plus, clashing pid's wouldn't explain why it happens with only one udp openvpn (though sometimes tonight it sticks around for long stretches with udp 443 alone, to the point that I'm tempted to say that it works like that--until you change the port or add tcp, anyway).

Without the above command, here's what it looks like right now just before the udp one disappears (different pids). In this run, I have them both on 443.

tcp        0      0 127.0.0.1:587           0.0.0.0:*               LISTEN      598/sendmail: MTA:
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      476/apache2
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      548/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      598/sendmail: MTA:
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      1567/openvpn
tcp6       0      0 :::22                   :::*                    LISTEN      548/sshd
udp        0      0 0.0.0.0:443             0.0.0.0:*                           1574/openvpn

maikcat

FYI 2.1.3 is ok....even 2.1.1 works ok with me...

although i am not debian guru (btw do you have selinux enabled?) in centos/redhat
multiple instances on same port/different proto (like yours setup) works ok

if you choose a port greater than 1024 for both , same behavior?

my mistake ,its writepid not pid.. :oops:

system logs show nothing??

Michael.

rseiler

The "getenforce" command doesn't exist, so to my knowledge Selinux, which is an add-on for Debian, is not active/installed.

A port over 1024 makes no difference.

On the writepid angle, I noticed that "ps ax" shows this as the default command in my normal tcp 443-only config:

1721 ?  Ss 0:02 /usr/sbin/openvpn --writepid /var/run/openvpn.server.pid --cd /etc/openvpn --config /etc/openvpn/server.conf

openvpn.server.pid contains the expected pid.

When I add my udp conf and restart openvpn, I see:

3679 ? Ss 0:00 /usr/sbin/openvpn --writepid /var/run/openvpn.server.pid --cd /etc/openvpn --config /etc/openvpn/server.conf
3686 ? Ss 0:00 /usr/sbin/openvpn --writepid /var/run/openvpn.serverudp.pid --cd /etc/openvpn --config /etc/openvpn/serverudp.conf

Each of the pid files is written and updated correctly, fwiw.

On /var/logs (is there another place to look?), it's not that there's nothing, but nothing of interest. The OpenVPN logs are still the same as above, but of the 3 other files with new timestamps and not beginning with "mail," I show this on or near the time of a process disappearance:

syslog:

Feb 28 08:00:01 npv /USR/SBIN/CRON[3653]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Feb 28 08:00:03 npv sm-msp-queue[3669]: My unqualified host name (npv) unknown; sleeping for retry
Feb 28 08:01:03 npv sm-msp-queue[3669]: unable to qualify my own domain name (npv) -- using short name

auth.log:

Feb 28 08:00:01 npv CRON[3652]: pam_unix(cron:session): session opened for user smmsp by (uid=0)
Feb 28 08:01:03 npv CRON[3652]: pam_unix(cron:session): session closed for user smmsp

xconsole.log:

Feb 28 08:00:01 npv CRON[3652]: pam_unix(cron:session): session opened for user smmsp by (uid=0)
Feb 28 08:00:01 npv /USR/SBIN/CRON[3653]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Feb 28 08:00:03 npv sm-msp-queue[3669]: My unqualified host name (npv) unknown; sleeping for retry
Feb 28 08:01:03 npv sm-msp-queue[3669]: unable to qualify my own domain name (npv) -- using short name
Feb 28 08:01:03 npv CRON[3652]: pam_unix(cron:session): session closed for user smmsp

rseiler

Was it important to mention that this exists on OpenVZ? I'm beginning to think it might be.

Out of curiosity, I fired up a Debian 6 in my local VirtualBox. Despite this Debian having far more things running and actually a good deal less free memory, the same OpenVPN configuration runs two instances consistently. I've not seen the second process disappear.

I've tried both the default 2.1.3 and now 2.3 on both platforms, so that doesn't matter.

What I think does matter is OpenVZ, though I don't know how yet. I do have the option to start over with Ubuntu, which I might try, but I expect to see the same thing happen.

rseiler

Interesting: I tried Access Server instead on the same system. By default, it runs a tcp process and one for udp.

The problem doesn't happen with Access Server.