Multi-Server-Cluster

Q-Wulf

First off
I'm not a native english speaker, new to OpenVPN and not very versed in networking

Detailed Problem discription:
I am experimenting with OpenVPN as a way to facilitate IP Multicast for a proxmox-HA-cluster.

I am currently running 6 "small" dedicated Servers on 3 Continents. 2 in US, 2 in EU and 2 in ASIA. Each in a different physical location. each is running proxmox, I want to create a HA-Cluster for Virtual Machine fail overs.
I want to be able to substitute each of the "small" dedicated servers for "bigger" ones, as needed and/or be able to add more machines to the Proxmox Cluster.
Should one of the Machines fail, routes be unreachable or i substitute a Server, or do Maintanance on it, I want only That Server to be affected, not my hole Cluster.

I have "kinda" managed to do this using one of the Proxmox hosts and installing OpenVPN in a Virtual Machine, and then connecting the host and 2 test servers to it. It works fine , par one issue, single point of failure. If the machine with the OpenVPN server fails, my Cluster fails.

Another thing worth to mention is, that in Proxmox i need to configure the following way on each server:

127.0.0.1 localhost.localdomain localhost
10.xxx.xxx.1  vpn.host1.example.com host1 
10.xxx.xxx.2  vpn.host2.example.com host2 
10.xxx.xxx.3  vpn.host1.example.com host3
aaa.bbb.ccc.ddd hostx.expample.com hostx  # x = 1-3 for this example. and aaa.bbb.ccc.ddd = public IP.

That means i need to use static IP's assigned to the VPN-Clients, that can communicate to with each other.

Possible Solution

So i am wondering if i can do the following and if so how, or if there is better way of achieving the same result.:

Install OpenVPN as a Server in a Virtual Machine on each Physical Host.
Install OpenVPN on the Physical host as a client.
Connect the OpenVPN client on each of the Hosts to their corresponding OpenVPN-Server, with Static IP assigned to the client
"Tie" the Open-VPN-Servers on each of the Virtual Machines together, so Clients connected from different servers can access each other.

I'm trying to achieve a Mesh-Topology for the OpenVPN-Servers.

Kind regards,
Q-Wulf

maikcat

hi there,

i admire your courage for using proxmox... :ugeek:

anyway, to your questions:

So i am wondering if i can do the following and if so how, or if there is better way of achieving the same result.:

Install OpenVPN as a Server in a Virtual Machine on each Physical Host.
Install OpenVPN on the Physical host as a client.
Connect the OpenVPN client on each of the Hosts to their corresponding OpenVPN-Server, with Static IP assigned to the client
"Tie" the Open-VPN-Servers on each of the Virtual Machines together, so Clients connected from different servers can access each other.

AFAIK proxmox uses debian,why dont you installing openvpn directly on host pc?

yes you can install it on physical host as a client,
yes you can assign static ips (per certificate) to your clients (check ccd directive)

regards,

Michael.

Q-Wulf

Thanks for the Quick reply.

maikcat wrote[...]

So i am wondering if i can do the following and if so how, or if there is better way of achieving the same result.:

Install OpenVPN as a Server in a Virtual Machine on each Physical Host.
Install OpenVPN on the Physical host as a client.
Connect the OpenVPN client on each of the Hosts to their corresponding OpenVPN-Server, with Static IP assigned to the client
"Tie" the Open-VPN-Servers on each of the Virtual Machines together, so Clients connected from different servers can access each other.
[...]
AFAIK proxmox uses debian,why dont you installing openvpn directly on host pc?

Yes, Proxmox is based upon Debian. I simply like to keep different "services" separate from each other.

maikcat wrote[...]
yes you can install it on physical host as a client,
yes you can assign static ips (per certificate) to your clients (check ccd directive)
[...]

The part i dont know how to do is this one:
" 'Tie' the Open-VPN-Servers (...) together"

Lets say
Host 1's Open VPN-Server handels IP's in the 10.1.1.x net. Host 1 as a client is assigned 10.1.1.1
Host 2's Open VPN-Server handels IP's in the 10.1.2.x net. Host 1 as a client is assigned 10.1.2.2
Host 1's Open VPN-Server handels IP's in the 10.1.3.x net. Host 1 as a client is assigned 10.1.3.3
Host 1's Open VPN-Server handels IP's in the 10.1.4.x net. Host 1 as a client is assigned 10.1.4.4
Hosts are different physical machines in different Data-Centers.
How do i make it so that the Clients on 4 different OpenVPN-Servers can talk to each other.

e.g. Client 10.1.1.1 --> OpenVPN on Host 1 --> OpenVPN on Host 4 --> 10.1.4.4

Thats the part i dont know and have been unable to find a solution for, i'm probably just missing the right "buzzword" to find it in the manuals.

edit: ty for the quick reply. Almost forgot that 😃

maikcat

if you want to setup openvpn in lan-to-lan scenario you will need:

use ccd files so you can assign static ips to peers and inform openvpn about the remote subnet
(iroute statement inside ccd files)

add (using the route command) the appropriate routes for the remote subnets

enable ip forwarding on ALL client/servers

push the appropriate routes to connected clients...

one more thing,you mention MESH network....
the above is not exactly MESH type but STAR type network...
if you want MESH you will need multiple client/server configs and probably a routing protocol...(ospf?)

ps: are you using drbd with proxmox?

Michael.

Q-Wulf

ps: are you using drbd with proxmox?

On the Production environment i am currently using 4 Serverclusters (4 geographical locations) that each implement Proxmox-HA with 2 Nodes and DRBD (and as the third wheel for Cluster-Quorum). It all runs in a Vrack (something i'd like to avoid, cause of the Costs involved and being limited to a Single Datasever and Provider).

[…]
one more thing,you mention MESH network....
the above is not exactly MESH type but STAR type network...
if you want MESH you will need multiple client/server configs and probably a routing protocol...(ospf?)[…]

Since we just lost Internet connectivity here at the Office and therefor my Testing Cluster (the one utilizing a Single VPN-Server located on a dedicated Server in a Datacenter - my first Trial) lost connectivity to the 2 other Cluster Nodes (nice use case isn't it ? Image the VPN-Server was at this office, now the outside world would not be able to access any of the Machines) i had the time to make a visual for the way i image this to work (topology wise):

Big Version (1.1 MB) https://www.dropbox.com/s/t5wq0rgt8js3csw/Proxmox_Multi-VPN-Server.png

[…]and probably a routing protocol...(ospf?)[…]

i read up on ospf in relation to Openvpn and found this great guide:
http://www.vyatta4people.org/highly-available-openvpn-connection-between-two-offices/

I do not think that i will need such an elaborate configuration.
Lets say in the above Picture OpenVPN-Server 2 (10.1.2.254 - public IP 147.5.4.3), its Host System (10.1.2.2 Public IP 199.8.23.118) fail, then they are not reachable from the other Servers and that is exactly what i want. The other 4 Nodes in the Cluster can still communicate and will restart the VM's that ran on the Host that failed, or its corresponding OpenVPN-Server.

As i understand it what ospf does is make sure that packets from one server can reach another server by finding an alternative route (and the shortest of those alternates). However since Proxmox uses IP multicast, a packet send by a Proxmox Node (10.1.1.1, 10.1.2.2, 10.1.3.3, 10.1.4.4, 10.1.X.X) will always be send to EVERY Client connected on my Virtual Network (Thats why Proxmox support currently say you should not use more then 16 Servers per Cluster, cause of Bandwith congestion)

Anyways, i think using a routing protocol ontop of OpenVPN makes it more complex then it needs to be.

The questions right now are basically

What do i need to set in OpenVPN:

so that Server 10.1.1.254 knows where to find Server 10.1.2.254, 10.1.3.254, 10.1.4.254 and 10.1.X.254
Clients like 10.1.1.1 and 10.1.2.2m 10.1.3.3, 10.1.4.4, 10.1.X.X are in the same Network

The Result should be that i can add the following to the /etc/hosts of each Proxmox Node

10.1.1.1 Proxmox-Node-1
10.1.2.2 Proxmox-Node-2
10.1.3.3 Proxmox-Node-3
10.1.4.4 Proxmox-Node-4
10.1.X.X Proxmox-Node-X

I think i might need an example for a single server, from that i probably can extrapolate

Ty for getting me this far tho.

maikcat

just for the record,if you want to pass multicast traffic over openvpn you need to
use tap & bridge mode...

MESH network is a challenge..

for a simple lan-to-lan you will need to add:

server side
----------------

server.conf

route x.x.x.x 255.255.255.0 <--the remote subnet
push "route x.x.x.x 255.255.255.0" <--local subnet

also you will need to create a ccd file named after remote location cert name with the following contents:

iroute x.x.x.x 255.255.255.0 <-- remote subnet

enable ip forwarding on both ends ,reconnect and you are done...

to add more networks on server just create new ccd files with the appropriate iroutes and add route directive into server conf...

ps: in drbd are you using protocol C?

Michael.

Q-Wulf

i think I might have gotten it now.

for reference:

OpenVPN-Server configs
Okay, lets assume i'm doing the configs for

OpenVPN-Server 137.5.43.8 (resolves to host1.vpn.example.org)
OpenVPN-Server 147.5.4.3 (resolves to host2.vpn.example.org)
OpenVPN-Server 205.9.113.48 (resolves to host3.vpn.example.org)

"Server: 137.5.43.8 /etc/openvpn/server.conf"

# Local ip to listen on
local 10.1.1.254

[…]
# Use routed VPN
dev tap
[…] 
# SSL/TLS root certificate (ca), certificate
# (cert), and private key (key).
ca /etc/openvpn/easy-rsa/keys/ca.host1.vpn.example.org.crt
cert /etc/openvpn/easy-rsa/keys/host1.vpn.example.org.crt
key /etc/openvpn/easy-rsa/keys/host1.vpn.example.org.key # This file should be kept secret

[…]
# OpenVPN IP Pool for clients
# Clients will be assigned addresses from 
server 10.1.0.0 255.255.0.0
 
# Maintain a record of client <-> virtual IP address
ifconfig-pool-persist ipp.host1.vpn.example.org.txt
 


# Push routes to the clients
push "route 10.1.1.0 255.255.240.0"
push "route 10.1.2.0 255.255.240.0"
push "route 10.1.3.0 255.255.240.0"

"Server: 137.5.43.8 /etc/openvpn/ipp.host1.vpn.example.org.txt"

node1.user, 10.1.1.1

"Client: 137.5.43.8 /etc/openvpn/ccd/node1.user"

iroute 10.1.2.0 255.255.255.0
iroute 10.1.3.0 255.255.255.0

"Server: 147.5.4.3 /etc/openvpn/server.conf"

# Local ip to listen on
local 10.1.2.254

[…]
# Use routed VPN
dev tap
[…] 
# SSL/TLS root certificate (ca), certificate
# (cert), and private key (key).
ca /etc/openvpn/easy-rsa/keys/ca.host2.vpn.example.org.crt
cert /etc/openvpn/easy-rsa/keys/host2.vpn.example.org.crt
key /etc/openvpn/easy-rsa/keys/host2.vpn.example.org.key # This file should be kept secret

[…]
# OpenVPN IP Pool for clients

# Clients will be assigned addresses from

server 10.1.0.0 255.255.0.0
 
# Maintain a record of client <-> virtual IP address
ifconfig-pool-persist ipp.host2.vpn.example.org.txt
 
# Push routes to the clients
push "route 10.1.2.0 255.255.240.0"
push "route 10.1.1.0 255.255.240.0"
push "route 10.1.3.0 255.255.240.0"

"Server: 147.5.4.3 /etc/openvpn/ipp.host2.vpn.example.org.txt"

node2.user, 10.1.2.2

"Client: 147.5.4.3 /etc/openvpn/ccd/node2.user"

iroute 10.1.1.0 255.255.255.0
iroute 10.1.3.0 255.255.255.0

Server: 205.9.113.48 /etc/openvpn/server.conf"

# Local ip to listen on
local 10.1.3.254

[…]
# Use routed VPN
dev tap
[…] 
# SSL/TLS root certificate (ca), certificate
# (cert), and private key (key).
ca /etc/openvpn/easy-rsa/keys/ca.host3.vpn.example.org.crt
cert /etc/openvpn/easy-rsa/keys/host3.vpn.example.org.crt
key /etc/openvpn/easy-rsa/keys/host3.vpn.example.org.key # This file should be kept secret

[…]
# OpenVPN IP Pool for clients
# Clients will be assigned addresses from 
server 10.1.0.0 255.255.0.0
 
# Maintain a record of client <-> virtual IP address
ifconfig-pool-persist ipp.host3.vpn.example.org.txt
 
# Push routes to the clients
push "route 10.1.3.0 255.255.240.0"
push "route 10.1.1.0 255.255.240.0"
push "route 10.1.2.0 255.255.240.0"

"Server: 205.9.113.48 /etc/openvpn/ipp.host3.vpn.example.org.txt"

node3.user, 10.1.3.3

then on the /etc/hosts for every OpenVPN-Server i add the following, so that each server knows where he can find the other one.

10.1.1.1  host1.vpn.example.org
10.1.2.2  host2.vpn.example.org
10.1.3.3  host3.vpn.example.org

OpenVPN-Server 137.5.43.8 (resolves to host1.vpn.example.org)
OpenVPN-Server 147.5.4.3 (resolves to host2.vpn.example.org)
OpenVPN-Server 205.9.113.48 (resolves to host3.vpn.example.org)

You mean like this ?

maikcat

some things i noticed...

# Local ip to listen on
local 10.1.1.254
# Use routed VPN
dev tap
ca /etc/openvpn/easy-rsa/keys/ca.host1.vpn.example.org.crt
cert /etc/openvpn/easy-rsa/keys/host1.vpn.example.org.crt
key /etc/openvpn/easy-rsa/keys/host1.vpn.example.org.key # This file should be kept secret
server 10.1.0.0 255.255.0.0
# Maintain a record of client <-> virtual IP address
ifconfig-pool-persist ipp.host1.vpn.example.org.txt
# Push routes to the clients
push "route 10.1.1.0 255.255.240.0"
push "route 10.1.2.0 255.255.240.0"
push "route 10.1.3.0 255.255.240.0"

you cant use the same ip range on lan + vpn...
also you cant use /16 mask on server while remote networks overlap it...

assuming 10.1.x.0 networks are used on THEIR LANs then you can use 10.2.x.0 for virtual interfaces,
also pushing 3 routes with 255.255.240.0 mask is unecessary...

lets asume that your first server 137.5.43.8 uses this config:

# Local ip to listen on
local 10.1.1.254

[…]
# Use routed VPN
dev tap
ca /etc/openvpn/easy-rsa/keys/ca.host1.vpn.example.org.crt
cert /etc/openvpn/easy-rsa/keys/host1.vpn.example.org.crt
key /etc/openvpn/easy-rsa/keys/host1.vpn.example.org.key # This file should be kept secret
server 10.2.1.0 255.255.255.0
# Maintain a record of client <-> virtual IP address
ifconfig-pool-persist ipp.host1.vpn.example.org.txt
push "route 10.1.1.0 255.255.255.0"
route 10.1.2.0 255.255.255.0

ccd file used for 147.5.4.3 client

iroute 10.1.2.0 255.255.255.0

with the above you will have lan-to-lan between 147.5.4.3 & 137.5.43.8

i repeat..MESH IS complicated...

Michael.

Q-Wulf

Okay, finally found the time to test this on my 3 test-machines;
There have been a couple more "mistakes" in the configs i posted.

As soon as the dh dh4096.pem is done calculating - on the very slow machine - I will post the working configs.

with your above post we are basically connecting the OpenVPn-Server1 and the OpenVPN-Server2, by making openVPN-Server2 a Client of OpenVPN-Server1;
Thus enabling all clients behind both servers to talk to each other.

Can i simultaneously make OpenVPN-Server1 a Client of OpenVPN-Server2 and vice versa ??

Cause if that works i'll do something like this:

Server:			First Client		Second Client
OpenVPN-Server-1	OpenVPN-Server-2	OpenVPN-Server-3
OpenVPN-Server-2	OpenVPN-Server-1	OpenVPN-Server-3
OpenVPN-Server-3	OpenVPN-Server-2	OpenVPN-Server-1

And that should be able to create a true mesh topology and not make me cry should one of the servers die.

I assume i have to use different virtual Subnets for above example:

Server (Virt. Ip)	First CLient		Second Client
VPNS-1 (10.1.1.0)	VPNS-2 (10.1.1.20)	VPNS-3 (10.1.1.30)
VPNS-2 (10.1.2.0)	VPNS-1 (10.1.1.10)	VPNS-3 (10.1.1.30)
VPNS-3 (10.1.3.0)	VPNS-2 (10.1.1.20)	VPNS-1 (10.1.1.10)

Q-Wulf

I made a mistake when posting the "mesh" query, its obviously supposed to be like this:

Q-Wulf wrote I assume i have to use different virtual Subnets for above example:

Server (Virt. Ip)	ProxmoxNode		1st Server as CLient	2nd Server as Client
VPNS-1 (10.1.1.1)	Node-1 (10.1.1.2)	VPNS-2 (10.1.1.20)	VPNS-3 (10.1.1.30)
VPNS-2 (10.1.2.1)	Node-2 (10.1.2.2)	VPNS-1 (10.1.2.10)	VPNS-3 (10.1.2.30)
VPNS-3 (10.1.3.1)	Node-3 (10.1.3.2)	VPNS-2 (10.1.3.20)	VPNS-1 (10.1.3.10)

found this on the untangle forums:
http://forums.untangle.com/tips-tricks/14426-how-configure-linux-openvpn-client-connect-multiple-openvpn-servers.html

basically on the client instead of

 dev tun

They do

 dev tun1

so in my example i could probaly do

 dev tap1
dev tap2
dev tap3

for the different client .conf's

Q-Wulf

While my dh4096 is still generating (atom 1,2 GHZ, single thread) i just realized that there is solution out there, that is WAYYYYY easier.

Assume i "run 3 clients on each proxmox node simultaniously using tap1, tap2, tap3 (check above)

Client		Client_IP		Server			Server_IP
Proxmox-Node-1	10.1.1.10	-->	OpenVPn-Server-1 	10.1.1.1
Proxmox-Node-1	10.1.2.10	-->	OpenVPn-Server-2 	10.1.2.1
Proxmox-Node-1	10.1.3.10	-->	OpenVPN-Server-3	10.1.3.1

Proxmox-Node-2	10.1.1.20	-->	OpenVPn-Server-1 	10.1.1.1
Proxmox-Node-2	10.1.2.20	-->	OpenVPn-Server-2 	10.1.2.1
Proxmox-Node-2	10.1.3.20	-->	OpenVPN-Server-3	10.1.3.1

Proxmox-Node-3	10.1.1.30	-->	OpenVPn-Server-1 	10.1.1.1
Proxmox-Node-3	10.1.2.30	-->	OpenVPn-Server-2 	10.1.2.1
Proxmox-Node-3	10.1.3.30	-->	OpenVPN-Server-3	10.1.3.1

then in the /etc/hosts file of each Proxmox-Node i set doublicate entries like this:

For Proxmox-Node1:

10.1.1.20	proxmo-node-2
10.1.2.20	proxmo-node-2
10.1.3.20	proxmo-node-2

10.1.1.30	proxmox-node-3
10.1.1.30	proxmox-node-3
10.1.1.30	proxmox-node-3

For Proxmox-Node2:

10.1.1.10	proxmo-node-1
10.1.2.10	proxmo-node-1
10.1.3.10	proxmo-node-1

10.1.1.30	proxmox-node-3
10.1.1.30	proxmox-node-3
10.1.1.30	proxmox-node-3

For Proxmox-Node3:

10.1.1.20	proxmo-node-2
10.1.2.20	proxmo-node-2
10.1.3.20	proxmo-node-2

10.1.1.10	proxmox-node-3
10.1.1.10	proxmox-node-3
10.1.1.10	proxmox-node-3

Now it should not matter if one proxmox-node or its corresponding OpenVPN-Server goes down. The unaffected nodes can still "talk" by using the unaffected OpenVPn-Servers.

Upside is: I don't have to worry about making subnets behind the "clients" routeable and i don't have to worry about routing loops either.

Q-Wulf

As promised the working configs.
IP's and FQDN have been anonymized

Reference:

FQDN				Remote_IP
Proxmox-Node-1.example.org	178.4.9.2
OpenVPN-Server-1.example.org	154.66.31.62
Proxmox-Node-2.example.org	5.115.83.175
OpenVPN-Server-2.example.org	186.5.64.120
Proxmox-Node-3.example.org	191.21.16.18
OpenVPN-Server-3.example.org	203.1.5.7

OpenVPN Server Configs:

OpenVPN-Server-1:

cat /etc/openvpn/server.conf

server.conf
local 154.66.31.62
port 1194
proto udp
dev tap
ca ca.crt
cert server.crt
key server.key  # This file should be kept secret
dh dh4096.pem
server 10.1.1.0 255.255.255.0
ifconfig-pool-persist /etc/openvpn/ipp.txt
client-config-dir ccd
route 10.1.1.0 255.255.255.
client-to-client
keepalive 10 120
tls-auth ta.key 0 # This file is secret
comp-lzo
max-clients 5
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
log-append  openvpn.log
verb 3

cat /etc/openvpn/ipp.txt

ProxmoxNode1OpenvpnServer1,10.1.1.10
ProxmoxNode2OpenvpnServer1,10.1.1.20
ProxmoxNode3OpenvpnServer1,10.1.1.30

ls -1 /etc/openvpn/ccd/

ProxmoxNode1OpenvpnServer1.user
ProxmoxNode2OpenvpnServer1.user
ProxmoxNode3OpenvpnServer1.user

cat /etc/openvpn/ccd/ProxmoxNode1OpenvpnServer1.user

ifconfig-push 10.1.1.1 10.1.1.254

cat /etc/openvpn/ccd/ProxmoxNode2OpenvpnServer1.user

ifconfig-push 10.1.1.1 10.1.1.254

cat /etc/openvpn/ccd/ProxmoxNode3OpenvpnServer1.user

ifconfig-push 10.1.1.1 10.1.1.254

OpenVPN-Server-2:

cat /etc/openvpn/server.conf

local 186.5.64.120
port 1194
proto udp
dev tap
ca ca.crt
cert server.crt
key server.key  # This file should be kept secret
dh dh4096.pem
server 10.1.2.0 255.255.255.0
ifconfig-pool-persist /etc/openvpn/ipp.txt
client-config-dir ccd
route 10.1.2.0 255.255.255.
client-to-client
keepalive 10 120
tls-auth ta.key 0 # This file is secret
comp-lzo
max-clients 5
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
log-append  openvpn.log
verb 3

cat /etc/openvpn/ipp.txt

ProxmoxNode1OpenvpnServer2,10.1.2.10
ProxmoxNode2OpenvpnServer2,10.1.2.20
ProxmoxNode3OpenvpnServer2,10.1.2.30

ls -1 /etc/openvpn/ccd/

ProxmoxNode1OpenvpnServer2.user
ProxmoxNode2OpenvpnServer2.user
ProxmoxNode3OpenvpnServer2.user

cat /etc/openvpn/ccd/ProxmoxNode1OpenvpnServer2.user

ifconfig-push 10.1.2.1 10.1.2.254

cat /etc/openvpn/ccd/ProxmoxNode2OpenvpnServer2.user

ifconfig-push 10.1.2.1 10.1.2.254

cat /etc/openvpn/ccd/ProxmoxNode3OpenvpnServer2.user

ifconfig-push 10.1.2.1 10.1.2.254

OpenVPN-Server-3:

cat /etc/openvpn/server.conf

local 203.1.5.7
port 1194
proto udp
dev tap
ca ca.crt
cert server.crt
key server.key  # This file should be kept secret
dh dh4096.pem
server 10.1.3.0 255.255.255.0
ifconfig-pool-persist /etc/openvpn/ipp.txt
client-config-dir ccd
route 10.1.3.0 255.255.255.
client-to-client
keepalive 10 120
tls-auth ta.key 0 # This file is secret
comp-lzo
max-clients 5
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
log-append  openvpn.log
verb 3

cat /etc/openvpn/ipp.txt

ProxmoxNode1OpenvpnServer3,10.1.3.10
ProxmoxNode2OpenvpnServer3,10.1.3.20
ProxmoxNode3OpenvpnServer3,10.1.3.30

ls -1 /etc/openvpn/ccd/

ProxmoxNode1OpenvpnServer3.user
ProxmoxNode2OpenvpnServer3.user
ProxmoxNode3OpenvpnServer3.user

cat /etc/openvpn/ccd/ProxmoxNode1OpenvpnServer3.user

ifconfig-push 10.1.3.1 10.1.3.254

cat /etc/openvpn/ccd/ProxmoxNode2OpenvpnServer3.user

ifconfig-push 10.1.3.1 10.1.3.254

cat /etc/openvpn/ccd/ProxmoxNode3OpenvpnServer3.user

ifconfig-push 10.1.3.1 10.1.3.254

Proxmox-Node-Configs (OpenVPn-Client):

Proxmox-Node-1:

cat /etc/openvpn/ProxmoxNode1OpenvpnServer1.ovpn

client
dev tap1
proto udp
remote OpenVPN-Server-1.example.org 1194
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
ca OpenvpnServer1.ca.crt
cert ProxmoxNode1OpenvpnServer1.crt
key ProxmoxNode1OpenvpnServer1.key
ns-cert-type server
tls-auth OpenvpnServer1.ta.key 1
comp-lzo
verb 3

cat /etc/openvpn/ProxmoxNode1OpenvpnServer2.ovpn

client
dev tap2
proto udp
remote OpenVPN-Server-2.example.org 1194
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
ca OpenvpnServer2.ca.crt
cert ProxmoxNode1OpenvpnServer2.crt
key ProxmoxNode1OpenvpnServer2.key
ns-cert-type server
tls-auth OpenvpnServer2.ta.key 1
comp-lzo
verb 3

cat /etc/openvpn/ProxmoxNode1OpenvpnServer3.ovpn

client
dev tap3
proto udp
remote OpenVPN-Server-3.example.org 1194
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
ca OpenvpnServer3.ca.crt
cert ProxmoxNode1OpenvpnServer3.crt
key ProxmoxNode1OpenvpnServer3.key
ns-cert-type server
tls-auth OpenvpnServer3.ta.key 1
comp-lzo
verb 3

Start clients with

openvpn --config /etc/openvpn/ProxmoxNode1OpenvpnServer1.ovpn
openvpn --config /etc/openvpn/ProxmoxNode1OpenvpnServer2.ovpn
openvpn --config /etc/openvpn/ProxmoxNode1OpenvpnServer3.ovpn

cat /etc/hosts

127.0.0.1      	localhost.localdomain 		localhost
10.1.1.10 	node1.vpn.local 		node1 	pvelocalhost
10.1.2.10 	node1.vpn.local 		node1 	pvelocalhost
10.1.3.10 	node1.vpn.local 		node1 	pvelocalhost
10.1.1.20	node2.vpn.local			node2
10.1.2.20	node2.vpn.local			node2
10.1.2.20	node2.vpn.local			node2
10.1.1.30	node3.vpn.local			node3
10.1.2.30	node3.vpn.local			node3
10.1.3.30	node3.vpn.local			node3
178.4.9.2	Proxmox-Node-1.example.org	node1

Proxmox-Node-2:

cat /etc/openvpn/ProxmoxNode2OpenvpnServer1.ovpn

client
dev tap1
proto udp
remote OpenVPN-Server-1.example.org 1194
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
ca OpenvpnServer1.ca.crt
cert ProxmoxNode2OpenvpnServer1.crt
key ProxmoxNode2OpenvpnServer1.key
ns-cert-type server
tls-auth OpenvpnServer1.ta.key 1
comp-lzo
verb 3

cat /etc/openvpn/ProxmoxNode2OpenvpnServer2.ovpn

client
dev tap2
proto udp
remote OpenVPN-Server-2.example.org 1194
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
ca OpenvpnServer1.ca.crt
cert ProxmoxNode2OpenvpnServer2.crt
key ProxmoxNode2OpenvpnServer2.key
ns-cert-type server
tls-auth OpenvpnServer1.ta.key 1
comp-lzo
verb 3

cat /etc/openvpn/ProxmoxNode2OpenvpnServer3.ovpn

client
dev tap3
proto udp
remote OpenVPN-Server-3.example.org 1194
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
ca OpenvpnServer3.ca.crt
cert ProxmoxNode2OpenvpnServer3.crt
key ProxmoxNode2OpenvpnServer3.key
ns-cert-type server
tls-auth OpenvpnServer3.ta.key 1
comp-lzo
verb 3

Start clients with

openvpn --config /etc/openvpn/ProxmoxNode2OpenvpnServer1.ovpn
openvpn --config /etc/openvpn/ProxmoxNode2OpenvpnServer2.ovpn
openvpn --config /etc/openvpn/ProxmoxNode2OpenvpnServer3.ovpn

cat /etc/hosts

127.0.0.1      	localhost.localdomain 		localhost
10.1.1.10 	node1.vpn.local 		node1 	
10.1.2.10 	node1.vpn.local 		node1 	
10.1.3.10 	node1.vpn.local 		node1 	
10.1.1.20	node2.vpn.local			node2	pvelocalhost
10.1.2.20	node2.vpn.local			node2	pvelocalhost
10.1.2.20	node2.vpn.local			node2	pvelocalhost
10.1.1.30	node3.vpn.local			node3
10.1.2.30	node3.vpn.local			node3
10.1.3.30	node3.vpn.local			node3
5.115.83.175	Proxmox-Node-2.example.org	node2

Proxmox-Node-3:

cat /etc/openvpn/ProxmoxNode3OpenvpnServer1.ovpn

client
dev tap1
proto udp
remote OpenVPN-Server-1.example.org 1194
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
ca OpenvpnServer1.ca.crt
cert ProxmoxNode3OpenvpnServer1.crt
key ProxmoxNode3OpenvpnServer1.key
ns-cert-type server
tls-auth OpenvpnServer1.ta.key 1
comp-lzo
verb 3

cat /etc/openvpn/ProxmoxNode3OpenvpnServer2.ovpn

client
dev tap2
proto udp
remote OpenVPN-Server-2.example.org 1194
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
ca OpenvpnServer1.ca.crt
cert ProxmoxNode3OpenvpnServer2.crt
key ProxmoxNode3OpenvpnServer2.key
ns-cert-type server
tls-auth OpenvpnServer1.ta.key 1
comp-lzo
verb 3

cat /etc/openvpn/ProxmoxNode3OpenvpnServer3.ovpn

client
dev tap3
proto udp
remote OpenVPN-Server-3.example.org 1194
resolv-retry infinite
nobind
user nobody
group nogroup
persist-key
persist-tun
ca OpenvpnServer3.ca.crt
cert ProxmoxNode3OpenvpnServer3.crt
key ProxmoxNode3OpenvpnServer3.key
ns-cert-type server
tls-auth OpenvpnServer3.ta.key 1
comp-lzo
verb 3

Start clients with

openvpn --config /etc/openvpn/ProxmoxNode2OpenvpnServer1.ovpn
openvpn --config /etc/openvpn/ProxmoxNode2OpenvpnServer2.ovpn
openvpn --config /etc/openvpn/ProxmoxNode2OpenvpnServer3.ovpn

cat /etc/hosts

127.0.0.1      	localhost.localdomain 		localhost
10.1.1.10 	node1.vpn.local 		node1 	
10.1.2.10 	node1.vpn.local 		node1 	
10.1.3.10 	node1.vpn.local 		node1 	
10.1.1.20	node2.vpn.local			node2	
10.1.2.20	node2.vpn.local			node2	
10.1.2.20	node2.vpn.local			node2	
10.1.1.30	node3.vpn.local			node3	pvelocalhost
10.1.2.30	node3.vpn.local			node3	pvelocalhost
10.1.3.30	node3.vpn.local			node3	pvelocalhost
191.21.16.18	Proxmox-Node-3.example.org	node3

afterwards i just added the nodes to Proxmox as you usually do and it worked. I now can reboot one OpenVPN-Server or one Proxmox-Node and the Cluster is still able to communicate as its suppsed to. No more single point of Failure caused by OpenVPN 😃

maikcat wrote[...]
i repeat..MESH IS complicated...

Michael.

no Kidding. Thats just 3 Nodes. Imho Proxmox can handle up to 16 Nodes *gg*