LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Business solution to host your own OpenVPN server with web management interface and bundled clients.
Post Reply
User avatar
Danran
OpenVPN User
Posts: 28
Joined: Tue Jun 29, 2021 9:21 am

LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by Danran » Fri Jul 09, 2021 12:04 am

**ENVIRONMENT:**

I have successfully set up a LEMP email/webserver running off of my home IP address, with my domains dns pointing to my home IP. My server is behind a firewall router, which is behind my ISP router. I believe this would be called a double NAT setup. To get all of the email/web server ports working, I had to forward port 443, port 80, and the corresponding email server ports, from my 'ISP router' to my 'firewall router', then, forward those same ports from my 'firewall router' to my 'LEMP Webserver'.

**THE PROBLEM:**

My issues is that my home IP is a not a static IP, so I cannot set the proper Reverse DNS and PTR records for the email server. In an attempt to solve this, I have created a cloud AWS Openvpn Access Server EC2 Instance, and have assigned it an elastic IP, which now has the proper PTR and reverse DNS records for my server.

**THE GOAL:**

The idea, was to tunnel the webserver as a client to the AWS EC2 Openvpn Access Server instance for the sole purpose of using and AWS IP address, DNS address, and setting the proper PTR Records on amazons elastic IP's settings. The desired outcome for all of this, would be that my dns records for my website www.mywebsite.com point to Amazon's IP address, instead of my home address, resulting in my websites ip address being that of Amazon's IP, as opposed to my Home IP.

**STEPS I HAVE TAKEN:**

1. So far, I have installed openvpn client on my webserver, and successfully tunneled it to amazons AWS EC2 OpenVPN Access Server, using openvpnclient on the webserver/email server. There are currently no DNS leaks on my webserver, and my public ip address and DNS address on the web server both point to Amazon's corresponding public IP and DNS.
2. I have changed my DNS records on domain registrars website, to point to the newly created AMAZON OpenvpnAS Public IP address.
3. I have opened all webserver (443,80) ports and email server (25,993,etc.) ports using UFW on the EC2 Instance.

**THE QUESTION:**

With all of the above settings set, My webserver/email server is still innaccessable when going to www.mywebsite.com. Now that my my server is being successfully tunneled as a client to the OpenVPNAS, and the DNS records are pointing to the EC2 IP address,

1. WHAT ELSE MUST I DO IN ORDER TO GET MY EMAIL SERVER/WEBSERVER TO PROPERLY RESOLVE THROUGH THE OPENVPN TUNNEL?

2. Should I be forwarding ports from the ec2 instance public IP to the vpn client, and if so, how exactly do I do such a thing on an openvpnas instance? The command line instructions to use DMZ forwarding errors out. So what how exactly can I forward the aws public ip and ports 443 and 80 to the client vpn webserver?

chilinux
OpenVPN Power User
Posts: 156
Joined: Thu Mar 28, 2013 8:31 am

Re: LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by chilinux » Fri Jul 09, 2021 6:13 am

It sounds like you have an EC2 instance that has only a single public IP that you expect to share back to the LEMP server.

If that is the case, then you will need to set up Destination Network Address Translation (DNAT) which would go in the PREROUTING/POSTROUTING chains of iptables.

If you have used UFW to create entries in the INPUT chain of iptables then that just allows the packets to services running on the immediate EC2 server rather than forwarded across the tunnel. It is DNAT instead that will perform the packet forwarding and IP translation you want.

There is some changes you should make in OpenVPN AS first.

Under: Configuration -> Network Settings -> Multi-Daemon Mode -> TCP port number
Change the port from 443 to a different port that you don't plan to switch to performing DNAT on.

Make sure whatever port you set is also allowed by UFW.

Next, Under: Configuration -> VPN Settings -> Routing -> Should client Internet traffic be routed through the VPN?
Make sure this is set to Yes

You may want to set the LEMP server's user account to have a consistent static intranet address.
This can be doing under User Management -> User Permissions -> [LEMP user name] -> More Settings -> IP Addressing -> Use Static
Then also supply the intranet IP under VPN Static IP Address

You should then be able to set up DNAT to/from the static VPN intranet address via UFW by following the examples given in this document:
https://bobcares.com/blog/ufw-port-forwarding/

Lastly, whenever you restart UFW, it may flush all the iptables rules including the ones OpenVPN AS expects to have in place to work. As such, every time you restart UFW, follow it by then restarting OpenVPN AS to allow it to re-establish it's own iptables entries.

User avatar
Danran
OpenVPN User
Posts: 28
Joined: Tue Jun 29, 2021 9:21 am

Re: LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by Danran » Sat Jul 10, 2021 8:37 pm

This is extremely helpful, and thus far, I have done everything except for the two last steps (setting up DNAT and flushing the tables). I find the instructions at bobcares.com a bit vague. Maybe you can help me out here.

My opvnpnas public (amazon static) ip is

Code: Select all

n.nnn.nnn.nnn
Which is NOT LISTED in

Code: Select all

ifconfig
or

Code: Select all

ip addr show
On my openas server, the command

Code: Select all

ip addr show
returns:

Code: Select all

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc fq_codel state UP group default qlen 1000
    link/ether aa:aa:aa:aa:aa:aa brd ff:ff:ff:ff:ff:ff
    inet bbb.bb.b.bbb/20 brd ccc.cc.cc.ccc scope global dynamic eth0
       valid_lft 3367sec preferred_lft 3367sec
    inet6 fe90::d8:11ff:fb20:296c/64 scope link
       valid_lft forever preferred_lft forever
5: as0t0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 200
    link/none
    inet 173.21.223.1/20 scope global as0t0
       valid_lft forever preferred_lft forever
    inet6 fb70::2010:f7fc:849d:8380/64 scope link stable-privacy
       valid_lft forever preferred_lft forever
On my vpn client (the web server), I am successfully connected to my EC2 Openvnpas Server and the command

Code: Select all

ip addr show
returns:

Code: Select all

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether yy:yy:yy:yy:yy:yy brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.20/24 brd 192.168.30.255 scope global dynamic wlan0
       valid_lft 863798sec preferred_lft 863798sec
    inet6 fe70::bcd5:22ff:fbe5:32d2/64 scope link
       valid_lft forever preferred_lft forever
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
    link/none
    inet 10.8.0.20/24 brd 10.8.0.255 scope global tun0
       valid_lft forever preferred_lft forever
On the tutorial that you instructed me to use, it instructs you to put the following rules in

Code: Select all

/etc/ufw/before.rules
.

Code: Select all

*nat
:PREROUTING ACCEPT [0:0]
-A PREROUTING -i eth0 -d 2xx.2xx.1xx.0 -p tcp --dport 80 -j DNAT --to-destination 1xx.1xx.2xx.0:8080
# setup routing
-A POSTROUTING -s 1xx.1xx.2xx.0/24 ! -d 1xx.1xx.2xx.0/24 -j MASQUERADE
COMMIT
But it doesn't explain which ip addresses I am supposed to enter for the multiple (4 to be exact) ip address entries listed in the example. What should I be entering there as a replacement of the examples ip addresses?

Lastly, does the order matter where these rules are supposed to go in /etc/ufw/before.rules? Should it be at the top of the page, the bottom of the page, or in betweeen something?

chilinux
OpenVPN Power User
Posts: 156
Joined: Thu Mar 28, 2013 8:31 am

Re: LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by chilinux » Sun Jul 11, 2021 1:40 am

If I am reading your information correctly, you need to move packets arriving at bbb.bb.b.bbb to 10.8.0.20 and back again.

The example would become:

Code: Select all

*nat
:PREROUTING ACCEPT [0:0]
-A PREROUTING -i eth0 -d bbb.bb.b.bbb -p tcp --dport 80 -j DNAT --to-destination 10.8.0.20:80
# setup routing
-A POSTROUTING -s 10.8.0.0/24 ! -d 10.8.0.0/24 -j MASQUERADE
COMMIT
You will need to repeat the PREROUTING rule for each port you want to forward (443, 25, 993, etc).

If you find iptables confusing/frustrating, you can accomplish the same goal by running NGINX on the EC2 server to act as the reverse proxy.

Since you are already learning NGINX on the LEMP server, you might find that easier to configure than iptables.

xinetd also provides reverse proxy forwarding.

User avatar
openvpn_inc
OpenVPN Inc.
Posts: 1332
Joined: Tue Feb 16, 2021 10:41 am

Re: LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by openvpn_inc » Mon Jul 12, 2021 10:14 am

Hello Danran,

> 2. Should I be forwarding ports from the ec2 instance public IP to the vpn client, and if so, how exactly do I do such a thing on an openvpnas instance? The command line instructions to use DMZ forwarding errors out. So what how exactly can I forward the aws public ip and ports 443 and 80 to the client vpn webserver?

Yes that's what you should do. You can use the DMZ function to forward ports on the IP of your EC2 instance to your connected VPN client. Of course with AWS you have to keep in mind that its external public IP address is not actually on the EC2 instance. It's a 1:1 NAT to the private IP address of your EC2 instance, so in this case you need to specify just the local private IP of the EC2 instance that your Access Server is on - not the attached public IP.

See this document. This has been tested by me today to work with OpenVPN Access Server 2.9.2 on Amazon AWS:
https://openvpn.net/vpn-server-resource ... ss-server/

You have to only do these instructions to make this work. None of the other things mentioned in this topic are necessary to achieve this goal. The DMZ functionality is a standard function built into Access Server.

If you run into any problems with this please let us know the error message you got.

Kind regards,
Johan
Image OpenVPN Inc.
Answers provided by OpenVPN Inc. staff members here are provided on a voluntary best-effort basis, and no rights can be claimed on the basis of answers posted in this public forum. If you wish to get official support from OpenVPN Inc. please use the official support ticket system: https://openvpn.net/support

chilinux
OpenVPN Power User
Posts: 156
Joined: Thu Mar 28, 2013 8:31 am

Re: LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by chilinux » Tue Jul 13, 2021 7:20 pm

If the client DMZ shares the same IP address that OpenVPN AS is bound to, what happens with port 443 when the DMZ user is connected?

Does it continue to go to OpenVPN AS or is it forwarded on to the LEMP server?

If it is forwarded on to the LEMP server, shouldn't OpenVPN AS stop making references to TCP 443 in the client ovpn configuration files it generates?

User avatar
Danran
OpenVPN User
Posts: 28
Joined: Tue Jun 29, 2021 9:21 am

Re: LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by Danran » Wed Jul 14, 2021 6:21 pm

openvpn_inc wrote:
Mon Jul 12, 2021 10:14 am
Hello Danran,

> 2. Should I be forwarding ports from the ec2 instance public IP to the vpn client, and if so, how exactly do I do such a thing on an openvpnas instance? The command line instructions to use DMZ forwarding errors out. So what how exactly can I forward the aws public ip and ports 443 and 80 to the client vpn webserver?

Yes that's what you should do. You can use the DMZ function to forward ports on the IP of your EC2 instance to your connected VPN client. Of course with AWS you have to keep in mind that its external public IP address is not actually on the EC2 instance. It's a 1:1 NAT to the private IP address of your EC2 instance, so in this case you need to specify just the local private IP of the EC2 instance that your Access Server is on - not the attached public IP.

See this document. This has been tested by me today to work with OpenVPN Access Server 2.9.2 on Amazon AWS:
https://openvpn.net/vpn-server-resource ... ss-server/

You have to only do these instructions to make this work. None of the other things mentioned in this topic are necessary to achieve this goal. The DMZ functionality is a standard function built into Access Server.

If you run into any problems with this please let us know the error message you got.

Kind regards,
Johan
So in Openvpnas web interface>user management>user permissions> myuser> more settings...
I put...
myamazonec2privateip:tcp/443
myamazonec2privateip:tcp/80

and my webserver is now properly working and serving my site on the EC2 instances vpn tunnel! THANK YOU BOTH FOR THE HELP!

Before I mark the thread as solve, I just want to verify a few more things. First, would be the question above from user "chilinux".

Then, I also noticed that my Amazon EC2 instance's private IP is the same ip as my EC2 instances eth0 interface. Does that sound right to you? Lastly, to double check, I do not need to set any firewall rules or do anything else after this, correct? (Maybe off topic) Is there anything in terms of security or functionality that you can think of that should be used on an ec2 openvpnas instance? I already installed fail2ban, but only have it set up for ssh. The only thing I noticed Is my web server is running relatively slow off of the vpn instance, but I'm assuming there is no way of speeding this up and this is due to the natural lag of the openvpn?

chilinux
OpenVPN Power User
Posts: 156
Joined: Thu Mar 28, 2013 8:31 am

Re: LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by chilinux » Thu Jul 15, 2021 8:01 pm

Your EC2 eth0 should be set to the instance private IP. That is normal.

Also, the DMZ feature should allow OpenVPN AS to take care of all the iptables rules for you. Changing UFW shouldn't be required.

Other security hardening I would recommend:

Move away from using passwords to authenticate SSH sessions. Instead use ssh-keygen and authorized_keys file to perform key based authentication. Once you confirmed you can authenticate via a ssh key, modify the sshd_config to set PasswordAuthentication to no. After making the modification, restart the ssh server. Once you get rid of password based ssh authentication, I personally see no reason to continue to use fail2ban.

Additional hardening beyond that would depend on if this is a production or a test/hobby instance of OpenVPN AS. If you are using this as a production server then do *NOT* proceed to the next suggestion. Otherwise, for a test instance, you can try playing with turning on SElinux. Doing this goes against the official documentation of OpenVPN AS. Depending on what features you use or what changes are made in future updates to OpenVPN AS, you need to be prepared to audit the events SElinux prohibits and update the SElinux policies accordingly.

I have been trying OpenVPN AS on CentOS 7 with SElinux enabled on a *test* system. So far I haven't run into any problems. But for a test system, I see problems with SElinux as a learning opportunity and consider SElinux something worth learning for anyone interested in server hardening.

To be clear (if I wasn't already), I am only recommending this for a *test* system with expectations that functionality may break and that SElinux is outside of the scope of OpenVPN AS support. For a production system, enabling SElinux is not recommended.

As far as performance, it is hard to tell without knowing the EC2 instance type you are using. For example, with a c4.large instance has AES-NI support built into the hardware and OpenVPN overhead should be minimal. Running "openssl speed" on different instance types should give you an idea of the level of overhead encrypting/decrypting traffic should have on that instance.

User avatar
Danran
OpenVPN User
Posts: 28
Joined: Tue Jun 29, 2021 9:21 am

Re: LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by Danran » Thu Sep 16, 2021 8:56 am

So I think I need to re-open this thread, as I am still having issues regarding this post. I HAVE set up DMZ forwarding on the AWS Instance, and the WEB SERVER is serving properly. However, I seemed to overlook one big obstacle with all of this. I neglected to mention that the Web Server is also a postfix/dovecot email server as well. The reason for my neglect, is that I assumed that if my web server worked accross the VPN, that my email server would do just the same. However, I have hit a serious wall when it comes to the EMAIL/WEBSERVER receiving emails from an outside source such as gmail on a different computer.

To start/continue, it seems as though (according to answers from https://community.linuxbabe.com/t/email ... 4?u=danran and https://serverfault.com/questions/50313 ... hrough-vpn) my VPS (running openvpn-as) doesn't forward the public IP address of gmail to my mail/web server, and my mail server is only seeeing the IP address of the VPS (10.10.9.1).

I need to find a way for my VPS to forward the Public IP address of my gmail to my email/web server, and for my webserver to recognize that forwarded Public IP address. As of now, my email server is only seeing 10.10.9.1.

You can see how postfix is denying my emails due to the IP 10.10.9.1 not having a reverse hostname (which it shouldn't) in the postfix logs below:

Code: Select all

Sep 15 09:37:14 mail postfix/smtpd[4323]: NOQUEUE: reject: RCPT from unknown[10.10.9.1]: 450 4.7.1 Client host rejected: cannot find your reverse hostname, [10.10.9.1]; from=<danran@gmail.com> to=<admin@facl.xyz> proto=ESMTP helo=<mail-io1-f54.google.com>
Sep 15 09:37:14 mail postfix/smtpd[4323]: disconnect from unknown[10.10.9.1] ehlo=2 starttls=1 mail=1 rcpt=0/1 bdat=0/1 quit=1 commands=5/7
The author of linuxbabe.com has suggested that I set up an SMTP & IMAP Proxy with HAProxy, but I am almost certain their must be a cleaner solution that can be done strictly with openvpn on my VPS.

Could anyone provide some help in understanding what is going on here, and what I need to do exactly to fix this?

Thanks in advance for any help!

User avatar
openvpn_inc
OpenVPN Inc.
Posts: 1332
Joined: Tue Feb 16, 2021 10:41 am

Re: LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by openvpn_inc » Thu Sep 16, 2021 2:39 pm

Hello Danran,

I tend to agree with the assessment that setting up an SMTP proxy on the Access Server itself to run mail operations is a reasonable solution. It would be actually using the real IP, and be a front to the outside world, while in the backend your mailserver connected via VPN is handling stuff. For IMAP I don't know if that's such a great solution but it could work.

The VPN client machine has a private IP, it does not have a public IP. A private IP cannot be routed on the Internet. So therefore NAT must be applied. The way Access Server does it is by replacing the source and destination IP addresses, so that the traffic can be handled within the VPN subnet, and replies in and out go through the Access Server's public IP. This is done because we cannot assume that the VPN tunnel is doing full redirection of traffic, it could be just split-tunnel. And then you'd get asymmetric routing which would break things if request from outside came in and your VPN client tried to respond to it and it's not forwarded back through the VPN tunnel.

What you could try is to turn off the functionality that Access Server implements for DMZ, the ability to forward traffic coming on a public IP on the Access Server and NATting it to a VPN client. And then you ensure this VPN client has a static IP in the VPN network, and then you create your own iptables NAT rule. One that does NAT outgoing but does not NAT incoming. Basically the source address would remain intact and the VPN client can see the original IP. However this does require that the VPN client is configured in such a way that it sends all its Internet traffic through the VPN tunnel, or else it will fail.

Access server already does the outgoing NAT. You just need to forward traffic coming in on certain ports on the public IP of the Access Server to a specific VPN internal IP of this VPN client, without doing it with NAT. So just a port forward. The Access Server has no configuration option for this, you'd have to do it yourself in iptables.

Kind regards,
Johan
Image OpenVPN Inc.
Answers provided by OpenVPN Inc. staff members here are provided on a voluntary best-effort basis, and no rights can be claimed on the basis of answers posted in this public forum. If you wish to get official support from OpenVPN Inc. please use the official support ticket system: https://openvpn.net/support

User avatar
Danran
OpenVPN User
Posts: 28
Joined: Tue Jun 29, 2021 9:21 am

Re: LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by Danran » Thu Sep 16, 2021 4:38 pm

Thank you for such a prompt reply! However, I am having a bit of difficulty grasping this. Let me try to summarize here....
wrote: What you could try is to turn off the functionality that Access Server implements for DMZ, the ability to forward traffic coming on a public IP on the Access Server and NATting it to a VPN client.
The sentance structure is a bit confusing to me here. What exactly do you mean? That I need to somehow turn off what access server does with default dmz forwarding rules, by using iptables? In other words, I need to stop access server from forwarding the incoming traffic to the vpn, and instead NAT it to the VPN Cleint? I dunno. Would you remind revising that sentance? I'm not grasping it.
And then you ensure this VPN client has a static IP in the VPN network,
My VPN client does have a static IP assigned to it by the access server, if that is what you mean.
and then you create your own iptables NAT rule.
This makes me ask, where does Openvpn-AS Store its default iptables NAT rules? How can I ovveride those rules to create my own NAT rules?
One that does NAT outgoing but does not NAT incoming. Basically the source address would remain intact and the VPN client can see the original IP. However this does require that the VPN client is configured in such a way that it sends all its Internet traffic through the VPN tunnel, or else it will fail.
This is not a problem for me, I absolutely want all of my traffic being tunneled through the vpn. No split tunnels for me.
Access server already does the outgoing NAT. You just need to forward traffic coming in on certain ports on the public IP of the Access Server to a specific VPN internal IP of this VPN client, without doing it with NAT.
What do you mean by a "specific VPN internal IP of this VPN client"? and "without doing it with NAT"? I'm totally confused here. Do you have some examples, or could you possibly revise your sentance structure?
So just a port forward. The Access Server has no configuration option for this, you'd have to do it yourself in iptables.
To me, it sounds like you are saying that basically I am already set up and ready to go, except for port forwarding. So all I have to do is create a custom IPTABLES rule that forwards ports from my AS Public IP address, to my VPN Static Client? If so, can you give me a good example of a rule? I am very new to IPTABLES and generally stick to using UFW as my firewall.

It seems like I only have a small grasp (if any) on what you are sayin here in full. Could you please correct me where my assumptions have gone wrong here? Any further help getting me to obtain a grasp on this is certainly much appreciated.

Thank you Johan!

User avatar
openvpn_inc
OpenVPN Inc.
Posts: 1332
Joined: Tue Feb 16, 2021 10:41 am

Re: LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by openvpn_inc » Fri Sep 17, 2021 2:29 pm

Hello,

When you use the DMZ function in Access Server, it forwards traffic and it uses NAT. That means the source IP of requests coming in changes. And that means you don't see the original IP of the requests coming from outside. And that's your problem. So the idea is to turn off DMZ function in Access Server. And then implement your own solution in iptables. But do so without NAT. So that the original source IP remains intact.

You can find more about iptables related settings in Access Server here:
https://openvpn.net/faq/how-do-i-preven ... inux-host/

Please note that we do not extend our support to custom iptables.

Kind regards,
Johan
Image OpenVPN Inc.
Answers provided by OpenVPN Inc. staff members here are provided on a voluntary best-effort basis, and no rights can be claimed on the basis of answers posted in this public forum. If you wish to get official support from OpenVPN Inc. please use the official support ticket system: https://openvpn.net/support

User avatar
Danran
OpenVPN User
Posts: 28
Joined: Tue Jun 29, 2021 9:21 am

Re: LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by Danran » Fri Sep 17, 2021 4:07 pm

This is much more clear! Thank you for the answer. I understand now!

User avatar
Danran
OpenVPN User
Posts: 28
Joined: Tue Jun 29, 2021 9:21 am

Re: LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by Danran » Mon Jun 06, 2022 10:56 pm

Dear @chilinux although this is a bit off topic, it still uses the very same principles. So, I actually did get this entire thing to work including the emails on the LEMP server being properly forwarded to and from the Public of the Open-VPN server. The results of this are that my LEMP email/web server now works perfectly. Nothing whatsoever is wrong with it. However, Now I am running a second LEMP server and I am trying to rig it up to another Instance running as a VPN server, to once again, have a perfectly working LEMP Email and Web server. In short, I want to do the exact same thing that I successfully did before, HOWEVER, I must have somehow forgotten a step, because for the life of me I cannot recreate this setup on another LEMP server/VPN-Instance-Server setup. I am desperately hoping you may be able to check my work here and give me a second eye on the below steps I have taken to reproduce this exact scenario. So far, here is exactly what I have done on both the LEMP Web server, and the VPS VPN-server.

Also, just for the record and transparency, I have also posted a similar question in https://serverfault.com/questions/11016 ... not-get-ng but it gets absolutely no attention, which is why i am resorting to you in hopes that you can save me here.


Lets begins:

TOPIC: Tunneling a LEMP server through an oracle instance with OpenVpn. Cannot get Nginx Certbot certificates on LEMP server. How to debug?

I have a LEMP server at home running Ubuntu 22.02 and an Oracle cloud instance running Ubuntu 20.04. The Oracle cloud instance is acting as a OpenVpn server. The LEMP server at home is acting as the OpenVpn Client, and is being tunneled through the Oracle server in order to obtain an IP address that is different from my home IP address. I have set up this OpenVpn Client/Server configuration using OpenVpn server script from the command line. The OpenVpn client is up and running and can successfully ping the Oracle Server. The LEMP Server (OpenVpn client) also successfully resolves to the Oracle Servers Public IP address. I have also installed openresolv on the VPN Client, and bind9 on the VPN server to use the Oracle Instance's DNS from the LEMP server (the OpenVpn Client). My Domain Registrar is pointing it's DNS to the same IP as my Oracle Server. Now, I am attempting to install prosody on the LEMP server (the vpn client), and cannot obtain certificates through the certbot nginx plugin. It seems as though something is blocking port 80/443 and that port 80/443 is not open. When I run (on my LEMP server) the command:

Code: Select all

sudo certbot -v --nginx --agree-tos --redirect --hsts --staple-ocsp --email example@example.com -d chat.example.com
I get the following error output:

Code: Select all

   Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator nginx, Installer nginx
    Requesting a certificate for chat.example.com
    Performing the following challenges:
    http-01 challenge for chat.example.com
    Waiting for verification...
    Challenge failed for domain chat.example.com
    http-01 challenge for chat.example.com
    
    Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
      Domain: chat.example.com
      Type:   connection
      Detail: 150.136.56.232: Fetching http://chat.example.com/.well-known/acme-challenge/GlzBhvxB_hDYefMW48qaHq3I-qc_NArj7VWml54bofM: Connection refused
    
    Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.
    
    Cleaning up challenges
    Some challenges have failed.
    Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
I am using UFW as my firewall, and my UFW status on the LEMP Server (the VPN Client) is as follows:

Code: Select all

    Status: active
    
         To                         Action      From
         --                         ------      ----
    [ 1] 22/tcp                     ALLOW IN    my.local.lan.ip/24
    [ 2] 43211/tcp                  ALLOW IN    my.local.lan.ip/24
    [ 3] 5222,5269/tcp              ALLOW IN    Anywhere
    [ 4] 80,443/tcp                 ALLOW IN    Anywhere
    [ 5] 5222,5269/tcp (v6)         ALLOW IN    Anywhere (v6)
    [ 6] 80,443/tcp (v6)            ALLOW IN    Anywhere (v6)
My UFW Status on the Oracle Cloud Instance is:

Code: Select all

    Status: active
    
         To                         Action      From
         --                         ------      ----
    [ 1] Anywhere                   ALLOW IN    10.10.10.0/24
    [ 2] 22/tcp                     ALLOW IN    my.home.public.ip
    [ 3] 22/tcp                     ALLOW IN    my.work.public.ip
    [ 4] 5222,5269/tcp              ALLOW IN    Anywhere
    [ 5] 1194/udp                  ALLOW IN    my.home.public.ip
    [ 6] 80,443/tcp                 ALLOW IN    Anywhere
    [ 7] 5222,5269/tcp (v6)         ALLOW IN    Anywhere (v6)
    [ 8] 80,443/tcp (v6)            ALLOW IN    Anywhere (v6)
Port 1194/udp is my OpenVpn port for both the Oracle Instance, as well as the Lemp Server. I have also forwarded my public IP address on the Oracle Instance to the VPN client, so that the client can send and receive on the same public port that the Oracle Instance uses. Below is my /etc/ufw/before.rules file. The port forwarding modifications I made are under the comments entitled "Linuxbabe...".

Code: Select all

    #
    # rules.before
    #
    # Rules that should be run before the ufw command line added rules. Custom
    # rules should be added to one of these chains:
    #   ufw-before-input
    #   ufw-before-output
    #   ufw-before-forward
    #
    
    # Don't delete these required lines, otherwise there will be errors
    *filter
    :ufw-before-input - [0:0]
    :ufw-before-output - [0:0]
    :ufw-before-forward - [0:0]
    :ufw-not-local - [0:0]
    # End required lines
    
    
    # allow all on loopback
    -A ufw-before-input -i lo -j ACCEPT
    -A ufw-before-output -o lo -j ACCEPT
    
    # quickly process packets for which we already have a connection
    -A ufw-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
    -A ufw-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
    -A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
    
    # drop INVALID packets (logs these in loglevel medium and higher)
    -A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
    -A ufw-before-input -m conntrack --ctstate INVALID -j DROP
    
    # ok icmp codes for INPUT
    -A ufw-before-input -p icmp --icmp-type destination-unreachable -j ACCEPT
    -A ufw-before-input -p icmp --icmp-type time-exceeded -j ACCEPT
    -A ufw-before-input -p icmp --icmp-type parameter-problem -j ACCEPT
    -A ufw-before-input -p icmp --icmp-type echo-request -j ACCEPT
    
    # ok icmp code for FORWARD
    -A ufw-before-forward -p icmp --icmp-type destination-unreachable -j ACCEPT
    -A ufw-before-forward -p icmp --icmp-type time-exceeded -j ACCEPT
    -A ufw-before-forward -p icmp --icmp-type parameter-problem -j ACCEPT
    -A ufw-before-forward -p icmp --icmp-type echo-request -j ACCEPT
    
    ## Linuxbabe tutorial
    # allow forwarding for trusted network
    -A ufw-before-forward -s 10.10.10.0/24 -j ACCEPT
    -A ufw-before-forward -d 10.10.10.0/24 -j ACCEPT
    
    # allow dhcp client to work
    -A ufw-before-input -p udp --sport 67 --dport 68 -j ACCEPT
    
    #
    # ufw-not-local
    #
    -A ufw-before-input -j ufw-not-local
    
    # if LOCAL, RETURN
    -A ufw-not-local -m addrtype --dst-type LOCAL -j RETURN
    
    # if MULTICAST, RETURN
    -A ufw-not-local -m addrtype --dst-type MULTICAST -j RETURN
    
    # if BROADCAST, RETURN
    -A ufw-not-local -m addrtype --dst-type BROADCAST -j RETURN
    
    # all other non-local packets are dropped
    -A ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny
    -A ufw-not-local -j DROP
    
    # allow MULTICAST mDNS for service discovery (be sure the MULTICAST line above
    # is uncommented)
    -A ufw-before-input -p udp -d 224.0.0.251 --dport 5353 -j ACCEPT
    
    # allow MULTICAST UPnP for service discovery (be sure the MULTICAST line above
    # is uncommented)
    -A ufw-before-input -p udp -d 239.255.255.250 --dport 1900 -j ACCEPT
    
    # don't delete the 'COMMIT' line or these rules won't be processed
    COMMIT
    
    ### Linuxbabe - UFW Tutorial
    ## NAT table rules
    *nat
    # Nat Pre-Routing
    :PREROUTING ACCEPT [0:0]
    # !Http! Forward oracle.server.public.ip (Server Public IP) TCP port 80 to 10.10.10.2:80 (VPN Client IP)
    -A PREROUTING -i ens3 -d oracle.server.public.ip  -p tcp --dport 80 -j DNAT --to-destination 10.10.10.2:80
    
    # !HTTPS! Forward oracle.server.public.ip (Server Public IP) TCP port 443 to 10.10.10.2:443 (VPN Client IP)
    -A PREROUTING -i ens3 -d oracle.server.public.ip  -p tcp --dport 443 -j DNAT --to-destination 10.10.10.2:443
    
    # !PROSODY-5222! Forward oracle.server.public.ip (Server Public IP) TCP port 5222 to 10.10.10.2:5222 (VPN Client IP)
    -A PREROUTING -i ens3 -d oracle.server.public.ip  -p tcp --dport 5222 -j DNAT --to-destination 10.10.10.2:5222
    
    # !PROSODY-5269! Forward oracle.server.public.ip (Server Public IP) TCP port 5269 to 10.10.10.2:5269 (VPN Client IP)
    -A PREROUTING -i ens3 -d oracle.server.public.ip  -p tcp --dport 5269 -j DNAT --to-destination 10.10.10.2:5269
    
    # !PROSODY-BOSH-5280! Forward oracle.server.public.ip (Server Public IP) TCP port 5280 to 10.10.10.2:5280 (VPN Client IP)
    -A PREROUTING -i ens3 -d oracle.server.public.ip  -p tcp --dport 5280 -j DNAT --to-destination 10.10.10.2:5280
    
    # !PROSODY-BOSH-5281! Forward oracle.server.public.ip (Server Public IP) TCP port 5281 to 10.10.10.2:5281 (VPN Client IP)
    -A PREROUTING -i ens3 -d oracle.server.public.ip  -p tcp --dport 5281 -j DNAT --to-destination 10.10.10.2:5281
    COMMIT
    
    ### Linuxbabe - OpenVpn Tutorial
    # Nat Postrouting
    *nat
    :POSTROUTING ACCEPT [0:0]
    -A POSTROUTING -s 10.10.10.0/24 -o ens3 -j MASQUERADE
    
    # End each table with the 'COMMIT' line or these rules won't be processed
    COMMIT
My `/etc/nginx/nginx.conf` file on the LEMP Server (the OpenVpn client) looks like this:

Code: Select all

    #user  nginx;
    user www-data;
    worker_processes  auto;
    
    error_log  /var/log/nginx/error.log notice;
    pid        /var/run/nginx.pid;
    
    
    events {
        worker_connections  1024;
    }
    
    
    http {
        include       /etc/nginx/mime.types;
        default_type  application/octet-stream;
    
        log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
    
        access_log  /var/log/nginx/access.log  main;
    
        sendfile        on;
        #tcp_nopush     on;
    
        keepalive_timeout  65;
    
        #gzip  on;
    
        include /etc/nginx/conf.d/*.conf;
    }
My `/etc/nginx/conf.d/prosody.conf` file on the LEMP Server (VPN Client) looks like this:

Code: Select all

    server {
          listen 80;
          listen [::]:80;
          server_name chat.example.com;
    
          root /var/www/prosody/;
    
          location ~ /.well-known/acme-challenge {
             allow all;
          }
    }
Lastly, I have enabled IP Forwarding on the Oracle Server by uncommenting the line

Code: Select all

net.ipv4.ip_forward = 1
in

Code: Select all

/etc/sysctl.conf
.

After all of this, my LEMP server seems to be successfully using the Oracle Instance's VPN Server Tunnel, HOWEVER, it still cannot obtain certificates from certbot using the prosody.conf or any working nginx config file. As far as I can tell, with all of the research I've done, this setup (most importantly the VPN forwarding rules in UFW's before.rules), should successfully allow my LEMP server (the vpn client) to obtain certificates using the Oracle Instance's IP address. But it is NOT DOING SO, and I cannot obtain certificates for the life of me.

So my question is, what must I do to debug this, what is blocking my LEMP servers port 80/443, and what must I do to successfully obtain certbot certificates for prosody using my Oracle Instance's Public IP address? Essentially, what steps am I missing here? Any help would be insanely appreciated as I have posted all over the internet and scoured blogs and forums with no luck at all. I've been going at this problem for 4 months now, and I will just be darned because I can't figure out how I did it like I did the first time. Please help if you can! I willl buy youa beer or some coffee!

Thank you so much!

Dan Ran

User avatar
Danran
OpenVPN User
Posts: 28
Joined: Tue Jun 29, 2021 9:21 am

Re: LEMP Webserver as OpenVPN client to aws instance - Proper port forwarding?

Post by Danran » Mon Jun 06, 2022 11:06 pm

chilinux wrote:
Thu Jul 15, 2021 8:01 pm
Your EC2 eth0 should be set to the instance private IP. That is normal.

Also, the DMZ feature should allow OpenVPN AS to take care of all the iptables rules for you. Changing UFW shouldn't be required.

Other security hardening I would recommend:

Move away from using passwords to authenticate SSH sessions. Instead use ssh-keygen and authorized_keys file to perform key based authentication. Once you confirmed you can authenticate via a ssh key, modify the sshd_config to set PasswordAuthentication to no. After making the modification, restart the ssh server. Once you get rid of password based ssh authentication, I personally see no reason to continue to use fail2ban.

Additional hardening beyond that would depend on if this is a production or a test/hobby instance of OpenVPN AS. If you are using this as a production server then do *NOT* proceed to the next suggestion. Otherwise, for a test instance, you can try playing with turning on SElinux. Doing this goes against the official documentation of OpenVPN AS. Depending on what features you use or what changes are made in future updates to OpenVPN AS, you need to be prepared to audit the events SElinux prohibits and update the SElinux policies accordingly.

I have been trying OpenVPN AS on CentOS 7 with SElinux enabled on a *test* system. So far I haven't run into any problems. But for a test system, I see problems with SElinux as a learning opportunity and consider SElinux something worth learning for anyone interested in server hardening.

To be clear (if I wasn't already), I am only recommending this for a *test* system with expectations that functionality may break and that SElinux is outside of the scope of OpenVPN AS support. For a production system, enabling SElinux is not recommended.

As far as performance, it is hard to tell without knowing the EC2 instance type you are using. For example, with a c4.large instance has AES-NI support built into the hardware and OpenVPN overhead should be minimal. Running "openssl speed" on different instance types should give you an idea of the level of overhead encrypting/decrypting traffic should have on that instance.
I know it's been a long time since we talked about this subject, but I'm begging you to be my superhero right now. I successfully accomplished all I was trying to attempt in the original post, however, I am unable to reproduce the results on a different VPS and a different LEMP Server. Do you think you could pretty pretty please with a cherry on top look at this post:viewtopic.php?t=32609#p106856 and maybe give it a look and take a stab at it? I would be so so so grateful for any help and advice you could give. I am so so stuck and it's killing me. Please help if you have the time my friend! Cheers!

Post Reply