Connot reach services on router but can reach beyond

devanb

Thank you for taking the time to read my problem. Spent a few days and I am getting stuck.

I have a site to site openvpn cloud setup and devices on one side of the network can reach devices on the other side.

Each pfsense router can ping each other. I cannot, however, reach the webgui of the router using the ip address provided from openvpn cloud. If I use the private address of the remote router, I can access it.

This setup has worked well when using openvpn on a local server but is failing when using the cloud.

I need to have the routers talk to each other in order to utilize dynamic routing with BGP. BGP cannot reach the neighbor on the other side of the OpenVPN cloud.

I hope I am making sense.

Any help or direction would be greatly appreciated.

openvpn_inc

Hi devanb,

Since the two pfsense routers can ping each other it seems unlikely to have anything to do with OpenVPN, as pinging proves the traffic works. So my guesses are that either the web interface needs to be configured to listen on the specific IP address or interface for the OpenVPN tunnel connection, which may not be the case by default, or that there's a firewall in the pfsense router that's blocking this access. You might want to check with pfsense support to see if there's anything they can advise that might help in this regard.

Kind regards,
Johan

devanb

Thank you Johan for the review of my issues and setup.

I've reached out to pfSense but they also came to the same conclusion that I probably have a configuration issue.

Honestly, I suspect it is related to the fact I am not using an official openvpn cloud connector but instead deconstructing the ovpn into CA/CERT/TLS. Although it is connecting, I'm getting a ton of errors and unrecognized push commands.

Jul 6 22:39:01 	openvpn 	48583 	Options error: Unrecognized option or missing or extra parameter(s) in [PUSH-OPTIONS]:12: block-outside-dns (2.6_git)
Jul 6 22:39:01 	openvpn 	48583 	Options error: Unrecognized option or missing or extra parameter(s) in [PUSH-OPTIONS]:11: remote-cache-lifetime (2.6_git)
Jul 6 22:39:01 	openvpn 	48583 	Options error: option 'reneg-sec' cannot be used in this context ([PUSH-OPTIONS])
Jul 6 22:39:01 	openvpn 	48583 	Options error: Unrecognized option or missing or extra parameter(s) in [PUSH-OPTIONS]:4: client-ip (2.6_git)

I especially suspect the unrecognized option client-ip may be the culprit.

I also do not know if the setenv USERNAME "dd...." is actually making it to the cloud.

When my non-cloud openvpn client / servers connect, I'm not having any issues.

devanb

One more weird observation.

If I traceroute to Site B, packets are shown on the Site A cloud interface ( 1 10.3.103.34 10.040 ms !N 9.922 ms !N 9.784 ms !N) but nothing is on Site B. It seems as if something in the openvpn cloud is replying but not coming though to my actual router.

Corroborative evidence:

Site A packet capture including ICMP

23:37:09.470681 IP 10.3.103.18 > 10.3.103.34: ICMP echo request, id 20024, seq 6808, length 9
23:37:09.490083 IP 10.3.103.34 > 10.3.103.18: ICMP echo reply, id 20024, seq 6808, length 9
23:37:09.997538 IP 10.3.103.18 > 10.3.103.34: ICMP echo request, id 20024, seq 6809, length 9

Site B packet capture including ICMP

23:37:08.975473 IP 10.3.103.34 > 10.3.103.18: ICMP echo reply, id 58495, seq 6807, length 9
23:37:09.477693 IP 10.3.103.18 > 10.3.103.34: ICMP echo request, id 58495, seq 6808, length 9
23:37:09.477698 IP 10.3.103.34 > 10.3.103.18: ICMP echo reply, id 58495, seq 6808, length 9
23:37:10.013358 IP 10.3.103.18 > 10.3.103.34: ICMP echo request, id 58495, seq 6809, length 9

The ID of the request and replies do not match. The requests are coming from the cloud and not from each router.

For my wireguard site to site, the ID's match.

This is getting way over my head. Any insight appreciated.

openvpn_inc

Hello devanb,

The option errors are normal, because OpenVPN2 and OpenVPN3 have some minor differences. These particular options should not be an issue in my opinion. I see them here on an OpenVPN2 client too and all functionality works as expected.

About client-ip - if you're able to connect, it works. Cloud only provides static IP addresses and without IP the connection will likely fail. You can probably check which IP address is actually assigned to the pfsense device on the OpenVPN interface and compare it with what Cloud says it should be. If they match it's working correctly. It's just a difference in option handling between OpenVPN2 and OpenVPN3.

About setenv USERNAME - if you're able to connect, it works.

Regarding your weird observation - let's apply some simple logic to eliminate some options. Try disconnecting site B from Cloud and then try pinging from site A to Site B's pfsense router's Cloud VPN IP address. Do you get a response then? If yes, better re-check what exactly you're pinging, and check what IP address this router should be having. If you don't get a response, try connecting site B. Does it respond then? Seems like it's working then. Do similar tests for the subnet you're trying to reach behind the site B router.

There may be some 'oddness' since packets travel through the internal networks of OpenVPN Cloud and this is not a simple flat network. Wireguard for example is point A to point B and that's it. OpenVPN Cloud can cover use cases Wireguard can't, like using the same subnet in multiple locations at the same time, and running multiple same subnet IP ranges side-by-side but still keep them isolated. But I'd suggest doing the simple tests I mentioned to gather some more information.

Kind regards,
Johan

devanb

I thank you again for your time and detailed response. I didn't think about disconnecting only one side at a time.

OpenVPN cloud interfaces:
Site A IP: 10.3.103.18/28 GW 10.3.103.17
Site B IP: 10.3.103.34/18 GW 10.3.103.33
Both provided by OpenVPN cloud.

Ping from router to site A to site B: OK
Ping from router at Site B to router at Site A: OK

Try disconnecting site B from Cloud and then try pinging from site A to Site B's pfsense router's Cloud VPN IP address.

OvpnVPN cloud confirms only Site A is connected.
Ping from router at site A to router at site B: FAIL
Packet capture from Site A cloud interface:

21:38:32.984991 IP 10.3.103.18 > 10.3.103.34: ICMP echo request, id 47435, seq 0, length 64
21:38:33.986690 IP 10.3.103.18 > 10.3.103.34: ICMP echo request, id 47435, seq 1, length 64
21:38:35.010401 IP 10.3.103.18 > 10.3.103.34: ICMP echo request, id 47435, seq 2, length 64

No replies as expected.

If I try to reach ports such as TCP/443 across cloud, the requests are received but no replies (expected)

**I noticed now that I would expect to see BGP requests out on the interface, but I am not.**

I used the test port function to check tcp/62222 on 10.3.103.34 which is open. The requests went out but no reply (expected).

But when I reestablished all connections, that still failed, no requests received on the remote end.

I made sure I have no firewall rules present on my cloud interfaces.

So, I have two problems.
1. BGP is not sending out requests on the local cloud interface. (not a problem for this forum)
2. Requests for the webgui are not being received on the remote cloud interface. (maybe someone knows why?)

I don't know what else to do. I'll just sleep on it.

Thanks,

Devan

openvpn_inc

Hi,

I'd suggest that you disconnect router on site B from the VPN, and take the connection profile from the router on site B, and use that in a standard OpenVPN2 client instead on a desktop or laptop computer. Connect that and run ping tests and port tests again. See if the issue really is with the traffic not making it through OpenVPN Cloud, or if it's an issue in how pfsense is dealing with the traffic.

If it shows it's an issue in OpenVPN Cloud, open a support ticket on our support ticket system and explain what you've tested. They can then provide further support on your particular setup.

Kind regards,
Johan

devanb

I connected my phone to the same cloud network.

It cannot reach the webgui of the routers using the IPs provided from openvpn cloud:10.3.103.18 and 10.3.103.34.

It can reach the webgui of the routers using their respective private IP addresses 172.20.0.1 and 172.20.1.1.

I sincerely believe OpenVPN cloud is not routing packets destined to the IP addresses provided by the openvpn servers. Since I cannot add those destination addresses to the routing table on OpenVPN cloud, they just are not getting there. It's only a problem with dynamic routing which needs to assess route availability. I have submitted tickets and discussed but they don't believe me.

If anyone is using openvpn cloud with dynamic routing set up on the endpoint routers, please let me know.

Thanks,

Devan

openvpn_inc

Hello devan,

Sorry to go back a step again. I'm just trying to follow logical test steps. Can you connect a desktop or laptop to OpenVPN Cloud and pinging the addresses 10.3.103.18 and 10.3.103.34? Doing tests with a phone.... it may not be the best platform to do so. If you connect a desktop or laptop to OpenVPN and ping those addresses, do you get a ping reply? If so, the problem is not in routing in OpenVPN Cloud at all. If it's just the web interfaces not reachable that's still a firewall issue or configuration issue, most likely on the pfsense device.

Kind regards,
Johan

devanb

I know this is a dying thread but I would like to give an update.

In order to access services on the firewall, I created a virtual IP address assigned to localhost. I could then route the requests appropriately, using BGP.

Thanks for all the help.

-Devan