Site-to-site OpenVPN Server on GCE: Searching for good docs

FuriousGeorge

Unfortunately, I don't know enough about the networking side to ask good questions, so for now the purpose of this post is just to find good docs.

I have a GCE node on the left side:

External IP: 12.34.56.78
Internal IP: 10.254.0.2
Netmask: 10.255.255.255

(10.254.0.1 is used by the gateway in a GCE internal network, so I cannot use that address)

It runs Centos 7, but I'm not married to that particular distro, so I'm happy to scrap it.

My goal is to set up a site-to-site VPN to a BSD router (pfSense) on the right side with:

External IP: 87.65.43.21
Internal IP: 10.0.0.1
Netmask: 10.0.0.255

The ultimate end goal is to have multiple site-to-site VPNs up, with subnets like 10.30.0.0/16, 10.200.0.0/16, 10.80.0.0/16, etc, each obviously pointing to a unique LAN, with per subnet routing rules. I'd like to stay as close to the distro template's default settings, as they tend to come geared toward security out-of-the-box from Google. E.g. I'd rather not be disabling firewalld if I could avoid it, and I'd rather leave SElinux enabled unless it really complicates matters. Those things are secondary for now, however.

(Note that I'm not entirely sure this is possible behind GCE. I've just been assuming. I'm aware that there is a site-to-site VPN solution baked into GCS, but it costs a few cents per hour, and multiplied by multiple client domains, many with multiple locations per domain, and it starts adding up to real money very quickly.)

As I mentioned at the beginning of the post, I'm currently looking for docs. I working on a series of dependent assumptions in the search for docs. Currently I'm stuck at one of the early steps: setting up the bridged interface on the left side. I assume I want to set up a tap0 and br0 in order to accomplish that, but in the process of teaching myself how to do this, I've already managed to break the VM's networking twice, which is always a pain to recover from.

Any help with that step would be much appreciated, as the sample configs and most of the documentation I have come across assumes this part is done, it seems.

Traffic

FuriousGeorge wrotefor now the purpose of this post is just to find good docs

Did you read this:

FuriousGeorge wrote setting up the bridged interface

And this:

FuriousGeorge

Traffic wrote
FuriousGeorge wrotefor now the purpose of this post is just to find good docs
Did you read this:
OpenVPN Community Edition Documentation

HOWTO: For OpenVPN Community Edition

FuriousGeorge wrote setting up the bridged interface
And this:
Determining whether to use a routed or bridged VPN

Ethernet Bridging

Thanks. That was under my nose the whole time.

Unfortunately, running bridge start will lock me out of my instance, and I'm not sure why.

#!/bin/sh
#################################
# Set up Ethernet bridge on Linux
# Requires: bridge-utils
#################################
# Define Bridge Interface
br="br0"
# Define list of TAP interfaces to be bridged,
# for example tap="tap0 tap1 tap2".
tap="tap0"
# Define physical ethernet interface to be bridged
# with TAP interface(s) above.
eth="eth0"
eth_ip="10.254.0.2"
eth_netmask="255.255.255.0"
eth_broadcast="10.254.0.255"
for t in $tap; do
    openvpn --mktun --dev $t
done
brctl addbr $br
brctl addif $br $eth
 
for t in $tap; do
    brctl addif $br $t
done
echo "got here 3"
for t in $tap; do
    ifconfig $t 0.0.0.0 promisc up
done
 
 
###  This is where it fails
ifconfig $eth 0.0.0.0 promisc up
 
 
ifconfig $br $eth_ip netmask $eth_netmask broadcast $eth_broadcast

I'm guessing it has something to do with the peculiarities of the GCE platform, and seeking support accordingly.

FuriousGeorge

I've given up on the tap interface for the time being and I'm working on a tun based setup.

Host to client I got it working after manually adding a route to the windows laptop. I was able to ping the server across the VPN.

Then I tried to set up site-to-site.

I can only ping my own remote IP on the OpenVPN server LAN from the client side. I can't ping any devices on the client side from the server side. I've tried adding the whole 10.0.0.0/8 range to the route on the pfsense (BSD) based router on the client side.

Here is my server config:

port 1194
proto udp
dev tun
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/server.crt
key /etc/openvpn/keys/server.key  # This file should be kept secret
dh /etc/openvpn/keys/dh2048.pem
server 10.254.254.0 255.255.255.0
ifconfig-pool-persist ipp.txt
push "route 10.0.0.0 255.0.0.0"
keepalive 10 120
comp-lzo
persist-key
persist-tun
status openvpn-status.log
;log         openvpn.log
;log-append  openvpn.log
verb 3

The Client Side is the pfsense GUI:

[img]
http://snag.gy/ums2C.jpg
[/img]

The server output looks good to me:

Wed Mar 16 07:32:46 2016 69.115.109.62:46450 TLS: Initial packet from [AF_INET]69.115.109.62:46450, sid=71f1b1b7aa100a87
Wed Mar 16 07:32:48 2016 69.115.109.62:46450 VERIFY OK: depth=1, C=US, ST=New Jersey, L=Newark, O=LALALALA, OU=MyOrganizationalUnit, CN=ads-vpn-server-1,name=EasyRSA,emailAddress=me@myemail.copm
Wed Mar 16 07:32:48 2016 69.115.109.62:46450 VERIFY OK: depth=0, C=US, ST=New Jersey, L=Newark, O=LALALALA, OU=MyOrganizationalUnit, CN=ads-120elm-pfsense,name=EasyRSA,emailAddress=me@myemail.com
Wed Mar 16 07:32:48 2016 69.115.109.62:46450 Data Channel Encrypt: Cipher 'BF-CBC' initialized with 128 bit key
Wed Mar 16 07:32:48 2016 69.115.109.62:46450 Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Wed Mar 16 07:32:48 2016 69.115.109.62:46450 Data Channel Decrypt: Cipher 'BF-CBC' initialized with 128 bit key
Wed Mar 16 07:32:48 2016 69.115.109.62:46450 Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Wed Mar 16 07:32:48 2016 69.115.109.62:46450 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
Wed Mar 16 07:32:48 2016 69.115.109.62:46450 [ads-pfsense] Peer Connection Initiated with [AF_INET]69.115.109.62:46450
Wed Mar 16 07:32:48 2016 MULTI: new connection by client 'ads-pfsense' will cause previous active sessions by this client to be dropped.  Remember to use the --duplicate-cn option if you want multiple clients using the same certificate or username to concurrently connect.
Wed Mar 16 07:32:48 2016 MULTI_sva: pool returned IPv4=10.254.254.6, IPv6=(Not enabled)
Wed Mar 16 07:32:48 2016 MULTI: Learn: 10.254.254.6 -> ads-pfsense/69.115.109.62:46450
Wed Mar 16 07:32:48 2016 MULTI: primary virtual IP for ads-pfsense/69.115.109.62:46450: 10.254.254.6
Wed Mar 16 07:32:51 2016 ads-pfsense/69.115.109.62:46450 PUSH: Received control message: 'PUSH_REQUEST'
Wed Mar 16 07:32:51 2016 ads-pfsense/69.115.109.62:46450 send_push_reply(): safe_cap=940
Wed Mar 16 07:32:51 2016 ads-pfsense/69.115.109.62:46450 SENT CONTROL [ads-120elm-pfsense]: 'PUSH_REPLY,r
oute 10.254.254.1,topology net30,ping 10,ping-restart 120,ifconfig 10.254.254.6 10.254.254.5' (status=1)

Any help is much appreciated.

I'll work on getting the br0 iface working for posterity.

Traffic

FuriousGeorge wrote I can't ping any devices on the client side from the server side

See this HOWTO:
HOWTO: Expanding the scope of the VPN to include additional machines

I strongly advise you read the entire HOWTO ..

FuriousGeorge wroteUnfortunately, running bridge start will lock me out of my instance, and I'm not sure why.

Ensure your bridge has a gateway ..

cr2025x1

I think I can help you with this. I'm also in similar setup, using a pfSense router as the VPN server. pfSense is well integrated with OpenVPN, so the options can be manipulated via GUI and it will be much easier. I also successfully made the site-to-site VPN link and really enjoying it.

But first, two site's subnet address is colliding. GCD Node's 10.0.0.0/8 and pfSense's 10.0.0.0/24. GCD Node's subnet is including pfSense's subnet, as first 8 bits of address are same. GCD Node will think pfSense's subnet IP belong to its own subnet and try to send packets into its own subnet, not to pfSense.
Unless you change one subnet into other address(For example, you may change pfSense's subnet to 192.168.xxx.0/24), you will not be able to achieve your goal.

FuriousGeorge

I started with a fresh Centos 7.2 instance and disabled the firewall and selinux.

Server IP: 10.250.0.2
Server Subnet: 10.250.0.0/16

Tun0 Interface IP: 10.254.254.1
Tun0 Interface Subnet: 10.254.254.0/24

Client IP: 10.0.0.1
Client Subset: 10.0.0.0/24

Here is server_tun.conf:

;local a.b.c.d
;proto tcp
proto udp
;dev tap0
dev tun
;dev-node MyTap
ca /etc/openvpn/keys/ca.crt
cert /etc/openvpn/keys/server.crt
key /etc/openvpn/keys/server.key  # This file should be kept secret
dh /etc/openvpn/keys/dh2048.pem
;topology subnet
server 10.254.254.0 255.255.255.0
ifconfig-pool-persist ipp.txt
;server-bridge 10.250.0.2 255.255.255.0 10.250.0.3 10.254.0.254
push "route 10.250.0.0 255.255.0.0"
client-config-dir ccd
route 10.0.0.0 255.255.255.0
;learn-address ./script
;push "redirect-gateway def1 bypass-dhcp"
;push "dhcp-option DNS 208.67.220.220"
client-to-client
keepalive 10 120
;tls-auth /etc/openvpn/keys/ta.key 0 # This file is secret
;cipher BF-CBC        # Blowfish (default)
;cipher AES-128-CBC   # AES
;cipher DES-EDE3-CBC  # Triple-DES
comp-lzo
;max-clients 100
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
;log         openvpn.log
;log-append  openvpn.log
verb 3
;mute 20

Here is the ccd file for the client:

[/code]
iroute 10.0.0.0 255.255.255.0
[/code]

The docs go on to say:

Next, you must set up a route on the server-side LAN gateway to route the VPN client subnet (10.8.0.0/24) to the OpenVPN server (this is only necessary if the OpenVPN server and the LAN gateway are different machines).

I'm not sure I did this right. The gateway for a GCE slice is the first ip on the subnet, which in this case is 10.250.0.1. In the network settings for the 10.250.0.0/16 (in the GCS web front-end) I added a route for both the 10.254.254.0/24 and 10.0.0.0/24 subnets, and made the "next hop" point to the OpenVPN server (presumably ip address 10.250.0.2).

In addition, in the firewall settings for the subnet I made sure port 1194 was open for any source.

When I start the server I see this:

Fri Mar 18 05:31:31 2016 OpenVPN 2.3.10 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Jan  4 2016
Fri Mar 18 05:31:31 2016 library versions: OpenSSL 1.0.1e-fips 11 Feb 2013, LZO 2.06
Fri Mar 18 05:31:31 2016 Diffie-Hellman initialized with 2048 bit key
Fri Mar 18 05:31:31 2016 Socket Buffers: R=[212992->212992] S=[212992->212992]
Fri Mar 18 05:31:31 2016 ROUTE_GATEWAY 10.250.0.1
Fri Mar 18 05:31:31 2016 TUN/TAP device tun0 opened
Fri Mar 18 05:31:31 2016 TUN/TAP TX queue length set to 100
Fri Mar 18 05:31:31 2016 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Fri Mar 18 05:31:31 2016 /usr/sbin/ip link set dev tun0 up mtu 1500
Fri Mar 18 05:31:31 2016 /usr/sbin/ip addr add dev tun0 local 10.254.254.1 peer 10.254.254.2
Fri Mar 18 05:31:31 2016 /usr/sbin/ip route add 10.0.0.0/24 via 10.254.254.2
Fri Mar 18 05:31:31 2016 /usr/sbin/ip route add 10.254.254.0/24 via 10.254.254.2
Fri Mar 18 05:31:31 2016 GID set to nobody
Fri Mar 18 05:31:31 2016 UID set to nobody
Fri Mar 18 05:31:31 2016 UDPv4 link local (bound): [undef]
Fri Mar 18 05:31:31 2016 UDPv4 link remote: [undef]
Fri Mar 18 05:31:31 2016 MULTI: multi_init called, r=256 v=256
Fri Mar 18 05:31:31 2016 IFCONFIG POOL: base=10.254.254.4 size=62, ipv6=0
Fri Mar 18 05:31:31 2016 ifconfig_pool_read(), in='ads-120elm-pfsense,10.250.0.3', TODO: IPv6
Fri Mar 18 05:31:31 2016 succeeded -> ifconfig_pool_set()
Fri Mar 18 05:31:31 2016 IFCONFIG POOL LIST
Fri Mar 18 05:31:31 2016 Initialization Sequence Completed

I don't know where that 10.254.254.2 address is coming from.

I am able to connect from the pfsense client, but I cannot ping the ip addresses of either the tun or eth interfaces from a computer on the client lan, nor can I ping any computer on the client lan from the server side, or the gateway, or the 10.254.254.4/5/6 address I see mentioned in the server's console output when the cllient connects.

Fri Mar 18 05:42:27 2016 69.115.109.60:45723 TLS: Initial packet from [AF_INET]69.115.109.62:45723, sid=34f4556a 9aae74a9
Fri Mar 18 05:42:30 2016 69.115.109.60:45723 VERIFY OK: depth=1, C=US, ST=CA, L=Newark, O=ADS, OU=MyOrganizationalUnit, CN=ads-vpn-server-1, name=EasyRSA, emailAddress=me@mymail.com
Fri Mar 18 05:42:30 2016 69.115.109.62:45723 VERIFY OK: depth=0, C=US, ST=CA, L=Newark, O=ADS, OU=MyOrganizationalUnit, CN=ads-pfsense, name=EasyRSA, emailAddress=me@mymail.com
Fri Mar 18 05:42:31 2016 69.115.109.62:45723 WARNING: 'dev-type' is used inconsistently, local='dev-type tun', remote='dev-type tap'
Fri Mar 18 05:42:31 2016 69.115.109.62:45723 WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1542', remote='link-mtu 1574'
Fri Mar 18 05:42:31 2016 69.115.109.62:45723 WARNING: 'tun-mtu' is used inconsistently, local='tun-mtu 1500', remote='tun-mtu 1532'
Fri Mar 18 05:42:31 2016 69.115.109.62:45723 Data Channel Encrypt: Cipher 'BF-CBC' initialized with 128 bit key
Fri Mar 18 05:42:31 2016 69.115.109.62:45723 Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Fri Mar 18 05:42:31 2016 69.115.109.62:45723 Data Channel Decrypt: Cipher 'BF-CBC' initialized with 128 bit key
Fri Mar 18 05:42:31 2016 69.115.109.62:45723 Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Fri Mar 18 05:42:31 2016 69.115.109.62:45723 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
Fri Mar 18 05:42:31 2016 69.115.109.62:45723 [ads-120elm-pfsense] Peer Connection Initiated with [AF_INET]69.115.109.62:45723
Fri Mar 18 05:42:31 2016 ads-120elm-pfsense/69.115.109.62:45723 MULTI_sva: pool returned IPv4=10.254.254.6, IPv6=(Not enabled)
Fri Mar 18 05:42:31 2016 ads-120elm-pfsense/69.115.109.62:45723 MULTI: Learn: 10.254.254.6 -> ads-120elm-pfsense/69.115.109.60:45723
Fri Mar 18 05:42:31 2016 ads-120elm-pfsense/69.115.109.62:45723 MULTI: primary virtual IP for ads-120elm-pfsense/69.115.109.60:45723: 10.254.254.6
Fri Mar 18 05:42:33 2016 ads-120elm-pfsense/69.115.109.62:45723 PUSH: Received control message: 'PUSH_REQUEST'
Fri Mar 18 05:42:33 2016 ads-120elm-pfsense/69.115.109.62:45723 send_push_reply(): safe_cap=940
Fri Mar 18 05:42:33 2016 ads-120elm-pfsense/69.115.109.62:45723 SENT CONTROL [ads-120elm-pfsense]: 'PUSH_REPLY,route 10.250.0.0 255.255.0.0,route 10.254.254.0 255.255.255.0,topology net30,ping 10,ping-restart 120,ifconfig 10.254.254.6 10.25
4.254.5' (status=1)

I'm only able to ping 10.254.254.6, and only from the client side.

The settings on the client side are the same as in the image in my last post:

I've tried with and without the remote networks set explicitly on the client side.

I've tried tcp instead of udp.

I've tried disabling the firewall and selinux.

I've tried the tap0 interface and server-bridge method. I was able to get the bridge-start script working by adding the default gateway at the end of the script. Unfortunately, though the docs said that computers on the server lan would be visible out of the box, they were not. As far as making the client-side subnet available to the server-side, I do not know enough about pfsense to follow that part of the docs (which is not descriptive).

Any help is much appreciated.

FuriousGeorge

(never mind...)

Traffic

FuriousGeorge wroteRe: Site-to-site OpenVPN Server on GCE: Searching for good
(never mind...)

what does this mean ?

FuriousGeorge wroteI don't know where that 10.254.254.2 address is coming from

This route is how OpenVPN handles --topology net30

--topology subnet is probably easier to configure .. See --topology in The Manual v23x

cr2025x1

If you're able to establish link between two point, then it is a routing and firewall problem.

However, the most difficult problem I had was to find what and where the problem is.
I suggest you to check each interfaces of both end with this command while ping to each other's internal subnets.

This can be used on pfSense:
tcpdump -n -i <Interface Name> 'icmp'
I don't know what command does the same thing for Linux but I think you can search it from the Internet.
With this command, you will be able to know your ping command packet reaches the interface you're listening.

If there is another computer in the client subnet with 10.0.0.2, for example, from 10.0.0.2 you may ping 10.250.0.2 If every config is done correctly, the ping request packet will be transported like this:
10.0.0.2 -> 10.0.0.1 -> 10.254.254.6(This may be skipped) -> (OpenVPN internal gateways, which may include multiple IPs with 10.254.254.xxx) -> 10.254.254.1(Server's Tun IP, also can be skipped) -> 10.250.0.2(Server's internal subnet IP)
From 10.0.0.2 to 10.250.0.2, if every routes is set correctly then you will be able to see the ping request from 10.250.0.2's interface.
If even requests are not seen from 10.250.0.2's interface via the command, then you will need to check every intermediate IPs to see if there's any problem. With the command above, you can know where the problem is.

If the route is set correctly and the requests arrive to 10.250.0.2, but if there is no reply to 10.0.0.2, the server, then it means it is a firewall problem. Firewall will have to be notified of subnets linked via OpenVPN.
The firewalls will usually not accept any packets from unknown subnets. Of course, you can use tcpdump command to know where the flow is blocked.
For example, without any settings, the Server's firewall will know 10.250.0.0/16 and 10.254.254.0/24, but it doesn't know about 10.0.0.0/24. So, even when reply packets from 10.0.0.2 arrives, the firewall will block the packet. And in this case, of course, you will see no response from 10.0.0.2. So we need to set a rule to pass packets from the source 10.0.0.0/24 to the server for the interface of 10.250.0.2 belongs.

FuriousGeorge

Traffic wrote
FuriousGeorge wroteRe: Site-to-site OpenVPN Server on GCE: Searching for good
(never mind...)
what does this mean ?

Sorry for mix up. I posted something then realized it wasn't relevant. That's all it means.

cr2025x1

And another thing.

Usually OSes dedicated to routing and firewall, such as pfSense, is set to enable IP forwarding as default.
However other common OSes' IP forwarding is usually turned off. You will need to confirm IP forwarding is online.

CentOS: https://www.centos.org/docs/5/html/Virtual_Server_Administration/s1-lvs-forwarding-VSA.html

If this option is not enabled, the system will not act as a gateway, so it will not forward any packet to its internal subnet computers.
It means that other computers in 10.250.0.0/16 except the server will not be able to reach 10.0.0.0/24.

Also, if you have access to routing table of GCE subnet's default gateway, you can set a static route and a firewall rule for the client's subnet to make other computers automatically accessible to 10.0.0.0/24.
If not, then you will have to set every each computers in GCE subnet a static route for 10.0.0.0/24.

CentOS: https://www.centos.org/docs/5/html/5.2/Deployment_Guide/s1-networkscripts-static-routes.html

FuriousGeorge

I've made some progress. Thanks guys for all the responses.

cr,

As you suspected there was a routing problem. I found a guide for setting up pfsense as a client.

First I added a new interface to pfsense tun interface, which was already there in the gui (called opt1).

Then I added a nat rule as per this tutorial:

Interface	Source	          Source Port   Destination	Destination Port	NAT Address	NAT Port	Static Port	Description	
WAN	 	127.0.0.0/8	               *	             *	                     500	WAN address	*	            YES	  Auto created rule for ISAKMP - localhost to WAN 	
 
WAN	 	127.0.0.0/8	               *                     *	                       *	WAN address	*	             NO     	Auto created rule - localhost to WAN 	

WAN	 	10.0.0.0/24	               *	               *                   500        WAN address	*	             YES	Auto created rule for ISAKMP - LAN to WAN 	

WAN	 	10.0.0.0/24	               *                       *	               *	WAN address	*                    NO	Auto created rule - LAN to WAN 	

WAN	 	10.250.0.0/16	       *	               *	            500	WAN address	*	              YES	Auto created rule for ISAKMP - OpenVPN client to WAN 	

WAN	 	10.250.0.0/16	       *	               *	               *	WAN address	*	               NO	Auto created rule - OpenVPN client to WAN 	

OPT1	 	0.0.0.0/0	               *	               *	               *	OPT1 address	*	               NO	OpenVPN NAT

(It seems to be doing something with port 500, wish I knew what.)

I can now ping the server side LAN from the client side, including both eth0 and tun0 ip addresses.

The only problem that seems to remain is that I cannot ping anything on the client-side LAN (10.0.0.0/24).

I assume it is another routing problem on the GCE side, but unfortunately I won't be able to start troubleshooting blindly again for another few hours.

That said, I wanted to update this thread to say thanks, and so that no one would spin their tires giving suggestions for stuff I might have already tried.

I'll try a few things, including setting up the br0 interface (or BSD equivalent) on the server side, and then I'll update this thread.

Thanks again.

PS I've disabled SELinux, and allowed all ports tcp and udp to pass the firewall on the GCE subnet (in the web based GCE control panel) for troubleshooting purposes.

cr2025x1

I don't recommend NAT. It will make your connection like an one-sided opaque glass. One side can see the other side, but the other side will not be able to see the former.

I assume you're NATing at the pfSense client. Then from the server's subnet side, the computers will see packets coming from 10.254.254.6, not from 10.0.0.x, even it is originally from 10.0.0.x computers.
They're responding to 10.0.0.x's ping because when they get ping requests, these packets will be seen as if they're from 10.254.254.6, due to NAT done by pfSense.
So, even you want to ping 10.0.0.x from the server's internal subnet, the computers will not be able to ping them; because they don't know where to send packets for 10.0.0.x.
If you don't want to access from 10.250.0.0/16 to 10.0.0.0/24, this is the right solution.
But I think you want 10.250.0.0/16's computers and 10.0.0.0/24's computers can see each other, so NAT shouldn't be used.

Also, you said you turned off the firewall from CentOS Linux, which is the server of OpenVPN connection, but what about pfSense's firewall? You know, pfSense also has its own firewall.
If you want to deactivate it, see System>Advanced>Firewall and NAT>Disable all packet filtering, but this will also deactivate NAT, as NAT is actually one of firewall's function.

Would you post here your route tables of each computers I listed?
1. OpenVPN Server (CentOS, am I right?)
2. OpenVPN Client (pfSense)
3. Another computer at the OpenVPN Server's side
4. Another computer at the OpenVPN Client's side
5. Default gateway of 10.250.0.0/16 at OpenVPN Server's side
6. Default gateway of 10.0.0.0/24 at OpenVPN Client's side

You can see their route tables via 'netstat -nr' command. It can be used on both FreeBSD(pfSense) and Linux(CentOS).

Also, recommended troubleshooting steps:

0.
From the server, ping 10.254.254.x(Client's OpenVPN subnet IP. It may differ.), and from the client, ping 10.254.254.1(Server's OpenVPN subnet IP, as you wrote.). Are both of these pings are responded?
- No: OpenVPN connection is not done properly. Check if setting is done correctly.
- Yes: Go to next question.

Q1-3: When both sides' firewall is turned off and no NATs:
1.
From the server, not from server's subnet's other computers, ping 10.0.0.1, and from the client, not from client's subnet's other computers, ping 10.250.0.2. Are both of these pings are responded?
- No: At least one of them doesn't know the proper route. Add correct static routes for them. (For pfSense: route add -net 10.250.0.0/16 10.0.0.5)
- Yes: Go to next question.

2.
Add a static route for subnet 10.0.0.0/24 to 10.250.0.2 at one of other computer at 10.250.0.0/16, and 10.250.0.0/16 to 10.0.0.1 at one of other computer at 10.0.0.0/24. And ping each other.
Are both of these pings are responded?
- No: At least one of the server and the client is not IP forwarding.
- Yes: Go to next question.

3.
Remove the static routes added to each computers during Question 2. Then add these static route to each subnets(10.250.0.0/16, 10.0.0.0/24)' default gateway and ping again from the same computer to each other.
Are both of these pings are responded?
- No: At least one of the routes are not added properly.
- Yes: Now both of the subnets can see each other. End of the problem. Right?

Traffic

FuriousGeorge wroteI can now ping the server side LAN from the client side, including both eth0 and tun0 ip addresses.

ok ..

FuriousGeorge wroteThe only problem that seems to remain is that I cannot ping anything on the client-side LAN (10.0.0.0/24).

Did you read the HOWTO .. like I suggested ?

HOWTO: Expanding the scope of the VPN to include additional machines

FuriousGeorge

Traffic wrote
FuriousGeorge wroteI can now ping the server side LAN from the client side, including both eth0 and tun0 ip addresses.
ok ..

FuriousGeorge wroteThe only problem that seems to remain is that I cannot ping anything on the client-side LAN (10.0.0.0/24).
Did you read the HOWTO .. like I suggested ?

HOWTO: Expanding the scope of the VPN to include additional machines

Yes, and I posted the contents of the ccd/file as well as server_tun.conf earlier. Why do you ask? Do you see something wrong?

Based on tcpdump (just started checking now) it appears icmp requests are being filtered in client side firewall. I'm looking for a setting.

Traffic

As a suggestion .. it is much easier to help if you limit one thread to one problem instead of this monolithic approach.
You are welcome to start more than one thread ..

There are so few people who try to help and so many who want help .. :geek:

FuriousGeorge

do you think i should start a new thread at this point, though?

FuriousGeorge

Using tcpdump, i see the icmp echos on the tun interface:

22:04:24.566518 IP 10.254.254.1 > 10.0.0.10: ICMP echo request, id 19492, seq 44, length 64

... but in pfsense I see nothing. I went to the nat rules and allowed all icmp traffic on the WAN interface, and I allowed all traffic on the OPT1 and OVPNC1 interfaces.

Still no luck, though I am able to ping the WAN ip address now :/

cr2025x1

The steps' questions are only valid when given condition is fulfilled.
And that's weird. You were able to make a site-to-site even when you used NAT on both sides? That is very weird.
You're talking about NATing packets outbound to WAN, right? If you're NATing packets going to VPN connection, well... It is just not possible as I know.
Well, however, if NAT is completely disabled then DHCP clients of the pfSense will not be able to reach the Internet, except the router itself.

Then I suggest another thing.

You said you can't ping to 10.0.0.1 from the server. However, the pfSense can ping the server side.
And you also said you're allowing every inbound packets. Are you saying about rules of WAN interface?
To me, it seems obvious that this is a firewall problem.
If that my guess is right, you should add the rule to LAN interface, not WAN interface, considering my experience.
And I will guess that the pfSense's 10.0.0.0/24 resides in its LAN interface, which I believe 'rl0'.
Because if I omit the whole process related to the OpenVPN tunneling, the ping request from server will first come to OpenVPN interface ovpnc1, not WAN. It will completely bypass WAN zone.
And the request packet will try to enter LAN zone, as the recipient of the packet is 10.0.0.1 and the interface with that address is in LAN zone.
But if the firewall has no rules at LAN zone to pass packets that the source of packet is 10.250.0.0/16, it will block it. I said LAN, not WAN.

https://www.dropbox.com/s/7zvsg6dpjke7y43/Screenshot_2016-03-20-13-57-26.jpg?dl=0
https://www.dropbox.com/s/67qcsa9j1u8kzxm/Screenshot_2016-03-20-14-00-03.jpg?dl=0