pfSense OVPN serv on PrivateVLAN Community PVLAN doesn't wor

bk

OpenVPN Experts - please help!

For the past week we have been trying to understand why OpenVPN packets terminating at pfSense 2.2.6 (or earlier) firewall's Site-to-Site or RA TUN interface are not routed to firewall's LAN interface.
Here is the server config:

dev ovpns2
verb 1
dev-type tun
dev-node /dev/tun2
writepid /var/run/openvpn_server2.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-128-CBC
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local *.*.*.19
ifconfig 10.1.8.1 10.1.8.2
lport 1195
management /var/etc/openvpn/server2.sock unix
push "route 172.31.5.0 255.255.255.0"
route 192.168.0.0 255.255.0.0
route 10.0.0.0 255.0.0.0
route 172.16.0.0 255.240.0.0
secret /var/etc/openvpn/server2.secret 
comp-lzo adaptive
push "route 10.131.0.0 255.255.0.0"

Here is the client config:

dev tun
persist-tun
persist-key
proto udp
cipher AES-128-CBC
auth SHA256
pull
resolv-retry infinite
remote *.*.*.19 1195
route 10.131.0.0 255.255.0.0
ifconfig 10.1.8.2 10.1.8.1
keepalive 10 60
ping-timer-rem
secret pfSense-udp-1195.secret
comp-lzo

The pfSense firewall's LAN interface is on a Community PVLAN provided by Cisco Nexus 1000V. All packets initiated from the PVLAN to the remote host go over the VPN tunnel to the host, which replies back, the pfSense sees the reply (recorded with packet capture) on its LAN interface coming from remote tunnel IP to the host on the PVLAN but the packets are never received at that host.
We have a ticket open with Cisco but they point to the pfSense being the culprit.

**Any IPSec tunnel to pfSense's WAN interface however works as expected and the hosts on the Community PVLAN are reachable over the tunnel.**

***The interesting part is that if the pfSense's LAN port is moved to regular VLAN the hosts on that VLAN are reachable over the OpenVPN tunnel.
So the issue occurs only when the FW's LAN and hosts are on Community PVLAN (e.g. community PVLAN=1102)... ***

Here are more details:
When PVLAN host with 10.131.102.17 IP pings a remote host with 192.168.1.182 IP over the VPN tunnel the reply packets are returned according to the pfSense FW's LAN port per the packet capture:

11:27:37.975400 IP 10.131.102.17 > 192.168.1.182: ICMP echo request, id 1, seq 1238, length 40
11:27:37.982019 IP 192.168.1.182 > 10.131.102.17: ICMP echo reply, id 1, seq 1238, length 40
11:27:42.546363 IP 10.131.102.17 > 192.168.1.182: ICMP echo request, id 1, seq 1239, length 40
11:27:42.557894 IP 192.168.1.182 > 10.131.102.17: ICMP echo reply, id 1, seq 1239, length 40
11:27:47.546806 IP 10.131.102.17 > 192.168.1.182: ICMP echo request, id 1, seq 1240, length 40
11:27:47.569788 IP 192.168.1.182 > 10.131.102.17: ICMP echo reply, id 1, seq 1240, length 40
11:27:52.547020 IP 10.131.102.17 > 192.168.1.182: ICMP echo request, id 1, seq 1241, length 40
11:27:52.570625 IP 192.168.1.182 > 10.131.102.17: ICMP echo reply, id 1, seq 1241, length 40

However on the Cisco Nexus 1000V switch ports only ICPM request packets are seen using the follows capture settings:
********************************
LTL : 49 (PVLAN host's port)
LTL : 53 (pfSense FW's LAN's port)
********************************
uswnex1# module vem 3 execute vempkt show capture info
Stage : Ingress
LTL : 53
VLAN : Unspecified
Filter : ip proto 1
Stage : Egress
LTL : 53
VLAN : Unspecified
Filter : ip proto 1
Stage : Drop
LTL : 53
VLAN : Unspecified
Filter : Unspecified
********************************

uswnex1# module vem 3 execute vempkt display detail all
********************** Entry 1 *************************

------Packet Entry Information------
Timestamp : Apr 06 11:52:10.231428
Packet Entry : 1
CPU : 7
Bytes Captured : 74

------Packet Length Information------
Packet Length : 74
Packet Buffer Length : 74
Packet Mapped Length : 74

------SF Packet Information------
Capture Stage : Egress
SF Packet Flags : Original
Source LTL : 49
Destination LTL : 53
HWBD : 19
Vlan/SegID : 1102

------Packet L3 Header Information------
Source IP Address : 10.131.102.17
Destination IP Address : 192.168.1.182
IP Protocol Type: 1

------Packet L2 Header Information------
Source MAC Address : *:*:*:b7:1e:7c
Destination MAC Address : *:*:*:b7:1e:20
Type : 2048

------Packet Platform Information------
NBL : 0xFFFFE00154E27AA0
Source : 2 - 0
Send Flags : 0x0
VMQ Queue ID : 0
NBL Type : 0
NBL Checksum Info : 0x220011
NBL 802.1Q Info : 0x0
Native Forwarding : 0x0
Virtual Subnet Id : 0x0
NBL Direction : Ingress
Dest Count : 1
Current Dest : 10 - 0
Dest Flags : 0

*********************************************************
********************** Entry 2 *************************

------Packet Entry Information------
Timestamp : Apr 06 11:52:14.841057
Packet Entry : 2
CPU : 7
Bytes Captured : 74

------Packet Length Information------
Packet Length : 74
Packet Buffer Length : 74
Packet Mapped Length : 74

------SF Packet Information------
Capture Stage : Egress
SF Packet Flags : Original
Source LTL : 49
Destination LTL : 53
HWBD : 19
Vlan/SegID : 1102

------Packet L3 Header Information------
Source IP Address : 10.131.102.17
Destination IP Address : 192.168.1.182
IP Protocol Type: 1

------Packet L2 Header Information------
Source MAC Address : *:*:*:b7:1e:7c
Destination MAC Address : *:*:*:b7:1e:20
Type : 2048

------Packet Platform Information------
NBL : 0xFFFFE00154E27AA0
Source : 2 - 0
Send Flags : 0x0
VMQ Queue ID : 0
NBL Type : 0
NBL Checksum Info : 0x220011
NBL 802.1Q Info : 0x0
Native Forwarding : 0x0
Virtual Subnet Id : 0x0
NBL Direction : Ingress
Dest Count : 1
Current Dest : 10 - 0
Dest Flags : 0

*********************************************************
********************** Entry 3 *************************

------Packet Entry Information------
Timestamp : Apr 06 11:52:19.841311
Packet Entry : 3
CPU : 0
Bytes Captured : 74

------Packet Length Information------
Packet Length : 74
Packet Buffer Length : 74
Packet Mapped Length : 74

------SF Packet Information------
Capture Stage : Egress
SF Packet Flags : Original
Source LTL : 49
Destination LTL : 53
HWBD : 19
Vlan/SegID : 1102

------Packet L3 Header Information------
Source IP Address : 10.131.102.17
Destination IP Address : 192.168.1.182
IP Protocol Type: 1

------Packet L2 Header Information------
Source MAC Address : *:*:*:b7:1e:7c
Destination MAC Address : *:*:*:b7:1e:20
Type : 2048

------Packet Platform Information------
NBL : 0xFFFFE0015AF52B60
Source : 2 - 0
Send Flags : 0x0
VMQ Queue ID : 0
NBL Type : 0
NBL Checksum Info : 0x220011
NBL 802.1Q Info : 0x0
Native Forwarding : 0x0
Virtual Subnet Id : 0x0
NBL Direction : Ingress
Dest Count : 1
Current Dest : 10 - 0
Dest Flags : 0

*********************************************************
********************** Entry 4 *************************

------Packet Entry Information------
Timestamp : Apr 06 11:52:24.841585
Packet Entry : 4
CPU : 7
Bytes Captured : 74

------Packet Length Information------
Packet Length : 74
Packet Buffer Length : 74
Packet Mapped Length : 74

------SF Packet Information------
Capture Stage : Egress
SF Packet Flags : Original
Source LTL : 49
Destination LTL : 53
HWBD : 19
Vlan/SegID : 1102

------Packet L3 Header Information------
Source IP Address : 10.131.102.17
Destination IP Address : 192.168.1.182
IP Protocol Type: 1

------Packet L2 Header Information------
Source MAC Address : *:*:*:b7:1e:7c
Destination MAC Address : *:*:*:b7:1e:20
Type : 2048

------Packet Platform Information------
NBL : 0xFFFFE00154E27AA0
Source : 2 - 0
Send Flags : 0x0
VMQ Queue ID : 0
NBL Type : 0
NBL Checksum Info : 0x220011
NBL 802.1Q Info : 0x0
Native Forwarding : 0x0
Virtual Subnet Id : 0x0
NBL Direction : Ingress
Dest Count : 1
Current Dest : 10 - 0
Dest Flags : 0
*********************************************************
uswnex1#

bk

OpenVPN experts - please help!

In our Hyper-V deployment there is a very strange issue with a OpenVPN server on a virtual pfSense FW when its LAN port is on a Community PVLAN and is the default gateway for all hosts on the same Community PVLAN.
All PVLANs are provided by Cisco Nexus 1000V switch. The regular VLANs are on the Hyper-V switch.

Everything else works as expected, except the OpenVPN site-to-site or RA client tunnels where all packets coming from the OpenVPN tunnel interface to LAN interface simply vanish...

Note:
1. The problem goes away if the pfSense's LAN interface and nodes behind it (Hyper-V VMs) are moved to any regular VLAN.
2. The IPSec site-to-site tunnels do work as expected in this Hyper-V/Cisco PVLAN environment.
3. The pfSense is unaware of any VLANs

Here is OpenVPN server config:


dev ovpns2
verb 1
dev-type tun
dev-node /dev/tun2
writepid /var/run/openvpn_server2.pid
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-128-CBC
auth SHA256
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local *.*.*.19
ifconfig 10.1.8.1 10.1.8.2
lport 1195
management /var/etc/openvpn/server2.sock unix
route 192.168.0.0 255.255.0.0
route 10.0.0.0 255.0.0.0
route 172.16.0.0 255.240.0.0
secret /var/etc/openvpn/server2.secret 
comp-lzo adaptive
push "route 172.31.5.0 255.255.255.0"
push "route 10.131.0.0 255.255.0.0"

... here is client config:

dev tun
persist-tun
persist-key
proto udp
cipher AES-128-CBC
auth SHA256
pull
resolv-retry infinite
remote *.*.*.19 1195
route 10.131.0.0 255.255.0.0
ifconfig 10.1.8.2 10.1.8.1
keepalive 10 60
ping-timer-rem
secret pfSense-udp-1195.secret
comp-lzo

We have a ticket open with Cisco but they are pointing to the pfSense being the culprit...

According to the pfSense packet capture on the LAN port, the ICMP packets sent from a VM (10.131.102.17) on the PVLAN are sent over the S2S OpenVPN tunnel to remote client (192.168.1.182) and the replays are sent back, but the VM never receives them:

13:15:03.330330 00:1d:d8:b7:1e:7c > 00:1d:d8:b7:1e:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 18320, offset 0, flags [none], proto ICMP (1), length 60)
10.131.102.17 > 192.168.1.182: ICMP echo request, id 1, seq 1765, length 40
13:15:03.336943 00:1d:d8:b7:1e:20 > 00:1d:d8:b7:1e:7c, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 20219, offset 0, flags [none], proto ICMP (1), length 60)
192.168.1.182 > 10.131.102.17: ICMP echo reply, id 1, seq 1765, length 40
13:15:08.315107 00:1d:d8:b7:1e:7c > 00:1d:d8:b7:1e:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 18321, offset 0, flags [none], proto ICMP (1), length 60)
10.131.102.17 > 192.168.1.182: ICMP echo request, id 1, seq 1766, length 40
13:15:08.321571 00:1d:d8:b7:1e:20 > 00:1d:d8:b7:1e:7c, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 20220, offset 0, flags [none], proto ICMP (1), length 60)
192.168.1.182 > 10.131.102.17: ICMP echo reply, id 1, seq 1766, length 40
13:15:13.315482 00:1d:d8:b7:1e:7c > 00:1d:d8:b7:1e:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 18322, offset 0, flags [none], proto ICMP (1), length 60)
...

However no reply packets are received at the Cisco N1KV switch port (LTL=53) where the pfSense is connected to.

nex1# module vem 3 execute vempkt show capture info
Stage : Ingress
LTL : 53
VLAN : Unspecified
Filter : ip proto 1
Stage : Egress
LTL : 53
VLAN : Unspecified
Filter : ip proto 1
Stage : Drop
LTL : 53
VLAN : Unspecified
Filter : Unspecified

nex1#
nex1# module vem 3 execute vempkt start
nex1# module vem 3 execute vempkt stop
Will suspend log after next 0 entries
nex1# module vem 3 execute vempkt stop
Suspended log
nex1#
nex1# module vem 3 execute vempkt display detail all
********************** Entry 1 *************************

------Packet Entry Information------
Timestamp : Apr 06 13:15:02.602002
Packet Entry : 1
CPU : 2
Bytes Captured : 74

------Packet Length Information------
Packet Length : 74
Packet Buffer Length : 74
Packet Mapped Length : 74

------SF Packet Information------
Capture Stage : Egress
SF Packet Flags : Original
Source LTL : 49
Destination LTL : 53
HWBD : 19
Vlan/SegID : 1102

------Packet L3 Header Information------
Source IP Address : 10.131.102.17
Destination IP Address : 192.168.1.182
IP Protocol Type: 1

------Packet L2 Header Information------
Source MAC Address : 00:1d:d8:b7:1e:7c
Destination MAC Address : 00:1d:d8:b7:1e:20
Type : 2048

------Packet Platform Information------
NBL : 0xFFFFE0015B08BB60
Source : 2 - 0
Send Flags : 0x0
VMQ Queue ID : 0
NBL Type : 0
NBL Checksum Info : 0x220011
NBL 802.1Q Info : 0x0
Native Forwarding : 0x0
Virtual Subnet Id : 0x0
NBL Direction : Ingress
Dest Count : 1
Current Dest : 10 - 0
Dest Flags : 0

Payload :
00000: 00 1d d8 b7 1e 20 00 1d d8 b7 1e 7c 08 00 45 00
00016: 00 3c 47 8f 00 00 80 01 00 00 0a 83 66 11 c0 a8
00032: 01 b6 08 00 46 77 00 01 06 e4 61 62 63 64 65 66
00048: 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76
00064: 77 61 62 63 64 65 66 67 68 69
*********************************************************
********************** Entry 2 *************************

------Packet Entry Information------
Timestamp : Apr 06 13:15:03.617679
Packet Entry : 2
CPU : 7
Bytes Captured : 74

------Packet Length Information------
Packet Length : 74
Packet Buffer Length : 74
Packet Mapped Length : 74

------SF Packet Information------
Capture Stage : Egress
SF Packet Flags : Original
Source LTL : 49
Destination LTL : 53
HWBD : 19
Vlan/SegID : 1102

------Packet L3 Header Information------
Source IP Address : 10.131.102.17
Destination IP Address : 192.168.1.182
IP Protocol Type: 1

------Packet L2 Header Information------
Source MAC Address : 00:1d:d8:b7:1e:7c
Destination MAC Address : 00:1d:d8:b7:1e:20
Type : 2048

------Packet Platform Information------
NBL : 0xFFFFE001539B2B60
Source : 2 - 0
Send Flags : 0x0
VMQ Queue ID : 0
NBL Type : 0
NBL Checksum Info : 0x220011
NBL 802.1Q Info : 0x0
Native Forwarding : 0x0
Virtual Subnet Id : 0x0
NBL Direction : Ingress
Dest Count : 1
Current Dest : 10 - 0
Dest Flags : 0

Payload :
00000: 00 1d d8 b7 1e 20 00 1d d8 b7 1e 7c 08 00 45 00
00016: 00 3c 47 90 00 00 80 01 00 00 0a 83 66 11 c0 a8
00032: 01 b6 08 00 46 76 00 01 06 e5 61 62 63 64 65 66
00048: 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76
00064: 77 61 62 63 64 65 66 67 68 69
*********************************************************
********************** Entry 3 *************************

------Packet Entry Information------
Timestamp : Apr 06 13:15:08.602318
Packet Entry : 3
CPU : 2
Bytes Captured : 74

------Packet Length Information------
Packet Length : 74
Packet Buffer Length : 74
Packet Mapped Length : 74

------SF Packet Information------
Capture Stage : Egress
SF Packet Flags : Original
Source LTL : 49
Destination LTL : 53
HWBD : 19
Vlan/SegID : 1102

------Packet L3 Header Information------
Source IP Address : 10.131.102.17
Destination IP Address : 192.168.1.182
IP Protocol Type: 1

------Packet L2 Header Information------
Source MAC Address : 00:1d:d8:b7:1e:7c
Destination MAC Address : 00:1d:d8:b7:1e:20
Type : 2048

------Packet Platform Information------
NBL : 0xFFFFE001539B2B60
Source : 2 - 0
Send Flags : 0x0
VMQ Queue ID : 0
NBL Type : 0
NBL Checksum Info : 0x220011
NBL 802.1Q Info : 0x0
Native Forwarding : 0x0
Virtual Subnet Id : 0x0
NBL Direction : Ingress
Dest Count : 1
Current Dest : 10 - 0
Dest Flags : 0

Payload :
00000: 00 1d d8 b7 1e 20 00 1d d8 b7 1e 7c 08 00 45 00
00016: 00 3c 47 91 00 00 80 01 00 00 0a 83 66 11 c0 a8
00032: 01 b6 08 00 46 75 00 01 06 e6 61 62 63 64 65 66
00048: 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76
00064: 77 61 62 63 64 65 66 67 68 69
*********************************************************
********************** Entry 4 *************************

------Packet Entry Information------
Timestamp : Apr 06 13:15:13.602580
Packet Entry : 4
CPU : 3
Bytes Captured : 74

------Packet Length Information------
Packet Length : 74
Packet Buffer Length : 74
Packet Mapped Length : 74

------SF Packet Information------
Capture Stage : Egress
SF Packet Flags : Original
Source LTL : 49
Destination LTL : 53
HWBD : 19
Vlan/SegID : 1102

------Packet L3 Header Information------
Source IP Address : 10.131.102.17
Destination IP Address : 192.168.1.182
IP Protocol Type: 1

------Packet L2 Header Information------
Source MAC Address : 00:1d:d8:b7:1e:7c
Destination MAC Address : 00:1d:d8:b7:1e:20
Type : 2048

------Packet Platform Information------
NBL : 0xFFFFE0015B08BB60
Source : 2 - 0
Send Flags : 0x0
VMQ Queue ID : 0
NBL Type : 0
NBL Checksum Info : 0x220011
NBL 802.1Q Info : 0x0
Native Forwarding : 0x0
Virtual Subnet Id : 0x0
NBL Direction : Ingress
Dest Count : 1
Current Dest : 10 - 0
Dest Flags : 0

Payload :
00000: 00 1d d8 b7 1e 20 00 1d d8 b7 1e 7c 08 00 45 00
00016: 00 3c 47 92 00 00 80 01 00 00 0a 83 66 11 c0 a8
00032: 01 b6 08 00 46 74 00 01 06 e7 61 62 63 64 65 66
00048: 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76
00064: 77 61 62 63 64 65 66 67 68 69
*********************************************************

bk

*BUMP*

Can't believe no one tried using OpenVPN tun/tap to Priviate VLAN (PVLAN)?!

http://www.cisco.com/c/en/us/tech/lan-switching/private-vlans-pvlans-promiscuous-isolated-community/index.html

PVLANs provide layer 2 isolation between ports within the same broadcast domain. There are three types of PVLAN ports:

Promiscuous - A promiscuous port can communicate with all interfaces, including the isolated and community ports within a PVLAN.
Isolated - An isolated port has complete Layer 2 separation from the other ports within the same PVLAN, but not from the promiscuous ports. PVLANs block all traffic to isolated ports except traffic from promiscuous ports. Traffic from isolated port is forwarded only to promiscuous ports.
Community - Community ports communicate among themselves and with their promiscuous ports. These interfaces are separated at Layer 2 from all other interfaces in other communities or isolated ports within their PVLAN.

Traffic

bk wroteIn our Hyper-V deployment there is a very strange issue with a OpenVPN server on a virtual pfSense FW when its LAN port is on a Community PVLAN and is the default gateway for all hosts on the same Community PVLAN.
All PVLANs are provided by Cisco Nexus 1000V switch. The regular VLANs are on the Hyper-V switch.

This is not the forum you are looking for .. :ugeek:

Try the mailing list.