multi-site bridged setup sending traffic to all sites

sathackr

I have a client-server set up with static keys, tap interfaces, in bridge/tap mode(bridging two lans across a wan. Been working like this for a while. We'll call it Main Site, and Client Site 1

I'm trying to add another site (Client Site 2) to the mix. I did it the way it seemed to make sense, and I can get traffic to flow, but both remote sites see data for each other. This is a problem, because one site(Client Site 1) has a 1gb/s connection, and the other(Client Site 2) a 20mb/s connection. When the 1gb site pulls info from Main Site(which also has a 1gb/s connection), it crushes Client Site 2.

Configs below.

#Main site openvpn config
#user nobody
#group nobody
#daemon
script-security 2

dev tap
secret static.key
cipher aes-128-cbc
fragment 0
mssfix 0
tun-mtu 12000
up /etc/openvpn/bridge_up.sh
down /etc/openvpn/bridge_down.sh

#Main site bridge_up.sh
#!/bin/bash
ifconfig eth1 up
brctl addbr br0
brctl addif br0 $dev eth1
ifconfig br0 192.168.150.1 netmask 255.255.255.0
ifconfig eth1 promisc up
ifconfig $dev promisc up

#Main site bridge_down.sh
#!/bin/bash
ifconfig br0 down
brctl delbr br0
ifconfig eth1 down

#Client site 1 config
#user nobody
#group nobody
#daemon

dev tap
secret static.key
remote <removed>
cipher aes-128-cbc
fragment 0
mssfix 0
tun-mtu 12000
script-security 2
up /etc/openvpn/bridge_up.sh
down /etc/openvpn/bridge_down.sh

bridge up/bridge down for both client sites are identical to the main, just different IP addresses.

client site 2 config is identical to client site 1.

at all sites eth0 is facing the WAN, and eth1 is facing the LAN

I also tried firing up a second openvpn daemon on Main site, listening on a different port, with a separate tap interface, adding that tap interface to the bridge, with the same results.

What have I screwed up that's causing this?

Any help is much appreciated.

Traffic

sathackr wroteThis is a problem, because one site(Client Site 1) has a 1gb/s connection, and the other(Client Site 2) a 20mb/s connection. When the 1gb site pulls info from Main Site(which also has a 1gb/s connection), it crushes Client Site 2

That is one of the downsides of a TAP setup .. :geek:

sathackr

So it sounds like this is the nature of the beast?

Why wouldn't it work like a normal switch, learn the MAC addresses of the devices attached to a 'port' and only forward that traffic(and of course, broadcast, and traffic to an unknown MAC address) out that port?

I don't understand why all traffic needs to be broadcast to all bridge ports.

Thanks for the responses 🙂

Traffic

sathackr wroteWhy wouldn't it work like a normal switch

Because it is not a switch ..

sathackr

Okay.

OpenVPN encapsulates any 802.3 frames that hit the tap interface.

bridge-utils does the bridging to the ethernet adapter.

ex. br0 contains eth1 tap0 tap1

bridge-utils normally would track what mac address is on which port and send only that traffic to that port. Of course, if the mac address isn't known, then it floods the packets until it learns the mac address.

OpenVPN interferes with this by enabling promiscuous mode on the tap interface so that all frames are captured. I assume bridge-utils respects this and forwards all packets to the tap interface. Herein lies the problem.

What I meant by my question about why doesn't it work like a switch, is, why does OpenVPN need the TAP interface in promiscuous mode? If it just (dis)encapsulates the ethernet frames sent to(from) it, bridge-utils should handle everything else, treating tap0/1/2/3 as bridge ports.

Obviously this isn't the case or OpenVPN wouldn't need to put the interface in promiscuous, but I don't understand why.

Traffic

sathackr wrotebut I don't understand why.

There is your problem ...

OpenVPN is not a swiss army knife for your network .. it is a Free, Open Source VPN daemon.

sathackr

You're right, that is my problem.

Thanks for the helpful responses. Guess I'll be figuring it out on my own, at least I have a better understanding of the OpenVPN community.

Traffic

You would do well to start with the established documentation:
https://community.openvpn.net/openvpn/wiki/BridgingAndRouting

There is a lot more.

sathackr

I've spent countless hours reading OpenVPN documentation in the past, and spent a good 4-5 hours looking for information on this issue yesterday before bothering someone else with my problem. I posted that around 03:00 my time, and had been working on the issue since around 13:00 -- I am a good internet user, and have a pretty decent understanding of networking, and try to give back when I can. I read the man page, the FAQs, the how-to's, the release notes, and the change logs. I didn't come in here demanding someone solve my problem for me when I haven't even googled or RTFM, or to report a "bug" -- I just asked if this was the intended behavior, and for some help understanding why.

I found no mention of it in the documentation anywhere. The closest thing to any mention of all packets going to all tap endpoints that I could find is https://openvpn.net/index.php/download/60-open-source/faq.html#bridge2 where it mentions broadcasts must traverse the VPN -- that is a given and should be, otherwise ARP would fail. But it doesn't say all traffic and this is where my understanding breaks down as to why it must.

Also the vast majority of the documentation for bridge-mode involves client PCs connecting directly to the openvpn server -- things like DHCP, IP address assignment, etc... -- none apply in a lan-to-lan bridge. I don't need to assign IP addresses, etc... to the bridge interfaces.

Since I could find no mention of the observed behavior in any of the documentation, so I assumed I had made a configuration error, hoped that someone else had made the same mistake and might be able to save me many more hours of troubleshooting.

After seeing your response, I then shifted to trying to understand why OpenVPN behaved, or needed to behave in this manner, with a goal of modifying it to fit my needs, possibly contributing back to the OpenVPN community if I could do it good enough. I dug through tunnel.c looking for something that might set me on the right track, and will be digging further until either I have a solution or a full understanding of why a solution can't exist.

Maybe it is documented somewhere and I just can't find it, it wouldn't be the first time I've looked for something that was right under my nose and couldn't find it.

Traffic

Traffic wrote
sathackr wroteWhy wouldn't it work like a normal switch
Because it is not a switch ..

sathackr wroteI dug through tunnel.c looking for something

You are looking for something which does not exist .. You are welcome to submit such code if you have time ..

To cut a long story short .. OpenVPN is not suitable for your needs.

I was trying to help you.

You could try multiple P2P VPN bridges or something with firewalling ...
But that is way beyond the scope of this Forum.

sathackr

I have an external solution -- a multi-port ethernet card and a switch -- bridge separate ports to separate tap devices on separate bridges and connect them with an external switch. But that's a kludge that at this point seems unnecessary.

Still don't understand why bridge-utils can't do this, and I can't take 'because I said so' for an answer without explanation or understanding. All I know so far is 'because it's not a switch'

This does nothing to further my own, or anyone else's knowledge.

It appears that it has been this way for a very long time. There are discussion list posts from 2004 regarding the issue, so I'm sure there is a good reason, probably a difficult to fix reason, and when I find it, I'll know a little more about networking, linux, and programming. And honestly I'll probably know more than if someone had just told me.

I suppose I do agree though, it's outside the scope of the Server Administration -> Configuration forum at this point.

Thank you for trying to save me the trouble, but I'll find the answer myself.