bitblit
I have a server setup that works but has an odd quirk. I've been researching it on my vacation time and think I might have solved it, though I won't know until I get back. I'd like to describe my setup and get advice on whether it's sound or needs improvement. Sorry about the vagueries. This is a proprietary setup so I can't state specifics. For my industry, I'm using OpenVPN in a relatively novel and non traditional manner, giving users capability they never had. I plan on submitting a white paper detailing the setup to a trade journal eventually.
I have one server that hosts a paired sets of proprietary virtual machines. I deploy them in pairs. They work together. Each pair comprises one system. I deploy as many systems as I need to meet individual users needs. Currently, 10 but will soon grow to 20. VM "A" and "B" are similar but have different nonroutable 172.16.x.x IP's. However all the "A"a are identical and all the "B"s are identical. In other words, I have 10 identical "A"s all with 172.16.0.1 addresses and 10 identical "B"s with 172.16.1.1 addresses. Each of the paired virtual machine systems is hard coded with nonroutable 172.16.x.x addresses. None of these VM's can and should access anything beyond its private network. Each private network is isolated. VM "A" can only access it's "B" counterpart and any tools used by the user "bound" to that deployed system. Same with the "B". Neither can access the VMs in the other private (but identical) networks.
For each deployed system, I have the following configuration:
bridgeN
tapN_openvpn <- OpenVPN
tapN_vm_a <- VM A
tapN_vm_b <- VM B
(I'm on vacation and going from memory...)
N: ID number of the deployed system, so TAP and bridge names are unique. I have a strategy for setting bridge and TAPs IPs uniquely. They aren't accessed directly by the VMs, but I wasn't sure if they had to be unique. I thought it would also make debugging easier since I can ping them and possibly setup tools to poll network metrics.
bridge:
- has a unique ip on the same 172.16 subnet
- MAC not set (I think this is source of quirk)
tap_openvpn:
- has a unique IP on the same 172.16 subnet
- MAC not set
tap_vm_a:
- has a unique IP on the same 172.16 subnet
- MAC not set
- VM A MAC is set but different than the TAP. All VM A MACs are identical.
tap_vm_b
- has a unique IP on the same 172.16 subnet
- MAC not set
- VM B MAC is set but different than the TAP. All VM B MACs are identical.
I just multiply this for each deployed system.
I have a separate OpenVPN config files for each deployed system with the correct IPs, separate log files, and separate server ports. Tunnels are compressed and encrypted over TCP. I "bind" users to a deployed system by giving them the correct config file for their OpenVPN client.
Things work, but there's an odd quirk. Usually it doesn't work right away. Often I can see ARP requests go out to the VM TAP and can see the response with Wireshark on the VM TAP, but it doesn't appear on the bridge. Sometimes it's A. Sometimes it's B. Simetimes it's the remote client. I couldn't figure out what the trigger was, but with enough random thrashing about, it would start working. I've only used one VM at a time so far ( the VMs dove have that interconnectivity yet), but I now suspect when it does work, the other one doesn't. And when ARP works on the remote client but not the active VM, it would work with the other VM. Sometimes I don't do anything and it just magically works. Sometimes i set ping to loop, scratching my head in bewilderment, and it magically goes from network inaccessible to working.
I think the quirk is due to the fact that the bridge takes the IP of the attached device with the lowest MAC value. I think it works when the randomly generated MACs result in the bridge taking the MAC of the currently unused VM TAP. I think the simplest fix is to add a dummy TAP with a low MAC, letting the bridge take that MAC immediately and ensuring it doesn't reconfigure itself.
Also, I think if I set the VM TAP MACs the same as the VM itself, disable TAP ARP proxy, I can disable promiscuous mode to improve performance.
Advice?
bitblit
I think I fixed the problem with the proposed solution. When I came back from vacation, I tried it and haven't seen this quirk yet. I fixed it by adding a dummy tap and setting the Mac address very low to ensure its the lowest Mac connected to the bridge. The bridge then takes on that Mac addresses and never changes.
