The problem that people have been attempting to solve, for years, is the lack of space in the IPv4 addresses space.
There are currently more devices attached to the Internet or “the network” than there are addresses in the IPv4 space. This requires address overlap.
The smallest section of a network is the “subnet”. A subnet can hold anywhere from 2 to over a million devices.
Consider a small business network. They have three networks, a network that is connected to the Internet, labeled DMZ, a network for the security cameras, labeled CCTV, and the working network, labeled Internal.
They have a router between the Internal network and the DMZ. There is another router that takes traffic from the DMZ and transfers it to the Internet.
The CCTV network does not need to ever touch the DMZ network, nor does it really need to touch the Internal network. So they run a completely separate physical network so that CCTV traffic is never available on the Internal or DMZ networks.
This could become costly. Consider a situation where you need to connect multiple buildings. Maybe some of those buildings can be connected with fiber, but others are using radio links. Radio links are expensive.
The traffic is low enough that there is no justification for a second radio link. Besides, it is difficult to run two radio links side-by-side.
The solution that was implemented is the Virtual LAN, or VLAN.
When you define a VLAN, you set a tag in the Ethernet frame, identifying which VLAN this frame belongs to. Now, we can put all the CCTV traffic on a VLAN and use the same physical network as we use for the Internal network. All is good.
This isn’t a complete solution, it is possible to configure a network card to listen to a particular VLAN, even if that device isn’t supposed to be on the VLAN. It is also another configuration point which smaller devices might not support.
As an example, I’ve never found a method to put my cell phone on a particular VLAN. It is likely possible, I’ve just never found it.
Same with my CCTV cameras. They exist only on the default, untagged, network.
One of the very nice parts of using a VLAN, is that you can have overlapping address space. I can have 192.168.88.0/24 on the physical network and 192.168.89.0/22 on the same physical network but with a VLAN tag of 87. They are overlapping address spaces, but they do not interfere with each other.
The solution was to allow a L2 switch port to be tagged. Now, by device which only uses the untagged frame can be plugged into a tagged port. All traffic coming from that port will have a VLAN tag added to it. All traffic sent to that port will have the VLAN tag stripped from it.
This means that a CCTV device sends and receives on the default (no tag) network. It reaches the switch and the packet is now on a VLAN. Another device on the Internal network is also on the same VLAN. That device, a monitoring station, can now see the CCTV footage.
If a port receives a frame that is tagged, it drops the frame. This keeps VLANs from leaking from their approved segment.
If there is a need for a port to accept multiple VLANs, it is configured as a trunk.
Thursday, I attempted to move ceph to an OVN network. This would eliminate the need for a VLAN and would give me a single subnet across multiple physical subnets. It failed.
Friday, I attempted to put a new L2 switch into place. The good news was that I didn’t need to break my entire network to do the testing.
The test computer has two NIC’s. One is connected to the management physical network. The other to the back plane network. I was able to establish a connection to the management port.
Once there, I could establish that I had full bandwidth to other nodes on the physical network, using the physical subnet. I could even reach multiple subnets using that same interface.
Then I tried the VLAN. The VLAN failed. There was no network traffic passing through.
It also looks like they do not have a large enough MTU.
Conclusion
I’m still black boxing this thing. It has been a painful trip. I have more than a few more tests to run. It is just overly painful trying to get there.