I own a pocket watch. It is beautiful, but I don’t use it very often.

I know that I own a couple of watches. One of them is a battery powered solar recharging thing.

My standard “watch” today is my cell phone.

When I was in high school, I was very interested in accurate time keeping. As was my father.

This meant that we would call the “time” phone number to set our watches, at least once a week.

My grandfather had a “railroad watch”. This was a wristwatch that was approved by the railroad for time keeping. It was approved by the SooLine for use as a time keeping device. Amazing, until that model of watch was approved, the railroad required the use of pocket watches.

This was because a level of accuracy was required that only pocket watches or well regulated wristwatches could maintain.

The big thing in my youth were “quartz” watches. Instead of using a tuning fork or a mechanical balance/regulator, they used the vibrations of a quartz crystal to keep track of the time.

What this meant was that you had devices that were now able to maintain the same wrong time over an extended period of time.

The user had to set them correctly.

As an example, for years, maybe even to today, my wife would set her car clock (and many other clocks) 10 minutes fast. “So she would be on time for appointments.”

I set my car to my phone’s reported time.

One of the fun things that I did as a kid was to call up the Naval Observatory to get the current time. This was reported from their atomic clock. One of the most accurate time keeping devices in the world.

Accurate Time

Many protocols require accurate time. It is wonderful that you have a time piece that is accurate to within 1 second per year, but if it is reporting the wrong time, it is not particularly useful to the protocol.

What we want, is to know what time it is right now, and then to set our time to that.

We get the current time from a known, accurate time source. Today, that is often GPS satellites.

If you have ever wondered how GPS works, it works because your device knows where each satellite is at any instant of time. Each satellite transmits its ID and the current time. Over and over again.

That is all they do.

And here is the magic, if your device knows what time it is, and it knows where the satellite should be at his time, it can calculate the distance by comparing the difference in time.

If you are directly under a GPS satellite, it takes about 67ms for the signal to reach your device. From this, we can use the speed of light to figure out the distance traveled. Then some simple math and we know the location of your device.

We can also get accurate time by listening for the atomic clocks via radio. If you know where you are, and you know where the clock is, you can calculate the delay between the atomic clock and your device, then match your device to the atomic clock.

Today, when people want to use that type of process, they use a GPS device and get the time ticks from the device.

How long did it take?

This is where it starts to get complicated.

The standard for communications with a GPS device is 4800 or 9600 baud across a 3 wire serial connection. The protocol, the text being transmitted, specifies the time when the last character is transmitted.

That data is being received. Your device is processing it. Your device takes a certain amount of time to process the record it just received. It takes time to process that record. All of that is latency.

If you do not know the latency in your device, you do not know what time it is. For grins, just think of that serial link being 300,000,000 meters long. That would put a 1-second latency by itself.

There are ways of calculating the latency, but I do not remember what they are.

Latency is the important piece of information.

Calculating Latency

Many network people have run ping. It is a tool for testing reachability and latency between your device and some other device on the Internet.


ping -c 5 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=116 time=11.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=116 time=11.1 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=116 time=11.6 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=116 time=11.1 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=116 time=11.0 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 11.022/11.179/11.616/0.220 ms


This is a test from one of my faster machines to a Google DNS server. This tells me that it takes 11.179 ms to reach that DNS server. Testing to one of my network timeservers, the average is 78.094 ms.

This means that the time reported by the timeserver will be off by some amount. In a simple world, we would guess that it is 1/2 of 78.094.

But, I use NTP. NTP does many transmissions to multiple timeservers to discover the actual time. It is reporting that the latency is 78.163512 ms. A little more accurate. It tells me that the dispersion is 0.000092 ms.

How does it know this? Because of many samples and because of four different time stamps.

When my device sends an NTP request packet, it puts the current time in it. When the server receives the packet, it puts the current time in it. When the server transmits the response, it adds the current time again. Finally, when the reply is received, the current time is added to the packet. This gives us four different time stamps from two different sources.

We compute the total latency via mine(R)-mine(S). We know the processing time by server(S)-server(R). The difference between server(R)-mine(S) and mine(R)-server(S) as the symmetry between the two paths the request and response traveled.

From these values, we can calculate the network distance, in seconds, between us and them.

Assume we transmit at time 0(M), it is received at 100(S), the response is transmitted at 105(S) and we receive it at 78(M).

How can we receive our reply before the server sent it? Easy, we have to different views of what time it is.

The latency is 78. This means that the halfway point was at 38. It took 5 to process the reply and get it on the wire again. If we do simple stuff, this means that our time is off from their time by 67.

But we can do better. By looking at the reported latency between the two legs, we can actually calculate how long it took for us to receive the reply.

NTP uses multiple timeservers to get a consensus as to the time. It monitors each timeserver to determine which one jitters the least.

All of this means that we can have very accurate times.

And having accurate measurements of the time, NTP will calculate how much the computer’s clock drifts over time. It will then modify the clock rate in parts per million to get the drift as close to zero as possible.

This means, that the longer your device runs NTP, the more accurate it becomes.

The Internet is a fantastic creature. I’m not speaking of the information you can find on the internet. Nor am I speaking of the entertainment that is available on the Internet.

The mere fact that you can ask for information at your desk or on your phone and somehow that request gets there, and the response gets back, is mind bogglingly complex.

Here is the dirty little secret about computers. It is all zeros and ones. There are no pictures, there are no videos, there are no songs nor even text, it is all zeros and ones.

We group these zeros and ones into units of different sizes. The three primary sizes are 8, 32, and 64, with a spattering of 16. At the lowest level, we think about these in groups of 8, called octets.

You might know them as “Bytes”.

Now, zeros and ones are a bit difficult to read and write. So we use base 16 to read and write bytes.

Base 16 has 16 digits, just like base 10 has 10 digits. 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9. are the digits of base 10.

For base 16, we add A, B, C, D, E, and F as the six extra digits.

So we have a 32-bit number that looks like this: 4C4F5645 in hex (base 16) and 1280267845 in base 10, and “LOVE” as ASCII.

It is all zeros and ones. It takes meaning when we decide how those bits will be interpreted.

When you ask Google to search for “The Vine of Liberty”, your browser starts with a name, which it needs to convert to an address. The name is “www.google.com”. Depending on where you are, one of the addresses will be 142.250.69.68.

This is a different representation of a 32-bit word. In this “dotted quad”, each number represents the decimal value of an 8-bit byte.

For you, the simple household, your device asks, “How can I get this message to 142.250.69.68?”

Your device looks up the address in the “routing table”. Your device likely only has a single entry in the routing table. The route of last resort, or default route.

When no other table entries match, then send the request to a default router

A router has a single job, to move packets (requests and responses) from one network to another. When your default router receives your device’s request, it looks up the IP address (142.250.69.68) in its routing table. Again, it is likely that there is only a single entry in that table, the default route.

This is the simple way that things work in simple networks. It continues to work until the moment when a router has to make a choice. Does it send the packet from network H (your home network) to network A or to network B.

That router will have a routing table. It will find a match for 142.250.69.68 in that table, which will tell that router which network to forward your request to.

If nothing about the Internet ever changed, that would be all that was needed. Every router would know how to get to every address and that would be it.

But it isn’t that easy. The Internet changes, constantly. This means that we need to be able to change those routing tables quickly and easily.

The answer to that issue is a routing protocol. The oldest was RIP. It doesn’t work well today as it sends too much data too often. Back in slower networking times, RIP was taking up nearly 70% of my bandwidth. We stopped that.

There are two major types of routing protocols, external and internal. The primary external protocol, today, is the Border Gateway Protocol, or BGP. I don’t have to worry about that.

What I do need to worry about is internal routing. For internal routing, I use a combination of static routes and OSPF.

And this is where it gets complex. The data center has two physical networks. A management network and a production network.

The management network runs on a single subnet, with each host having a unique address on that subnet.

The production network runs on multiple subnets, each subnet serving to isolate problems. In addition, traffic on the production network needs to be able to reach the Internet.

The management network requires zero routing. One network space. No connection to the outside world.

On top of the physical network are layered multiple other networks. There is the OVN NAS network. This is how each of the hypervisors gets access to block storage (and shared file systems). There is the OVN NAS data network. There is the OVN VM network, the container network.

In addition, there are other networks used inside the container environment.

Some of these networks exist in isolation. Others are used as transport networks. No traffic originates nor terminates in these transport networks.

But other networks need to be able to speak to each other.

That means that every device needs to know how to reach every address. This means that OSPF is doing magic all the time to make things work.

Why? Redundancy. Every device has at least two paths to the next hop. If the primary link fails, the secondary link takes over.

This is done by rebuilding the routing table.

OVN links don’t fail (unless the idiot driving the keyboard does something stupid). The physical network can fail. When this happens, OVN just routes the tunnels in different directions.

So why this rant?

Because I can’t get parts of this to work!

My need is to move the containers into the OVN.

And I can’t get routing to work consistently. ARGH!

Oh well. Filler done.