Nerd Babel

I have hundreds of dollars worth of GPS equipment. Not counting the cell phones we all carry with us.

I wanted to try to create a Stratum 0 NTP clock.

The last time I attempted this, I used a Garmin handheld GPS. Time to sync was in minutes and while the power draw as trivial, by the standards of the day, it would still burn through AA batteries.

Because you, kind readers, told me that there were cheap options, I went looking.

What I found was a GPS module that is about an inch square. For $15 I could have one delivered. It comes with a header containing VCC, GND, TXD, RXD, and PPS. I figured I could solder in the provided header then run them to a GPIO that has an attached UART.

Well, the darn things showed up a day early, and I didn’t really want to do any soldering. I plugged it in via the USB port, put it in the window. A few minutes later, it had a hard lock.

After installing gpsd and configuring, chrony I now have a system that is locked at less than 1ms accuracy, NOT using the PPS option.

That will be next week’s project. Getting that PPS signal to the motherboard.

If I had a Raspberry Pi with a good interface, not wifi, I can see that this would make a darn nice little timekeeper.

My wife read my article on passwords and “got it”. Which is nice. I was attempting to explain how password crackers use rule sets to modify input dictionaries to create more guesses from a single word list.

I decided to see how much things have advanced. To say I was shocked would be an understatement.

In 2013, the game “Battlefield” was hacked and the entire password database was captured.

This is not the major security threat you might instantly leap to, but it is bad.

Stealing Passwords

I worked in the Systems Group at my University. We were tasked with all software maintenance, installations, upgrades, and in house improvements to the operating system.

The systems group had taken the original manufacturer’s operating system and extended it to the point where it was no longer the same operating system. Having done this, we gave back all the code we had written to the manufacturer, who incorporated what they liked into their next release.

We had developed a long term backup plan. This plan was three tiered. We took daily backups of the entire file system. This was a rolling tape backup. There were 30 days of daily backups performed before the first tape was overwritten.

We also performed weekly backups. There were 52 weeks of weekly backups. So a total of 82 backup sets.

In addition to this, we did end of term backups. These were done just after the term ended. These tapes were kept.

What this meant was that if your file were to live for at least 24 hours, you would be able to recover to any particular day in the past 5 weeks of your file.

If your file were to exist over a weekend, you could recover that file to how it was on the weekend it was dumped for the past year. And if your file were to exist over the term break, it would exist for the lifetime of the storage. 9 track tapes now being dead, I’m not sure what the University did to preserve those old tapes.

In addition to these backups, we took a separate backup of the “password” file once a day. There were 30+ days of password file backups.

That is the setup. The actual story:

We used to give tours of the machine room. The operators enjoyed bragging about the quality of our backup system.

One of these tours, a little monster took one of the password backup tapes and put it in his backpack. He walked out of the machine room with that tape. Nobody noticed the missing tape for the next 30 days.

Said monster took that tape over to the engineering department, where they had their own 9 track tape drives. He read in the file.

He was presented with 10s of thousands of clear text passwords.

This had financial implications because we sold computer time.

We changed our policy to always encrypt the password file before it was written to tape. I have no idea if that encryption standard was any better than Sunday comic page ciphers.

No more Plain Text Passwords

The number of times somebody in a movie has gotten the idiot to give them somebody else’s password is astronomical. The truth is that most passwords are stored in an “encrypted” format. We don’t have access to your password.

We can reset your password, but we can’t tell you what it is because that isn’t recorded.

At the university, they were still storing passwords in plain text. They only encrypted the password when it was written to tape.

Modern systems store that password in an encrypted format. The old method was what is called “descrypt”.

The first two characters of the encrypted password is the “salt” and the rest is the DES hash of the password. This is NOT the same as encrypting your password with a secret and then being able to decrypt it with that same secret. Instead, we use your password to encrypt a given, known, piece of text. The encrypted result is what is stored.

When you provide your password, we encrypt the same text string with your password. If the resulting text matches what we have stored, you have proven you know the password.

Here are a couple of hashed passwords: SD2PFyBHY1oUY, q5M9nJsU/JSwI, sTd5NrAIMrisU, 8MbLuguRAeo92, $1$OcbNKu2y$l9faj.aCWodfonXiSlgnV0, $1$hh765lOJ$lrZ4jkCtUkG3qPBuFJQ/2., $5$2W0fdlfY.a/iXErF$xbzHcX8CfPc89vJkxsiC/BjDmqxI20Yk.Vj9OLL/6e2, and $5$HxfQ9B30d8GdmyPo$J6FWaeGKSez2cLbw3cktvaYgPvsTFaXdMzYp4yDcQjD.

These are all hashes of the same password, “hello world!”

Slow Them Down

Storing passwords in plain text is stupid. But computers are faster than you think. Thus, we want to slow down the speed at which computers can make guesses.

We do this by using a salt.

Consider the situation where you had 74,577,451,608 guesses you wanted to try. If you were to create the hash for each of those guesses, it might take you a bit of time. In the end, you would have them all. Now it is only seconds to look up the hash in a database/file and get the plaintext password used to generate that hash.

To fight this, we use the salt. The salt modifies the hashing process such that for any given password, there are many possible hashes to represent that password.

As shown above, even when using the same “hashing algorithm” we got many results.

This is to slow the guessing of passwords down.

And the results

In 2013, the game “battlefield” was cracked. They escaped with around a 1/4 million password hashes. These are not clear text, you can’t just type them into an account and get in, they are still “protected”.

I used a starting source of 184,000 known passwords. To this, I added an American and a British word list. I didn’t bother to get name lists for a total of 282,000 unique test words.

In the simplest case, with no salt applied, that is 184,000 * 282,000 different combinations to test.

In 2 minutes and 50 seconds, on my medium GPU and medium CPU, we tested 74,577,451,608 different passwords against 282,546 password hashes.

We were able to guess 7.30% of the passwords, or, 30943 passwords.

That is more than enough to make money.

And you can see how bad they can be.

What we are talking about is “authentication.” Authentication is the method of confirming that you are who you say you are.

There are three methods to determine authentication:

1. Something only you know
2. Something only you have
3. Something unique about you

In the old days, when people carried checkbooks with them and wrote checks for things, you would be asked to prove your identity before you could use a check. Proving your identity was a process where a person would first authenticate your identification card, and then they would verify that the identification card matched you.

A state issued identification card will have different aspects about it that should make identifying fakes easier for the trained person. In the those olden days, they would often have your Driver’s License number be a SoundEx of your last name. SoundEx was a simple encoding method that could be generated from a name.

If the SoundEx didn’t match the DL number, it was a fake.

For the most part, people trusted DLs. They were relatively difficult to fake, and it was often easy to spot fakes.

This is an example of something you have, your DL, and something unique about you. Your picture and description.

Computer Authentication

Computers authenticate you with the use of two pieces of information, the first is your “name”. The second is your password.

Your name can be an email address or a username. While the pair, username and password, are required, only the password is a secret. Or should be a secret.

In a perfect world, this would be good enough. In this imperfect world, see Password Security/Password Managers

We will assume that your password is strong and will not be cracked in this century.

What we want to protect against is people stealing your username and password. Be that by phishing or by tricking you, or by lifting your keyboard to read your password on a PostIt note.

We need to improve our overall security posture by adding something besides “something only you know” to the equation.

Biometrics

This is just a fancy word for something unique about you. What you look like. What you sound like. What the patterns of ridges on your fingers look like. What the blood vessels in your eye look like. These are things that are unique about you.

The super fancy eye scanner in NCIS is a myth. While it might actually work in practice, it will be expensive and is only part of the equation.

Fingerprint scanners are a joke. Facial recognition has more downsides than positives. And don’t have a sore throat if you are using vocal recognition.

Most low-cost fingerprint scanners don’t do a good job. They scan something they think is a fingerprint on a finger. That scan is processed and turned into a series of identified markers. That is turned into some sort of “value”. That value is what is actually compared and authenticates.

To reduce false negatives, these scanners often do a poor job of discriminating. They are also fairly weak at detecting live vs. Memorex.

Finally, if you have a fingerprint scanner or some other sort of biometric authenticator, bad actors can forcibly use your body to unlock your stuff.

It is far too common of an occurrence to have customs or law enforcement hold your finger to your phone’s scanner to unlock your phone. Don’t use biometrics to secure your devices. Oh, currently the courts find this to be legal and not a violation of your civil rights.

This takes use too:

Security Devices

A security device is a device that only you have that can communicate with other devices to help authenticate you.

Notice it is a helper, it is not the be all, end all.

The most common security device in use today is a mobile or cell phone.

The assumption is that you are the person holding your phone and that your phone can only be unlocked by you. This means that they can send you a text message, and you will have to unlock your phone to get the code they sent.

Except… Often the code is visible even when the phone is locked. The phone might be unlocked for other reasons. Or somebody cloned your phone and is getting the same SMS messages that you are.

In addition to that, some people have their devices configured to read messages to them. Or worse, they have configured their phones to read messages on command.

My favorite example of this was when I was working on a female friend’s car. She had a new boy and they were texting hot and heavy. Every time she received a new message, her phone would announce “To hear the message say “read message”.

At one point her phone announced, and I spoke up, “read message”.

She ran when her phone started to read the message out loud. It was just as spicy as I expected.

While the phone is very convent, it isn’t very secure.

Still, phones can be used as an authenticator.

This is a magic pseudo random number generator. The authenticator reads a seed from the remote device and attaches it to a particular site or device.

The two can generate the same pseudo random number at any point in time, based on the shared seed.

The site requests you provide the code from the authenticator. You unlock your phone, run the authenticator, find the correct device, copy the code from your phone to your computer to log in.

It is a fairly cheap and easy method and requires very little extra.

A number of my clients use this type of authenticator, and WordPress/WordFence does as well. It is an acceptable option if your phone is kept locked.

Better still, turn on extra security. The authenticator I use allows me to set a PIN for the application. Without the PIN, something only I know, the authenticator will not run.

Security Tokens

These supply a different form of security. They are designed to prove to a remote system, or local, that you have something that is unique.

A key.

One type of security token generates is a physical rendition of the phone authenticator. The one that I used required me to enter a PIN. It did not matter what PIN you entered, it generated numbers. If you entered the numbers from a correct PIN, you were logged in. If you entered the numbers from an incorrect PIN, the system would alert administrators or security, depending on how it was configured.

In other words, the system administrators and security personal could set them up to provide “panic” or “distress” codes.

Mine didn’t have that feature. If I put the wrong code in I couldn’t log in. Guess I wasn’t that important in the grand scheme of things.

Which takes me to my favorite authentication key, the YubiKey.

This is a small device, about the size of a thumb drive, but much thinner.

They have USB-A or USB-C connectors and some have NFC capabilities. They are small enough and light enough that I carry one of them attached to my key ring, along with a magic USB drive that contains a working version of Linux.

When properly configured, when a website needs a 2FA action, it will request that you insert the device. A small LED flashes, you touch the LED and the flashing stops. Some magic happens, and the website confirms that you have the right device.

If you have the NFC version, you can just tap the key to the back of your phone to accomplish the same thing as plugging it into a device.

In general, you should have two of them. Just in case you lose one.

Conclusion

Two-Factor Authentication adds a significant improvement to your security stance. They can almost completely stop phishing attacks.

Even if you are tricked into providing your credentials to a phishing website, when they attempt to use those credentials, they do not have the second factor to complete the authentication process.

Using your phone as your security device isn’t as strong as an authenticator. Using an authenticator application on your phone, is.

Combine these with a good password manager and you have a strong, secure system.

Until you find that the bad guys just ignore all that authentication stuff and took your computers.

Password Security

There are four ways of cracking a password.

1. Guess the password
2. Brute Force the password
3. Go around the password authentication
4. Trick the password from the owner

If your password is easy to guess, then it is a weak password. Examples of weak passwords are: password, 1234, YOUR_NAME, BIRTHDAYS. Many things use a four digit PIN. When guessing them, the best place to start is the set of numbers between 1950 and 2005, followed by 1930-1949, and 2006-2024. Years of importance to you.

Brute force is when you try all possible passwords. Back in the days of the TRS-80, there was a password on some part of the operating system. I wrote a simple brute force cracker for it.

Once it was running, my host and I got ready to go to dinner. Before we got out the door, the program stopped.

I assumed the program failed. Turned out that the password was so weak, three or four characters long, that it only took a few minutes to try all the passwords to that point.

Going around a password is sometimes easier than it should be. People don’t bother to log out. When I was visiting my father, I sat down at his computer. It was unlocked. I was able to “be” him if I had wished. I didn’t have to bother with a password.

There is an entire industry devoted to tricking people into handing over their passwords. It is so bad that it has its name, “phishing”.

And anybody can get caught in the net. I was caught just once. My wife’s school was phished, hard. The entire school got an email that looked legitimate from an administrator for the district. Her account then automatically sent it to me because I was in her address book.

I opened because it was from my wife. It had a good subject line. It looked legit.

It didn’t do anything to me because I run Linux, but it caused a great deal of damage to the school district.

Besides phishing, there is looking for the passwords that people have written down.

Again, using my father, the password for my mother’s computer was written on a PostIt note stuck to the inside of her laptop.

There is no need to guess, force or phish when the password is just given to you.

The Balancing Act

It is rather oxymoronic that the harder it is to remember a password, the harder it is to crack the password. If your password is “happyfaces” it might be easy to remember, but it is also easy to guess.

On the other hand, “wynt>Otchib5” is difficult to remember and difficult to guess. The password generator I used gave that to me as “wynt-GREATER_THAN-Otch-ib-FIVE” as how I might pronounce it and remember it. Still, it isn’t going to work

When passwords get too difficult to remember, people need to write them down. You would be amazed at the number of personal, and business, computers which have a file named “passwords”. People write them down.

The other thing that happens is that people remember one “good” password, then use it over and over again. If they ever lose that password, they lose access to everything, or the bad hat gets access to everything.

Many people think they will be tricky and use character substitution. Instead of “password” they write, “p@55w0rd”, and think they are clever. They aren’t.

There is a scene in Schindler’s List where they have just cleared the ghetto. Now they are searching for hidden Jews. The German’s come in, and they know where to look. They are experts at finding people. They’ve done this before. They know all the hiding places.

If you think you have found something clever that will make your password “unguessable”, you are mistaken.

Long Passwords Are Better(?)

Let’s assume that you are going to use a password that can’t be guessed easily. This leaves the brute force method.

This is a matter of mathematics. The larger the symbol set, the better. Longer passwords are better.

Consider a four digit pin, there are 10,000 possible PINs. As a password, that sucks.

But if we increase the symbol set to digits and letters, we get a slightly better result: 36^4 = 1,679,616. Still not strong.

But let’s say you go all out and have a symbol set of all ASCII printable tokens. There are 128 ASCII tokens, of which 94 are printable. This gives us 81,450,625 different passwords. Which still sucks, but it is getting better.

Now, let’s just make the password longer, call it 8 characters, at that point our results would be: 6,634,204,312,890,625. This is a strong password. Unfortunately, it is likely to be nearly impossible to remember.

My default is 12 characters.

Creating Strong Passwords You Can Remember

When we go back to that original statement, “The larger the symbol set, the better.” What if I told you that there is a symbol set of approximately 100,000 symbols, that you already know?

That symbol set is the set of all common English words.

What we would like to see is a number near 6 Quadrillion. With a symbol set of 100,000 words, 3 words give you 1 Quadrillion and four words give you 118,495,929,354,657,605,136.

This doesn’t consider word separators or case. Here is one such random password, “farm particularly wild refer”. If you modify the spaces to be different characters, or capitalize some letters, even if it only the first letter, you get even better results.

So what’s the problem? The issue is that it doesn’t look like a strong password. Many password checkers will see that long password and reject it because it doesn’t have special characters.

For me, a programmer, I can put together a simple program, take the string above, feed it into sha256sum to get 256 bits of pseudo noise. Extracting the printable characters, I get “dLuxo8x’H54MBd”

Now I have a good password I can remember, which can be used to generate a password which the rest of the world will accept as strong.

Password Managers

Password managers are supposed to fix much of this. They exist to store your passwords in a “secure” form, which you can then extract when needed. In addition, they will generate strong passwords for you to use.

I, personally, use four password managers and have used a fifth.

The first, most people are aware of, is the password manager built into your browser. I use Firefox and Chrome, so those are two password managers. My Linux system has another password manager built in. Finally, I use “Keeper” and have used “Last Pass”.

I love Keeper, I pay for the version I use, but there might be a free version. For me, it is worth it. One of the reasons it is worth it to me, is that with the paid version I can share access to password folders or individual passwords.

I never liked “LastPass” but I can’t say why. I do know they were cracked within the last few years. Because of their security model, when they were cracked, the bad guys extracted all the passwords.

Keeper stores all passwords encrypted. Only you have the decryption key. Thus, if they were to lose everything, they would not expose your passwords.

The browser managers are there because I was using them before Keeper. I’m slowly phasing them out.

I’m also looking into a self-hosted version of a password manager. I have not decided on which one, if any, I will try.

Chicken and Egg

The problem with all password managers is that there is a single point of failure. That is the password to access your password manager.

Which takes us back to “Long passwords work better”. Generate a random four – word password, I used xkcd Password Generator but you can just open a physical dictionary and randomly select four words.

Memorize those four words. Then you can use that as your master password.

Make the move to a good password manager. Use one that distrusts the government.

Two Factor Authentication

I need to look at my articles to see if one already exists, if it doesn’t, I’ll write something up.

Resiliency is a goal. I’m not sure if we ever actually reach it.

In my configuration, I’ve decided that the loss of a single node should be tolerated. This means that any hardware failure that takes a node of line is considered to be within the redundancy tolerance of the data center.

This means that while every node has at least two network interfaces, I am not going to require separate PSUs with dual NIC’s, each with two 10Gbit interfaces. Instead, each node has two 10Gbit interfaces and a management port at 1 to 2.5 gigabits RJ45 copper.

Each node is connected to two switches. Each switch has a separate fiber, run via a separate path, back to a primary router. Those primary routers are cross connected with two fibers, via two different paths.

Each of the primary routers has a fiber link to each of the egress points. I.e., two paths in/out of the DC.

The NAS is a distributed system where we can lose any room and not lose access to any data. We can lose any fiber, and it will have NO effect on the NAS. We can lose any switch and not have it affect the NAS.

We can lose any one router and not impact the NAS.

So far, so good.

Each compute node (hypervisor and/or swarm member) is connected to the NAS for shared disk storage. Each compute node is part of the “work” OVN network. This means that the compute nodes are isolated from the physical network design.

Our load balancer runs as a virtual machine with two interfaces, one is an interface on the physical network. The other is on the OVN work network.

This means that the VM can migrate to any of the hypervisors with no network disruption. Tested and verified. The hypervisor are monitored, if the load balancer becomes unavailable, they automaticity reboot the load balancer on another hypervisor.

So what’s the issue?

That damn Load Balancer can’t find the workers if one specific node goes down. The LB is still there. It is still responding. It just stops giving answers.

I am so frustrated.

So I’m going to throw some hardware at it.

We’ll pick up a pair of routers running pfSense. pfSense will be augmented with FRR and HAProxy to provide load balancing.

Maybe, just maybe, that will stabilize this issue.

This is a problem I will be able to resolve, once I can spend time running diagnostics without having clients down.

In upgrading from copper to fiber, I’ve been exploring the different options and learning as I go. Some learning curves have been steep, others have been “relearning” what I already knew.

One of the biggest things I needed to learn is that there are “switches” that are actually “routers”. That was mind-bending.

The other is that the network dudes talk about VLAN and Tagged VLAN. They are different things. In the environments I’ve been working in, there are only tagged VLANs which are called “VLAN”. Same name, different meaning.

The starting place when moving from copper to fiber is to understand what a Small Form-Factor Pluggable is. This is the magic that makes it all happen. This is standardized into SFP and SFP+. The SFP standard only supports 1G and lower speeds.

The SFP+ supports higher speed modules. 10G, 25G, 40G and 100G are all standards I’ve seen.

I’m only working with 10G modules, at this time.

They have modules that are RJ45 copper that will run at slower speeds or up to 10G. The only issue is that they draw more power and run hot. Can’t touch them when running hot.

The fix for this is to purchase a switch or router that has RJ45 Ethernet ports and at least one SFP+ port.

I found a small, six port, switch. This comes with 4 RJ45 ports, rated at 2.5G each, and 2 SFP+ ports rated at 10G each. Cool.

This allows me to daisy-chain them if I wanted.

In reality, it meant that I had one host connected at 10G while the others were at 2.5G.

I also found a L2/L3 “switch” that looks much like the switch above.

Having done the upgrades, I started looking into upgrading the router between the outside world and the DMZ. The routers I’ve been getting to not support any crypto, so they don’t have good VPN capability, something I want.

So I went looking. Looking for a “motherboard with SFP”. Something interesting popped. A mini ITX motherboard with 4 SFP+ ports and 4 RJ45 ports along with HDMI, VGA and the standard USB ports. It also provided space for two M.2 SSD modules, 2 DDR4 slots and two 6GByte SATA ports.

It might not be the fastest computer on the block, but it looks like a good starting point.

This leads me to other motherboards of the same ilk. And what I found was a bunch of these motherboards. And the port layouts all look the same. The specifications all look the same.

What we have is a “standard” motherboard which is put in a “standard” case along with a wall wart, HDMI cable and a mounting bracket. The branding stays the same.

I have an L2 switch that I’m going to take apart in a bit. It has a limit of 1550 byte packets, making it useless for my new network. I wonder if I will find an M.2 module in that box or something else that allows me to change the software.

Meanwhile, that motherboard is on my wish list. I’ll load pfSense on it along with FRR and replace my current router. Giving me a considerable boost in capabilities and letting me dispense with the VyOS configuration language. Which I really don’t like.

I own a pocket watch. It is beautiful, but I don’t use it very often.

I know that I own a couple of watches. One of them is a battery powered solar recharging thing.

My standard “watch” today is my cell phone.

When I was in high school, I was very interested in accurate time keeping. As was my father.

This meant that we would call the “time” phone number to set our watches, at least once a week.

My grandfather had a “railroad watch”. This was a wristwatch that was approved by the railroad for time keeping. It was approved by the SooLine for use as a time keeping device. Amazing, until that model of watch was approved, the railroad required the use of pocket watches.

This was because a level of accuracy was required that only pocket watches or well regulated wristwatches could maintain.

The big thing in my youth were “quartz” watches. Instead of using a tuning fork or a mechanical balance/regulator, they used the vibrations of a quartz crystal to keep track of the time.

What this meant was that you had devices that were now able to maintain the same wrong time over an extended period of time.

The user had to set them correctly.

As an example, for years, maybe even to today, my wife would set her car clock (and many other clocks) 10 minutes fast. “So she would be on time for appointments.”

I set my car to my phone’s reported time.

One of the fun things that I did as a kid was to call up the Naval Observatory to get the current time. This was reported from their atomic clock. One of the most accurate time keeping devices in the world.

Accurate Time

Many protocols require accurate time. It is wonderful that you have a time piece that is accurate to within 1 second per year, but if it is reporting the wrong time, it is not particularly useful to the protocol.

What we want, is to know what time it is right now, and then to set our time to that.

We get the current time from a known, accurate time source. Today, that is often GPS satellites.

If you have ever wondered how GPS works, it works because your device knows where each satellite is at any instant of time. Each satellite transmits its ID and the current time. Over and over again.

That is all they do.

And here is the magic, if your device knows what time it is, and it knows where the satellite should be at his time, it can calculate the distance by comparing the difference in time.

If you are directly under a GPS satellite, it takes about 67ms for the signal to reach your device. From this, we can use the speed of light to figure out the distance traveled. Then some simple math and we know the location of your device.

We can also get accurate time by listening for the atomic clocks via radio. If you know where you are, and you know where the clock is, you can calculate the delay between the atomic clock and your device, then match your device to the atomic clock.

Today, when people want to use that type of process, they use a GPS device and get the time ticks from the device.

How long did it take?

This is where it starts to get complicated.

The standard for communications with a GPS device is 4800 or 9600 baud across a 3 wire serial connection. The protocol, the text being transmitted, specifies the time when the last character is transmitted.

That data is being received. Your device is processing it. Your device takes a certain amount of time to process the record it just received. It takes time to process that record. All of that is latency.

If you do not know the latency in your device, you do not know what time it is. For grins, just think of that serial link being 300,000,000 meters long. That would put a 1-second latency by itself.

There are ways of calculating the latency, but I do not remember what they are.

Latency is the important piece of information.

Calculating Latency

Many network people have run ping. It is a tool for testing reachability and latency between your device and some other device on the Internet.


ping -c 5 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=116 time=11.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=116 time=11.1 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=116 time=11.6 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=116 time=11.1 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=116 time=11.0 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 11.022/11.179/11.616/0.220 ms


This is a test from one of my faster machines to a Google DNS server. This tells me that it takes 11.179 ms to reach that DNS server. Testing to one of my network timeservers, the average is 78.094 ms.

This means that the time reported by the timeserver will be off by some amount. In a simple world, we would guess that it is 1/2 of 78.094.

But, I use NTP. NTP does many transmissions to multiple timeservers to discover the actual time. It is reporting that the latency is 78.163512 ms. A little more accurate. It tells me that the dispersion is 0.000092 ms.

How does it know this? Because of many samples and because of four different time stamps.

When my device sends an NTP request packet, it puts the current time in it. When the server receives the packet, it puts the current time in it. When the server transmits the response, it adds the current time again. Finally, when the reply is received, the current time is added to the packet. This gives us four different time stamps from two different sources.

We compute the total latency via mine(R)-mine(S). We know the processing time by server(S)-server(R). The difference between server(R)-mine(S) and mine(R)-server(S) as the symmetry between the two paths the request and response traveled.

From these values, we can calculate the network distance, in seconds, between us and them.

Assume we transmit at time 0(M), it is received at 100(S), the response is transmitted at 105(S) and we receive it at 78(M).

How can we receive our reply before the server sent it? Easy, we have to different views of what time it is.

The latency is 78. This means that the halfway point was at 38. It took 5 to process the reply and get it on the wire again. If we do simple stuff, this means that our time is off from their time by 67.

But we can do better. By looking at the reported latency between the two legs, we can actually calculate how long it took for us to receive the reply.

NTP uses multiple timeservers to get a consensus as to the time. It monitors each timeserver to determine which one jitters the least.

All of this means that we can have very accurate times.

And having accurate measurements of the time, NTP will calculate how much the computer’s clock drifts over time. It will then modify the clock rate in parts per million to get the drift as close to zero as possible.

This means, that the longer your device runs NTP, the more accurate it becomes.

The Internet is a fantastic creature. I’m not speaking of the information you can find on the internet. Nor am I speaking of the entertainment that is available on the Internet.

The mere fact that you can ask for information at your desk or on your phone and somehow that request gets there, and the response gets back, is mind bogglingly complex.

Here is the dirty little secret about computers. It is all zeros and ones. There are no pictures, there are no videos, there are no songs nor even text, it is all zeros and ones.

We group these zeros and ones into units of different sizes. The three primary sizes are 8, 32, and 64, with a spattering of 16. At the lowest level, we think about these in groups of 8, called octets.

You might know them as “Bytes”.

Now, zeros and ones are a bit difficult to read and write. So we use base 16 to read and write bytes.

Base 16 has 16 digits, just like base 10 has 10 digits. 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9. are the digits of base 10.

For base 16, we add A, B, C, D, E, and F as the six extra digits.

So we have a 32-bit number that looks like this: 4C4F5645 in hex (base 16) and 1280267845 in base 10, and “LOVE” as ASCII.

It is all zeros and ones. It takes meaning when we decide how those bits will be interpreted.

When you ask Google to search for “The Vine of Liberty”, your browser starts with a name, which it needs to convert to an address. The name is “www.google.com”. Depending on where you are, one of the addresses will be 142.250.69.68.

This is a different representation of a 32-bit word. In this “dotted quad”, each number represents the decimal value of an 8-bit byte.

For you, the simple household, your device asks, “How can I get this message to 142.250.69.68?”

Your device looks up the address in the “routing table”. Your device likely only has a single entry in the routing table. The route of last resort, or default route.

When no other table entries match, then send the request to a default router

A router has a single job, to move packets (requests and responses) from one network to another. When your default router receives your device’s request, it looks up the IP address (142.250.69.68) in its routing table. Again, it is likely that there is only a single entry in that table, the default route.

This is the simple way that things work in simple networks. It continues to work until the moment when a router has to make a choice. Does it send the packet from network H (your home network) to network A or to network B.

That router will have a routing table. It will find a match for 142.250.69.68 in that table, which will tell that router which network to forward your request to.

If nothing about the Internet ever changed, that would be all that was needed. Every router would know how to get to every address and that would be it.

But it isn’t that easy. The Internet changes, constantly. This means that we need to be able to change those routing tables quickly and easily.

The answer to that issue is a routing protocol. The oldest was RIP. It doesn’t work well today as it sends too much data too often. Back in slower networking times, RIP was taking up nearly 70% of my bandwidth. We stopped that.

There are two major types of routing protocols, external and internal. The primary external protocol, today, is the Border Gateway Protocol, or BGP. I don’t have to worry about that.

What I do need to worry about is internal routing. For internal routing, I use a combination of static routes and OSPF.

And this is where it gets complex. The data center has two physical networks. A management network and a production network.

The management network runs on a single subnet, with each host having a unique address on that subnet.

The production network runs on multiple subnets, each subnet serving to isolate problems. In addition, traffic on the production network needs to be able to reach the Internet.

The management network requires zero routing. One network space. No connection to the outside world.

On top of the physical network are layered multiple other networks. There is the OVN NAS network. This is how each of the hypervisors gets access to block storage (and shared file systems). There is the OVN NAS data network. There is the OVN VM network, the container network.

In addition, there are other networks used inside the container environment.

Some of these networks exist in isolation. Others are used as transport networks. No traffic originates nor terminates in these transport networks.

But other networks need to be able to speak to each other.

That means that every device needs to know how to reach every address. This means that OSPF is doing magic all the time to make things work.

Why? Redundancy. Every device has at least two paths to the next hop. If the primary link fails, the secondary link takes over.

This is done by rebuilding the routing table.

OVN links don’t fail (unless the idiot driving the keyboard does something stupid). The physical network can fail. When this happens, OVN just routes the tunnels in different directions.

So why this rant?

Because I can’t get parts of this to work!

My need is to move the containers into the OVN.

And I can’t get routing to work consistently. ARGH!

Oh well. Filler done.

My first fiber switch turned out to be a L3 managed “switch”. Way cool. But I purchased a cheap switch and found that it completely undocumented.

It has taken me a while to figure things out.

The configuration GUI is an What You See Is All You Get type. There is enough there that you can get the switch up and running, but not enough to fully configure the L3 Switch.

To accomplish that, you need to use the CLI. Not a problem, I like CLI’s.

Of course, there is no documentation but for tab completion and very limited help screens.

I get it mostly working.

After playing with the Free Range Routing Suite (FRR) for a while and getting OSPF working on all of my hosts and the primary router, I was feeling pretty confident.

It seems that FRR took their configuration model almost directly from Cisco’s CLI. The number of times I used a Cisco help page to determine how to configure an OSPF setting is remarkable.

The new L3 switch turns out to have a Cisco like configuration language. And what isn’t Cisco like, is FRR like. Neither Cisco nor FRR, but close.

Today I had a tremendous success, I moved a ceph host from the physical network to the OVN network.

This included moving that segment of the network to a new subnet. And everything sort of worked.

The issue turned out to be a routing issue.

The correct answer is to turn on OSPF within the new physical router. It does support it, after all.

Having played with the damn thing for a few hours, breaking my network multiple times, I was about to give up when I happened to notice a strange value for a setting.

That setting? MTU, of course.

Even though every interface shows an MTU of 9000. Even though jumbo frames are turned on and using a 9000 byte frame.

Even though an MTU of 9000 is very much supported, the MTU of the “VLAN” was set to 1500.

Now, Cisco VLANs are not the same as a tagged VLAN. A tagged VLAN acts like a separate physical network. They are where you place interface settings. These VLANs can then be assigned to a physical port.

The physical port’s MTU overrides the VLAN MTU. This means my jumbo packets from host to host work.

The problem is that the VLAN MTU is maxed out at 2000 bytes. This seems to only affect the OSPF traffic and not the physical interface. But I’m dead in the water or I need to figure out how to do this differently.

Still, I didn’t pay an arm plus a leg for this physical router. I’ll get it to work.

In 1983, CCITT and ISO merged their network definition to create The Basic Reference Model for Open Systems Interconnection.

This is the “famous” seven layer model. Which works for ISO standards but is a poor match for the Internet.

The three layers we are interested in are:

1. Physical layer
2. Data link layer
3. Network layer

1 Physical Layer

The physical layer defines the electrical, mechanical, and procedural interface to the transmission medium. WTF?

Ok, let’s look at this in terms of some real examples. If you have a computer that is more than a few years old, it will have a network connection in it or a port that a network connection can be attached to.

The most common mechanical connection, the socket and connector, is the RJ-45. This is the thing that looks like a big telephone connector. Oh yeah, many of the youngsters don’t remember every plugging a phone into the wall.

This connector consists of 8 connectors. The location and form of these connectors defines part of the mechanical system.

The other part is that those 8 connectors are attached to four pairs of wires. The pairs of wire are twisted and bundled into a single cable. Each of the 8 wires are numbered, and the mechanical definition of the RJ-45 defines which wires are attached to which connector, at both ends.

When I say “numbered”, the physical reality is that the wires are color coded.

The electrical definition defines which wires are used for transmitting and which are used for receiving. It defines if the signals are ground referenced or differences between two wires.

Everything about how to connect the physical devices and how to transmit a signal are specified at Layer 1, the physical layer.

2 Data Link Layer

This layer defines how data is transmitted over the L1 physical network. It defines what how to use the physical layer.

For example, Frame Relay is a data link protocol for connecting distant devices. Each Protocol Data Unit (PDU), consists of a flag field, an address field, an information field, and a frame check sequence, or checksum field.

The information field contains the actual data (information) that is being transmitted.

The Frame Relay standard states that the information field must be at least 262 octets (bytes) and recommends that it support at least 1600 octets.

It is important to note that a length of 262 cannot be (easily) expressed in a single byte. This means that the length field must be at least 2 bytes wide.

While Frame Relay is still in use, today, it is not as common as it used to be. There are better options.

A much more common L2 protocol is Ethernet. This is called a Frame. The Frame consists of a preamble, start frame delimiter, destination address, src address, tag (or zeros), type or length, payload, CRC and a gap.

As originally defined, an Ethernet packet had a maximum length of 1500 octets.

Packet Size

In networking, we talk about sending a packet. A packet is a more generic term for “frame”. We have packets at the data link layer and at the network layer.

Every packet contains enough information to identify the source and destination of the packet, the length of the packet, and the payload. There will often be a header to identify more about the type of the packet.

As a packet moves through a network, it might be “fragmented” as it passes through a network segment which has an MTU smaller than the packet size.

There must be enough information to reconstruct the packet, even when the packet has become fragmented.

Fragmenting is something we want to avoid, if possible.

To that end, a part of the connection process is to discover the MTU for each device.

Consider a simple network segment. A network segment is a piece of the network that is connected at L2.

We have devices A and B. Device A is using a fiber physical layer and device B is using a copper physical layer. B is attached to switch 2, switch 2 is connected to switch 1, and switch 1 is connected to device A.

If all four devices are using old style Ethernet frames, then the MTU will default to 1500. A simple database backup is 3.3 GB. This means we will have to transmit at least 2,305,845 packets.

This requires each device to handle 2.3 million interrupts.

On the other hand, if we were to use jumbo packets, then we reduce this to around 384,307 packets. This is a huge savings in load on the network segment.

The two switches, as L2 devices, are going to either be store and forward switches, or simple hubs. Nobody uses hubs anymore. So they must be switches.

Each switch receives the packet, storing it, then transmits that packet on a different port.

The switch must be able to store the complete packet/frame. If it can not, it will drop the packet.

When designing your network, you want to make sure that all the switches on the network support the largest MTU you might be using.

Devices A and B will discover what their MTUs are. The smaller will rule. The switches, on the other hand, are transparent. They do not get a say in the MTU discovery.

What this means, is that you can have devices on the network that respond to simple testing, such as sending pings, but which fail for larger packets.

Conclusion of Rant

I accidentally purchased a switch (L2) when I was intending to purchase a router (L3).

This should not have been an issue. I intended to use some switches, regardless.

The specifications look good. MTU is documented as 12000.

I plug everything together and start testing. My first network test is always “ping”. If ping isn’t working, nothing else will work well enough.

That worked perfectly.

Then I attempted to login to the remote site using SSH. This silently failed, before timing out with destination unreachable.

Ping works, SSH doesn’t?

This makes no sense.

Until I found it. SSH does a key exchange with my RSA public key. The key size is 1679 bytes. This is larger than the supported MTU of switch 2 at 1500.

The network fails, silently.

So I have email out to the manufacturer, hoping for a positive response.