Nerd Babel

The amount of grief I’ve put up with to get this working buggers imagination.

To have a NTP stratum 1 server, you need to have a certain set of capabilities.

First, you need a stratum 0 device. This is an atomic clock or a GPS receiver.

You need a method to communicate with the GPS receiver.

Your clock needs to be network connected.

Each of these pieces must be done correctly with the least amount of jitter possible.

Jitter is how much a signal deviates from its target. If the jitter is zero, then we have a level of accuracy that depends solely on our reference clock.

The little GPS unit is self-contained. If it is supplied 3.3V of power, it will search for satellites and do the calculations to know what time it is and where it is.

The calculations turn out to be for someplace along the cable from the antenna to the GPS unit. Some highly accurate versions of the GPS SoC measure the length of the antenna feed and account for that in the calculations. Regardless, it is the time for a place a little offset from the actual GPS chip.

For me, that is a delay of around 10ns.

The GPS will communicate via a serial protocol. This means that we have a delay from when the message is received and when we can put our timestamp on the message. For me, that is around 140ms.

This can be discovered by tracking the time indicated by the serial GPS and the system/local clock. The local clock is synced to multiple remote NTP servers to get this number.

Unfortunately, there is about a 1ms jitter in this signal.

If I were to use a USB converter. I.e., serial to USB, that jitter goes up. I am seeing a jitter of 4 to 9 ms.

Using the serial directly is a good start.

But there is another signal that can help. That is the Pulse Per Second (PPS). We are using a 1second pulse.

IFF we can capture the time at which the pulse arrives, we can get a very accurate start of the second marker.

This requires that the hardware have a general purpose input/output(GPIO) pin available.

Most motherboards do not have exposed GPIO pins. Worse, some boards have GPIO pins, but there is no documentation on how to access them.

So the server board requires GPIO plus a method of accessing those pins.

There are two ways to discover a change of value, we can pole for it, or we can get an interrupt.

Consider you have your phone alerts silenced so you don’t get a noise every time you receive an email or message.

You have to check your phone for new messages. This is “poling”.

If somebody calls, your phone still rings. You then immediately check to see who it is and perhaps answer the phone.

This is an interrupt.

The default operation of a GPIO pin is poling driven. Even if it is generating an interrupt, that interrupt is only used to record the change of value.

What is needed is a high-performance interrupt handler. When an interrupt happens, the handler records the system clock. A user land process watches, either poling or interrupt, it doesn’t matter, for that value to change.

When it changes, the software knows that the GPS “knew” it was the start of the second when it created the pulse.

The amount of jitter is only as much time as it takes for the system to allocate a CPU and for that CPU to process the interrupt. In other words, really, really fast.

Currently, the jitter on my PPS reference clock is 300ns. Because of the many samples that have been taken, the PPS reference clock is currently running 17ns from the real time. That has been going down over the last few hours. By the time you read this, it is likely to be even less.

The PPS clock is so tight that the other clock sources hide the values, even in logarithmic form

This is an interesting graph, to me, as it indicates how the system clock is slowly being conditioned to keep more accurate time. It software currently says that the drift is -17.796271 ppm off which I think translates to 3.324ms

So how bad was this task?  More painful than I wanted it to be.

I’m fine with “dumb” computers.  I started programming on 6502s.  I’ve been bit slinging for 50 years.  Programming Arduino’s?  No problem.

Building a PC from components, installing any compatible operating system?  I do it a dozen times a week when developing.

The Raspberry Pi is a different animal.  It isn’t sold as a low-level system.  You can use it that way, but that is not how it is intended to be used. It is sold as a System On a Board (SOB) that runs a modern (Linux, Android) operating system.

This is where things get strange. When we are working with modern PCs, they have known hardware.  We boot the computer, run the OS, the OS has drivers to talk to the hardware.  Everything just works.

This is possible because PC’s have a Basic Input Output System (BIOS).  This is a low-level set of routines that are there to allow accessing certain parts of the hardware with a standard Application Protocol Interface (API).

Since every BIOS has the same API, OS vendors can use the BIOS to load enough of their software to continue booting.  The hardware is attached in known ways.  The hardware vendor supplies the drivers for their hardware.  Linux people write their drivers if needed.

So consider that SOB. It has a serial port.  The serial port is controlled by a standard UART.  That UART is programmed in a standard way.  They are all the same.

In order for that UART to work, the software needs to know where the UART is located in memory (or on the I/O bus). In addition, the pins that the UART uses have to be configured for the UART.  Most UART’s use standard pins on the GPIO header. The pins that the UART uses can be used in different modes for different things.

The problem comes from that address being different in every SOB or SOC.  A board could have one, two, or more GPIO driver chips.  It all depends on the designer.

The developers overcome this issue with what is called a “Device Tree”.

The device tree is a parsable description of devices and their locations in memory or on the I/O bus.

The board I purchased doesn’t have a supported modern OS.  The only OS that I could get to boot was released in 2016.  The OS is not really supported anymore.  The board itself was flaky. It would randomly reboot, or just power off.

The “modern” OS that should have worked didn’t even complete the boot.

In discussions with a community support person, we decided that there was hardware that was not being properly initialized in the kernel.  I.e., we had a bad Device Tree.

The replacement Banana Pi doesn’t have a supported modern OS.  It is fully supported by Arabian, which is a supported, modern OS.

When I first booted the system, it just worked.  I was thrilled.  It has continued to work properly.

Then I plugged the GPS in.  I could see it blinking.  This indicates that it has a lock and the PPS signal is being sent.

But I can’t get any input on the serial ports.

It turns out that the default device tree doesn’t activate that UART.  Once I figured that out, I had to find an overlay to the device tree to turn on the UART.

That was a pain, but it happened.

Working serial, no PPS.

With the tools on hand, I could monitor the GPIO pin and see the PPS.  But it wasn’t doing anything.

I loaded the correct kernel modules, still no PPS.

My Google Foo suggested that the device tree entry for PPS was missing.

Yep, there was no PPS overlay.

The Linux kernel documentation describes the Device Tree.  But no real examples, and nothing fully commented.

By comparing multiple sources, I finally was able to create a device tree overlay for PPS. I need to figure out how to return that DTD to the community.  The problem is, I don’t know what the hell I did.  I made it work.  I think I know what was done.  Nonetheless, it was truly a case of looking at different device tree overlays and picking out the parts that seemed to match what I needed to do.

I don’t think I’ve had this much difficulty hooking up a piece of hardware since 1983, when I was attempting to attach a DEC 10 MB hard drive to a computer that wasn’t really a DEC.

The only tasks remaining is to put everything in a case and move it to its long-term home, off the top of my computer.

I’ve known about “System On a Chip” and “System On a Board” for many years. I have one of the early embedded software development kits, including some TI chipset.

The most common style of these today is likely the Arduino class of SoC. These things are incredible.

An idea I pitched years ago, for potential military use, was a swarm of small single purpose computers that could be dropped in mass from an aircraft.

Something about the size of a quarter with one or two sensors, a mesh Wi-Fi system, GPS, and a battery. These would drop like the “helicopter” seed pods, scatter over an area, then set up a monitoring network. Using spread spectrum, low power, and burst transmissions, the network might be difficult to detect.

As the sensors detected “things”, they would report to a transmission unit, which would then send a report to home base.

This was all based on a small battery. We figured we could get these things in mass and get them ready to toss out the back of a C130 for less than $100 each.

Today, I can buy a chip that will do that, put it on a custom board with all components for less than $20 in low unit counts.

So SoC, way cool.

The other thing that has been happening is that the physical size requirements for a personal computer have gone way down. Whereas the original XT motherboard was 8.5×11 and the AT was 12×13, we are now seeing Mini-ITX at 6.7×6.7 and even Nano-ITX at 4.7×4.7.

My son found the Mini-ITX form factor computer a few years ago. A full computer that was the size of 3 boxes of .45cal. It weights less.

His computer came with 2 HDMI ports, 4 USB3.2 ports, a headphone jack, Wi-Fi, Bluetooth, and 1Gb Ethernet. It had upgradable memory and at least one M.2 port. This thing just works.

What is cool about it is that it mounts to the back of a monitor. He needs his keyboard and two power supplies.

It is what I got my lovely wife for Christmas this last year.

Can I ditch the big boxes?

Things go in circles. My primary machine lives in a case I purchased almost 20 years ago. It has a new power supply, an extra video card, and a SAS controller. It is designed to handle 9 internal drives plus the optical drive. It can connect to external SAS cabinets.

It was designed to have 36 TB of storage in a ZFS pool. It is heavy. It requires big fans. It is a workhorse.

But, I’m moving away from ZFS. It is a great system, I love it. It just does not meet my current needs. I’ve moved to ceph, a distributed system.

Ceph does not use raid technology the same way that ZFS does. Instead, it depends on having many nodes with lots of redundancy.

With ZFS, my normal was one parity drive for every 4 data drives. So a 20% overhead.

In its fast mode, ceph has a 200% overhead. For every block of data stored, it requires two copies. There are modes that are more optimized, but they all seem to have higher overhead than a raid system.

But because I don’t need to create 5 drive pools, I can do something different. More boxes with just a few drives in each.

I don’t need all that motherboard. I don’t need all that memory. I don’t even need all that much CPU.

Some of the servers I’m using only support two SATA drives, but have an NVMe slot. Still, it is a big box.

My answer was to go looking. I found a cheap ITX-Mini motherboard. It would have been cheaper off the boat, but I found it on Amazon. A used i5 CPU. It will handle an i7 and maybe an i9. A cheap 128GB NVMe, and some DDR3 ram. Toss in a CPU cooler and it is a fully functional system.

It only has a Gigabit Ethernet, but it has a PCI slot. It has four SATA ports.

The cheap case I found has four hot swap bays for SATA drives. It takes a Mini-ITX motherboard. This thing is perfect for what I require.

So yes, this can do the job. I can downsize.

What else is cool about these Mini-ITX boards?

They really are designed with specific markets in mind. I found one with 2 2.5Gb RJ45 and 2 10Gb SFP+ ports. It has GPIO ports, comes in an extruded aluminum heat sink/case. It replaced the old router and everything got faster.

There are versions with multiple RJ45 ports. Different layouts. Different CPU designs. Some come with a CPU on the board, making them a SoB. Some are AMD, some are Intel based.

What I couldn’t find is a board with four SATA ports, a PCIe slot, and GPIO.

Which brings me back to PI

The board I want needs to have hardware-driven Ethernet, GPIO pins, and serial IO. It also has to be cheap. I think I found it in the Banana PI M2 Ultra.

It has everything I need, it was very cheap, less than dinner for three at the fast food joint.

My step backward? It requires power that I can’t give it. GRRR. So it requires a dedicated power supply.

Oh well, my NTP server is almost here.

Correction(s):
I made the mistake of trusting Google’s AI answer.

Speed of light:

Sorry for that. A pepper grain is the size of a picosecond. A nanosecond is around 11.7 inches, which makes much more sense.

My mentor, Mike, had so much to teach me. Coming from University, I knew I was the best programmer on campus and better than anybody I had met.

That changed the day I met Mike. After being introduced to him, I went to my boss and said something like, “That is the most arrogant man I’ve ever met.”

Greg replied, “He’s earned it.”

When I had an opportunity to work with him, I found that yes, he was that good.

He was the first person to stretch my abilities in computer science. I will forever be thankful to him for that.

He had the privilege of meeting Admiral Grace. He was one of the many that were handed her “packet of Nanoseconds”.

This was Grace’s way of getting across to people just how fast computers were running.

In 1 ms, light will travel 299.79 meters. This is a reasonable rifle shot.

In 1 us, light will travel 0.2998 meters (299.8mm), or about 1.2 inches.

In 1 ns, light will travel 0.2998 mm. This is about the size of a single grain of ground pepper.

Just how fast?

My Cray X-MP/48 had a memory bank cycle time of 38ns. This means that light would be able to travel about 10mm or a little less than 0.5 inches.

My memory said that we had a 85ns wait time from accessing memory to having loaded that word into a register.

Your PC likely runs faster than that X/MP. It surely has more memory.

Frames of Reference

As stated above, my world was baby sitting a Cray Super Computer. We worked in nanoseconds. We were trying to optimize code to shave a few nanoseconds out of a loop.

Saving grains of pepper.

When I purchased some study lights for doing photoshoots, I didn’t buy the fancy radio controllers. Instead, I bought “slave” style lights.

With the slave style, you could tell your study light to fire when it detected the flash of another strobe.

Before I purchased these study lights, I went to Mike with concern. I had done the math.

From the moment my flash fired, a long sequence of things had to take place. The light had to travel from my strobe to the detector on the study light. There was a delay while the photoreceptor energized and “fired”. There was still more time as that signal propagated through the circuitry, and finally that light would fire.

My studio lights would be at different distances, we couldn’t even predict the sequence that they would fire.

According to my simple calculations, we could be talking as much as 2ms from the time my light fired until the last study light fired.

Mike pulled me back to the ground. My shutter speed would be set to 1/60 of a second. That is 16.6ms. If the study lights fired anytime while the shutter was open, I would get a good photo.

I was so focused on my reference frame, nanoseconds, I lost sight of the real-world application that was running in 10s of milliseconds.

pkoning Brings Reality to the Clocks

Here is the magic of GPS. It works by knowing the time and then calculating the distance to different satellites.

The more accurate the clock, the more accurate the location.

Communicating that time to an external device is where it gets interesting. The definition of NMEA tells use exactly when the second mark occurs during the transmission of the NMEA message.

Most GPS units default to transmitting at 9600 baud. Which for us is the same as 9600 bits/second. Each 8 bit byte sent requires start and stop bits. My rule of thumb is 10 bits per byte.

This means that it takes around 83ms to transmit one 80 character NMEA sentence from the GPS to the computer.

The instant when the second starts aligns with the edge of a signal of one of the characters in that sentence.

Now my issue was that I thought that the GPS unit had time that was “wrong” it was offset from the real world.

This is not the case. The real reason for the delay is in the time it takes to process the sentence. That should be fixed to the computer, not to the GPS unit.

Which brings us to PPS, or Pulse Per Second. This is a signal that indicates the start of a second. Depending on the GPS unit, this can be at ns accuracy. Even cheap units will get you sub us accuracy.

The processing time to handle the pulse is much lower than to handle a full NMEA sentence.

A PPS can be treated as the “real” time, without fear of being too far away from reality.

A couple of grains of pepper.

There are two network time protocols in use today. One is the NTP protocol, the other is PTP. I have no idea what the PTP looks like, I know that it requires hardware support.

The goal of NTP is to create a local clock that is accurate to less than 1ms from sources that have up to a hundred times that in latency.

The fact that this works at all simply amazes me.

I have 7 servers acting as an NTP cluster. That is to say, they all work to come to a consensus as to what time it is, and then each syncs to that time point.

They do this via actively querying each other every 64 seconds. The protocol knows that the referenced clock time is somewhere within the total time from query to response. Using complex statistics, it can get it much closer than just “somewhere” or “middle”.

As I am writing this, one server believes it knows the time to the network with a standard deviation of less than 500us. It has one nailed down to less than 83us.

Within the local cluster, it believes it knows the time within 50us for all the cluster members. For a few of the cluster members, they agree on the time within 3000ns (3us). That’s not bad.

So what are the problems.

The first problem is that I have a clock that claims to be very accurate, but which I know is slightly wrong.

The clock is fast by 957us with an SD of 57us. I believe it to be worse than that. The issue being that the clock is influencing the rest of the time cluster.

I did that because I had a usable fudge factor for the clock. Now I need to bring it much closer to “real time”.

To that end, I’ve reconfigured the server with the GPS clock to never use the GPS time. Instead, it will use the network servers to converge on “the time”. Once I know “the time” I will be able to adjust the GPS offset better.

The second issue is that USB injects jitter into the signal. We don’t know when the USB port received the time message from the GPS unit. Hopefully, we have a good estimate, but it is still very jittery.

It is nearly impossible to get down to 1ns or less with a clock that has a 500us jitter.

What does this mean? I need to stabilize the time signal. We do that with a PPS. This pulse tells us that the second happened on the rise or fall of the pulse. This is configurable. The second starts on the rising edge of the pulse. With a PPS input, we can reduce jitter to a few nanoseconds.

The issue still is, “What is the offset detecting the leading edge from the “real” start of the second?”

This value comes from the delay along the antenna cable and other such speed of light delays.

Which takes us to the conclusion of this article.

The GPS units I purchased came with a small ceramic antenna. The antenna is about 10 mm on a side. It has a 10 cm connector. This means the unit and the antenna are very close to each other. The antenna isn’t a great antenna.

With this taped to the inside of the window, I was picking up 3 satellites. I replaced it with a cheap, yet “real” antenna. I’m not locking on to 10 or more satellites. More locks mean better time keeping.

If you are doing this yourself, do yourself a favor and order a real antenna to go with your project.

In addition to being “real”, the cable is 3m long, giving me options on where to place it. And it is a water proof unit.

We are in the process of moving from the image above to the image below.

At least in terms of what the infrastructure looks like.

Today I decommissioned an EdgeRouter 4 which features a “fanless router with a four-core, 1 GHz MIPS64 processor, 3 1Gbit RJ45 ports, and 1G SFP port.”

When they say “MIPS64” you can think of it as being in the same class as an ARM processor. Not a problem for what it is.

The issue was that there are only 1Gb interfaces. That and I’ve come to hate the configuration language.

This has been replaced with a pfSense router running on a TopTon “thing.” I call it a thing because it is from China and intended to be rebranded. It doesn’t have a real SKU.

It is based on an N100 with 4 cores and 8 threads. 2 2.5Gb Ethernet ports, 2 10Gb SFP+ ports. It can be upgraded and has multiple extras.

Besides the hardware, this is an entirely different animal in terms of what it can do. It is first, and foremost, a firewall. Everything else it does is above and beyond.

It is running NTP with a USB GPS unit attached. It runs DHCP, DNS, HAProxy, OSPF and a few other packages. The IDS/IPS system is running in notify mode at this time. That will be changed to full functionality very shortly.

So what’s the issue? The issue is that everything changed.

On the side, as I was replacing the router, I jiggled one of the Ceph servers. Jiggling it caused it to use just a few watts more, and the power supply gave out. It is a non-standard power supply, so it will be a day or two before the replacement arrives.

When I went to plug the fiber in, the fiber was too short. This required moving slack from the other end of the fiber back towards the router to have enough length where it was needed.

Having done this, plugging in the fiber gave me a dark result. I did a bit of diagnostic testing, isolated the issue to that one piece of fiber. I ran spare fiber to a different switch that was on the correct subnet, flashy lights.

Turns out that I had to degrade the fiber from the other router to work with the EdgeRouter 4. Once I took that off, the port did light off. But that was a few steps down the road.

Now the issue is that all the Wi-Fi access points have gone dark. Seems that they are not happy. This required reinstalling the control software and moving them from the old control software instance to the new one. Once that was done, I could see the error message from the access point complaining about a bad DHCP server.

After fighting this for far too long, I finally figured out that the pseudo Cisco like router was not forwarding DHCP packets within the same VLAN. I could not make it work. So I disabled the DHCP server on the new router/firewall and moved it back to the Cisco like router. Finally, Wi-Fi for the phones and everything seems to be working.

At which point I can’t log into the Vine of Liberty.

I can see the pages, I can’t log into the admin side. It is timing out.

3 hours later, I figured out that there was a bad DNS setting on the servers. The software reaches out to an external site for multiple reasons. The DNS lookup was taking so long that the connection was dropping.

I think this is an issue that I have just resolved.

But there’s more.

Even after I got the DNS cleaned up, many servers couldn’t touch base with the external monitoring servers. Why?

Routing all looked good, until things hit the firewall. Then it stopped.

Checking the rules, everything looks good. Checking from my box, everything works. It is only these servers.

Was it routing? Nope, that was working fine.

That was one thing that just worked. When I turned down the old router, the new router distributed routing information correctly and took over instantly.

So the issue is that pfSense “just works.” That is, there are default configurations that do the right thing out of the box.

One of those things is outbound firewall rules.

Anything on the LAN network is properly filtered and works.

But what is the definition of the LAN network? It is the subnet directly connected to the LAN interface(s).

Because I knew that I would need to be able to access the routers if routing goes wrong, my computer has a direct connection to the LAN Network attached to the routers. The Wi-Fi access points live in on the same subnet. So everything for my machine and the wireless devices “just worked”

The rest of the servers are on isolating subnets. That are part of the building LAN but they are not part of the “LAN Network”.

I know this, I defined an alias that contains all the building networks.

Once I added that to the firewall rules, it just worked.

Tomorrow’s tasks include more DHCP fights and moving away from Traefik. Which means making better use of the Ingress network.

I have hundreds of dollars worth of GPS equipment. Not counting the cell phones we all carry with us.

I wanted to try to create a Stratum 0 NTP clock.

The last time I attempted this, I used a Garmin handheld GPS. Time to sync was in minutes and while the power draw as trivial, by the standards of the day, it would still burn through AA batteries.

Because you, kind readers, told me that there were cheap options, I went looking.

What I found was a GPS module that is about an inch square. For $15 I could have one delivered. It comes with a header containing VCC, GND, TXD, RXD, and PPS. I figured I could solder in the provided header then run them to a GPIO that has an attached UART.

Well, the darn things showed up a day early, and I didn’t really want to do any soldering. I plugged it in via the USB port, put it in the window. A few minutes later, it had a hard lock.

After installing gpsd and configuring, chrony I now have a system that is locked at less than 1ms accuracy, NOT using the PPS option.

That will be next week’s project. Getting that PPS signal to the motherboard.

If I had a Raspberry Pi with a good interface, not wifi, I can see that this would make a darn nice little timekeeper.

My wife read my article on passwords and “got it”. Which is nice. I was attempting to explain how password crackers use rule sets to modify input dictionaries to create more guesses from a single word list.

I decided to see how much things have advanced. To say I was shocked would be an understatement.

In 2013, the game “Battlefield” was hacked and the entire password database was captured.

This is not the major security threat you might instantly leap to, but it is bad.

Stealing Passwords

I worked in the Systems Group at my University. We were tasked with all software maintenance, installations, upgrades, and in house improvements to the operating system.

The systems group had taken the original manufacturer’s operating system and extended it to the point where it was no longer the same operating system. Having done this, we gave back all the code we had written to the manufacturer, who incorporated what they liked into their next release.

We had developed a long term backup plan. This plan was three tiered. We took daily backups of the entire file system. This was a rolling tape backup. There were 30 days of daily backups performed before the first tape was overwritten.

We also performed weekly backups. There were 52 weeks of weekly backups. So a total of 82 backup sets.

In addition to this, we did end of term backups. These were done just after the term ended. These tapes were kept.

What this meant was that if your file were to live for at least 24 hours, you would be able to recover to any particular day in the past 5 weeks of your file.

If your file were to exist over a weekend, you could recover that file to how it was on the weekend it was dumped for the past year. And if your file were to exist over the term break, it would exist for the lifetime of the storage. 9 track tapes now being dead, I’m not sure what the University did to preserve those old tapes.

In addition to these backups, we took a separate backup of the “password” file once a day. There were 30+ days of password file backups.

That is the setup. The actual story:

We used to give tours of the machine room. The operators enjoyed bragging about the quality of our backup system.

One of these tours, a little monster took one of the password backup tapes and put it in his backpack. He walked out of the machine room with that tape. Nobody noticed the missing tape for the next 30 days.

Said monster took that tape over to the engineering department, where they had their own 9 track tape drives. He read in the file.

He was presented with 10s of thousands of clear text passwords.

This had financial implications because we sold computer time.

We changed our policy to always encrypt the password file before it was written to tape. I have no idea if that encryption standard was any better than Sunday comic page ciphers.

No more Plain Text Passwords

The number of times somebody in a movie has gotten the idiot to give them somebody else’s password is astronomical. The truth is that most passwords are stored in an “encrypted” format. We don’t have access to your password.

We can reset your password, but we can’t tell you what it is because that isn’t recorded.

At the university, they were still storing passwords in plain text. They only encrypted the password when it was written to tape.

Modern systems store that password in an encrypted format. The old method was what is called “descrypt”.

The first two characters of the encrypted password is the “salt” and the rest is the DES hash of the password. This is NOT the same as encrypting your password with a secret and then being able to decrypt it with that same secret. Instead, we use your password to encrypt a given, known, piece of text. The encrypted result is what is stored.

When you provide your password, we encrypt the same text string with your password. If the resulting text matches what we have stored, you have proven you know the password.

Here are a couple of hashed passwords: SD2PFyBHY1oUY, q5M9nJsU/JSwI, sTd5NrAIMrisU, 8MbLuguRAeo92, $1$OcbNKu2y$l9faj.aCWodfonXiSlgnV0, $1$hh765lOJ$lrZ4jkCtUkG3qPBuFJQ/2., $5$2W0fdlfY.a/iXErF$xbzHcX8CfPc89vJkxsiC/BjDmqxI20Yk.Vj9OLL/6e2, and $5$HxfQ9B30d8GdmyPo$J6FWaeGKSez2cLbw3cktvaYgPvsTFaXdMzYp4yDcQjD.

These are all hashes of the same password, “hello world!”

Slow Them Down

Storing passwords in plain text is stupid. But computers are faster than you think. Thus, we want to slow down the speed at which computers can make guesses.

We do this by using a salt.

Consider the situation where you had 74,577,451,608 guesses you wanted to try. If you were to create the hash for each of those guesses, it might take you a bit of time. In the end, you would have them all. Now it is only seconds to look up the hash in a database/file and get the plaintext password used to generate that hash.

To fight this, we use the salt. The salt modifies the hashing process such that for any given password, there are many possible hashes to represent that password.

As shown above, even when using the same “hashing algorithm” we got many results.

This is to slow the guessing of passwords down.

And the results

In 2013, the game “battlefield” was cracked. They escaped with around a 1/4 million password hashes. These are not clear text, you can’t just type them into an account and get in, they are still “protected”.

I used a starting source of 184,000 known passwords. To this, I added an American and a British word list. I didn’t bother to get name lists for a total of 282,000 unique test words.

In the simplest case, with no salt applied, that is 184,000 * 282,000 different combinations to test.

In 2 minutes and 50 seconds, on my medium GPU and medium CPU, we tested 74,577,451,608 different passwords against 282,546 password hashes.

We were able to guess 7.30% of the passwords, or, 30943 passwords.

That is more than enough to make money.

And you can see how bad they can be.

What we are talking about is “authentication.” Authentication is the method of confirming that you are who you say you are.

There are three methods to determine authentication:

1. Something only you know
2. Something only you have
3. Something unique about you

In the old days, when people carried checkbooks with them and wrote checks for things, you would be asked to prove your identity before you could use a check. Proving your identity was a process where a person would first authenticate your identification card, and then they would verify that the identification card matched you.

A state issued identification card will have different aspects about it that should make identifying fakes easier for the trained person. In the those olden days, they would often have your Driver’s License number be a SoundEx of your last name. SoundEx was a simple encoding method that could be generated from a name.

If the SoundEx didn’t match the DL number, it was a fake.

For the most part, people trusted DLs. They were relatively difficult to fake, and it was often easy to spot fakes.

This is an example of something you have, your DL, and something unique about you. Your picture and description.

Computer Authentication

Computers authenticate you with the use of two pieces of information, the first is your “name”. The second is your password.

Your name can be an email address or a username. While the pair, username and password, are required, only the password is a secret. Or should be a secret.

In a perfect world, this would be good enough. In this imperfect world, see Password Security/Password Managers

We will assume that your password is strong and will not be cracked in this century.

What we want to protect against is people stealing your username and password. Be that by phishing or by tricking you, or by lifting your keyboard to read your password on a PostIt note.

We need to improve our overall security posture by adding something besides “something only you know” to the equation.

Biometrics

This is just a fancy word for something unique about you. What you look like. What you sound like. What the patterns of ridges on your fingers look like. What the blood vessels in your eye look like. These are things that are unique about you.

The super fancy eye scanner in NCIS is a myth. While it might actually work in practice, it will be expensive and is only part of the equation.

Fingerprint scanners are a joke. Facial recognition has more downsides than positives. And don’t have a sore throat if you are using vocal recognition.

Most low-cost fingerprint scanners don’t do a good job. They scan something they think is a fingerprint on a finger. That scan is processed and turned into a series of identified markers. That is turned into some sort of “value”. That value is what is actually compared and authenticates.

To reduce false negatives, these scanners often do a poor job of discriminating. They are also fairly weak at detecting live vs. Memorex.

Finally, if you have a fingerprint scanner or some other sort of biometric authenticator, bad actors can forcibly use your body to unlock your stuff.

It is far too common of an occurrence to have customs or law enforcement hold your finger to your phone’s scanner to unlock your phone. Don’t use biometrics to secure your devices. Oh, currently the courts find this to be legal and not a violation of your civil rights.

This takes use too:

Security Devices

A security device is a device that only you have that can communicate with other devices to help authenticate you.

Notice it is a helper, it is not the be all, end all.

The most common security device in use today is a mobile or cell phone.

The assumption is that you are the person holding your phone and that your phone can only be unlocked by you. This means that they can send you a text message, and you will have to unlock your phone to get the code they sent.

Except… Often the code is visible even when the phone is locked. The phone might be unlocked for other reasons. Or somebody cloned your phone and is getting the same SMS messages that you are.

In addition to that, some people have their devices configured to read messages to them. Or worse, they have configured their phones to read messages on command.

My favorite example of this was when I was working on a female friend’s car. She had a new boy and they were texting hot and heavy. Every time she received a new message, her phone would announce “To hear the message say “read message”.

At one point her phone announced, and I spoke up, “read message”.

She ran when her phone started to read the message out loud. It was just as spicy as I expected.

While the phone is very convent, it isn’t very secure.

Still, phones can be used as an authenticator.

This is a magic pseudo random number generator. The authenticator reads a seed from the remote device and attaches it to a particular site or device.

The two can generate the same pseudo random number at any point in time, based on the shared seed.

The site requests you provide the code from the authenticator. You unlock your phone, run the authenticator, find the correct device, copy the code from your phone to your computer to log in.

It is a fairly cheap and easy method and requires very little extra.

A number of my clients use this type of authenticator, and WordPress/WordFence does as well. It is an acceptable option if your phone is kept locked.

Better still, turn on extra security. The authenticator I use allows me to set a PIN for the application. Without the PIN, something only I know, the authenticator will not run.

Security Tokens

These supply a different form of security. They are designed to prove to a remote system, or local, that you have something that is unique.

A key.

One type of security token generates is a physical rendition of the phone authenticator. The one that I used required me to enter a PIN. It did not matter what PIN you entered, it generated numbers. If you entered the numbers from a correct PIN, you were logged in. If you entered the numbers from an incorrect PIN, the system would alert administrators or security, depending on how it was configured.

In other words, the system administrators and security personal could set them up to provide “panic” or “distress” codes.

Mine didn’t have that feature. If I put the wrong code in I couldn’t log in. Guess I wasn’t that important in the grand scheme of things.

Which takes me to my favorite authentication key, the YubiKey.

This is a small device, about the size of a thumb drive, but much thinner.

They have USB-A or USB-C connectors and some have NFC capabilities. They are small enough and light enough that I carry one of them attached to my key ring, along with a magic USB drive that contains a working version of Linux.

When properly configured, when a website needs a 2FA action, it will request that you insert the device. A small LED flashes, you touch the LED and the flashing stops. Some magic happens, and the website confirms that you have the right device.

If you have the NFC version, you can just tap the key to the back of your phone to accomplish the same thing as plugging it into a device.

In general, you should have two of them. Just in case you lose one.

Conclusion

Two-Factor Authentication adds a significant improvement to your security stance. They can almost completely stop phishing attacks.

Even if you are tricked into providing your credentials to a phishing website, when they attempt to use those credentials, they do not have the second factor to complete the authentication process.

Using your phone as your security device isn’t as strong as an authenticator. Using an authenticator application on your phone, is.

Combine these with a good password manager and you have a strong, secure system.

Until you find that the bad guys just ignore all that authentication stuff and took your computers.

Password Security

There are four ways of cracking a password.

1. Guess the password
2. Brute Force the password
3. Go around the password authentication
4. Trick the password from the owner

If your password is easy to guess, then it is a weak password. Examples of weak passwords are: password, 1234, YOUR_NAME, BIRTHDAYS. Many things use a four digit PIN. When guessing them, the best place to start is the set of numbers between 1950 and 2005, followed by 1930-1949, and 2006-2024. Years of importance to you.

Brute force is when you try all possible passwords. Back in the days of the TRS-80, there was a password on some part of the operating system. I wrote a simple brute force cracker for it.

Once it was running, my host and I got ready to go to dinner. Before we got out the door, the program stopped.

I assumed the program failed. Turned out that the password was so weak, three or four characters long, that it only took a few minutes to try all the passwords to that point.

Going around a password is sometimes easier than it should be. People don’t bother to log out. When I was visiting my father, I sat down at his computer. It was unlocked. I was able to “be” him if I had wished. I didn’t have to bother with a password.

There is an entire industry devoted to tricking people into handing over their passwords. It is so bad that it has its name, “phishing”.

And anybody can get caught in the net. I was caught just once. My wife’s school was phished, hard. The entire school got an email that looked legitimate from an administrator for the district. Her account then automatically sent it to me because I was in her address book.

I opened because it was from my wife. It had a good subject line. It looked legit.

It didn’t do anything to me because I run Linux, but it caused a great deal of damage to the school district.

Besides phishing, there is looking for the passwords that people have written down.

Again, using my father, the password for my mother’s computer was written on a PostIt note stuck to the inside of her laptop.

There is no need to guess, force or phish when the password is just given to you.

The Balancing Act

It is rather oxymoronic that the harder it is to remember a password, the harder it is to crack the password. If your password is “happyfaces” it might be easy to remember, but it is also easy to guess.

On the other hand, “wynt>Otchib5” is difficult to remember and difficult to guess. The password generator I used gave that to me as “wynt-GREATER_THAN-Otch-ib-FIVE” as how I might pronounce it and remember it. Still, it isn’t going to work

When passwords get too difficult to remember, people need to write them down. You would be amazed at the number of personal, and business, computers which have a file named “passwords”. People write them down.

The other thing that happens is that people remember one “good” password, then use it over and over again. If they ever lose that password, they lose access to everything, or the bad hat gets access to everything.

Many people think they will be tricky and use character substitution. Instead of “password” they write, “p@55w0rd”, and think they are clever. They aren’t.

There is a scene in Schindler’s List where they have just cleared the ghetto. Now they are searching for hidden Jews. The German’s come in, and they know where to look. They are experts at finding people. They’ve done this before. They know all the hiding places.

If you think you have found something clever that will make your password “unguessable”, you are mistaken.

Long Passwords Are Better(?)

Let’s assume that you are going to use a password that can’t be guessed easily. This leaves the brute force method.

This is a matter of mathematics. The larger the symbol set, the better. Longer passwords are better.

Consider a four digit pin, there are 10,000 possible PINs. As a password, that sucks.

But if we increase the symbol set to digits and letters, we get a slightly better result: 36^4 = 1,679,616. Still not strong.

But let’s say you go all out and have a symbol set of all ASCII printable tokens. There are 128 ASCII tokens, of which 94 are printable. This gives us 81,450,625 different passwords. Which still sucks, but it is getting better.

Now, let’s just make the password longer, call it 8 characters, at that point our results would be: 6,634,204,312,890,625. This is a strong password. Unfortunately, it is likely to be nearly impossible to remember.

My default is 12 characters.

Creating Strong Passwords You Can Remember

When we go back to that original statement, “The larger the symbol set, the better.” What if I told you that there is a symbol set of approximately 100,000 symbols, that you already know?

That symbol set is the set of all common English words.

What we would like to see is a number near 6 Quadrillion. With a symbol set of 100,000 words, 3 words give you 1 Quadrillion and four words give you 118,495,929,354,657,605,136.

This doesn’t consider word separators or case. Here is one such random password, “farm particularly wild refer”. If you modify the spaces to be different characters, or capitalize some letters, even if it only the first letter, you get even better results.

So what’s the problem? The issue is that it doesn’t look like a strong password. Many password checkers will see that long password and reject it because it doesn’t have special characters.

For me, a programmer, I can put together a simple program, take the string above, feed it into sha256sum to get 256 bits of pseudo noise. Extracting the printable characters, I get “dLuxo8x’H54MBd”

Now I have a good password I can remember, which can be used to generate a password which the rest of the world will accept as strong.

Password Managers

Password managers are supposed to fix much of this. They exist to store your passwords in a “secure” form, which you can then extract when needed. In addition, they will generate strong passwords for you to use.

I, personally, use four password managers and have used a fifth.

The first, most people are aware of, is the password manager built into your browser. I use Firefox and Chrome, so those are two password managers. My Linux system has another password manager built in. Finally, I use “Keeper” and have used “Last Pass”.

I love Keeper, I pay for the version I use, but there might be a free version. For me, it is worth it. One of the reasons it is worth it to me, is that with the paid version I can share access to password folders or individual passwords.

I never liked “LastPass” but I can’t say why. I do know they were cracked within the last few years. Because of their security model, when they were cracked, the bad guys extracted all the passwords.

Keeper stores all passwords encrypted. Only you have the decryption key. Thus, if they were to lose everything, they would not expose your passwords.

The browser managers are there because I was using them before Keeper. I’m slowly phasing them out.

I’m also looking into a self-hosted version of a password manager. I have not decided on which one, if any, I will try.

Chicken and Egg

The problem with all password managers is that there is a single point of failure. That is the password to access your password manager.

Which takes us back to “Long passwords work better”. Generate a random four – word password, I used xkcd Password Generator but you can just open a physical dictionary and randomly select four words.

Memorize those four words. Then you can use that as your master password.

Make the move to a good password manager. Use one that distrusts the government.

Two Factor Authentication

I need to look at my articles to see if one already exists, if it doesn’t, I’ll write something up.

Resiliency is a goal. I’m not sure if we ever actually reach it.

In my configuration, I’ve decided that the loss of a single node should be tolerated. This means that any hardware failure that takes a node of line is considered to be within the redundancy tolerance of the data center.

This means that while every node has at least two network interfaces, I am not going to require separate PSUs with dual NIC’s, each with two 10Gbit interfaces. Instead, each node has two 10Gbit interfaces and a management port at 1 to 2.5 gigabits RJ45 copper.

Each node is connected to two switches. Each switch has a separate fiber, run via a separate path, back to a primary router. Those primary routers are cross connected with two fibers, via two different paths.

Each of the primary routers has a fiber link to each of the egress points. I.e., two paths in/out of the DC.

The NAS is a distributed system where we can lose any room and not lose access to any data. We can lose any fiber, and it will have NO effect on the NAS. We can lose any switch and not have it affect the NAS.

We can lose any one router and not impact the NAS.

So far, so good.

Each compute node (hypervisor and/or swarm member) is connected to the NAS for shared disk storage. Each compute node is part of the “work” OVN network. This means that the compute nodes are isolated from the physical network design.

Our load balancer runs as a virtual machine with two interfaces, one is an interface on the physical network. The other is on the OVN work network.

This means that the VM can migrate to any of the hypervisors with no network disruption. Tested and verified. The hypervisor are monitored, if the load balancer becomes unavailable, they automaticity reboot the load balancer on another hypervisor.

So what’s the issue?

That damn Load Balancer can’t find the workers if one specific node goes down. The LB is still there. It is still responding. It just stops giving answers.

I am so frustrated.

So I’m going to throw some hardware at it.

We’ll pick up a pair of routers running pfSense. pfSense will be augmented with FRR and HAProxy to provide load balancing.

Maybe, just maybe, that will stabilize this issue.

This is a problem I will be able to resolve, once I can spend time running diagnostics without having clients down.