Nerd Babel

When I started writing, regularly, for Miguel, I took it upon myself to cover legal cases. Since that time, I’ve learned more than I really wanted to about our justice system.

As my mentor used to say, “The justice system is just a system.” As a systems’ person, that allowed me to look at cases through the lens of my experience analyzing large systems.

One of the first things I noticed was that most people reporting on cases didn’t provide enough information for us to look up what was actually written or said.

CourtListener.com has come to my rescue for most legal filings in the federal system. If you know the court and the docket number you can find that case on CourtListener.

Once you have the docket located, you can start reading the filings. These are stored as PDFs. Most of my PDF tools allow me to copy and paste directly from the PDF.

What isn’t available on CourtListener is Supreme Court dockets. I’ve talked to Mike and others, the issue seems to be something about scrapping the Supreme Court website as well as other stuff. I’m not sure exactly what.

I want to be able to keep up on all the current cases in the Supreme Court, what their status currently is, what has been filed. They entirety of the case. I’m not concerned about most of the cases, but often it is easier to get all than a selected portion.

To this end, I have code that uses patterns to pull cases from the Supreme Court docket without have a listing of cases.

This tool will have search capabilities and other tools shortly, for now, it works well enough.

I am using the PySide6, which is a python implementation of the Qt framework. For the most part, I’m happy with this framework. There are parts I don’t like, which I work around.

My most recent success was figuring out how to allow me to click on hyperlinks in text to bring up my PDF viewer. This was not as simple as I wanted it to be, but it is working.

The other night, I wanted to write about a current case. I had the case docket in my tool. I pulled up the docket, clicked on the link, and John Roberts’ order popped up in my viewer, exactly as it should.

I started writing. Went to pull the quote and nothing.

Copy and paste does not seem to be functional in my tool.

Which takes me to the rant, which @#$)*&@$) coordinate system should I be using to get the right text!

Qt is built around widgets. Every widget has its coordinate system. In addition, there is the global coordinate system.

Each widget also has a paintEvent() which is when it paints itself.

To start the process, I capture mousePress, mouseMove, and mouseRelease events. While the mouse button is down, I draw a rectangle from the place clicked to the current location of the mouse.

I attempt to draw the rectangle and nothing shows up on the screen.

Through debugging code, I finally figured out that I am not updating the right widget.

The QPdfView widget properly renders the PDF document in a scrollable window. I have made a subclass of QPdfView so I am catching all paint events. But even though I’m telling the system that I have to redraw (update) my widget, there are no paint events being sent to my widget.

Turns out that my widget only cares about update signals that require the framing content be redrawn. I.e. if the scroll bar changes, then I get a paint event. Once I figured this out, I was able to tell the viewport that it should update and things started working.

So now I can draw a frame on the screen. But what I want is to get the text from within that frame.

I asked the QPdfDocument for a new selection from point_start to point_end. It tells me nothing is selected.

Where do I currently sit? I have my frame in my PDFViewer coordinate system. I have the PDF document in a different coordinate system. The PDF coordinate system is modified by the scroll bars or viewport. The scroll bars and scroll area modify the actual coordinate system of the viewport contents.

Somehow, I need to figure out which of these coordinate systems is the right coordinate system to use to get the text highlighted by my mouse.

I’m tired of this fight.

That would be me.

Everything finally came together with the new system. Then I went and messed it all up.

The motherboard has a weak Ethernet. It is a 10/100 Ethernet, which is NOT a problem for a management interface. When I upgrade the box to have full redundancy, it will get a dual port fiber card.

What it does mean is that my Wi-Fi to it via a USB dongle is faster than if I were to plug it in.

Once the box was in position, I connected via Wi-Fi and finished configuration. I tested all the connectivity, and it all just worked.

At that point, I told it to join the cluster. It did with pleasure, and brought the cluster to a stop.

Did you catch my mistake? Yeah, I left that dongle in.

At the bottom of the barrel, we have 10base-T. I have some old switches in boxes that might support that. Above that is 100base-T, which is a good management speed. We can move data for upgrades and restores, but not the fastest. Some of my switches and routers do not support 100baseT.

Above that is where we start to get into “real” speeds. Gigabit Ethernet, or GigE. I’ve now moved to the next step, which is ports supporting 10G over fiber or cable, depending on the module I use. The next step-up would be 25Gbit. I’m not ready for that leap of cost.

Wi-Fi sits at around 200Mbit/s. Faster than “fast Ethernet” also known as 100base-T, but not at “real” speeds. Additionally, Wi-Fi is shared space, which means that it doesn’t always give that much.

So what happened? The Ceph(NAS) cluster is configured over an OVN logical network on 10.1.0.0/24. All Ceph nodes live on this network. Clients that consume Ceph services will also attach to this network. No issues.

When you configure an OVN node, you tell the cluster what IP address to use for tunnels back to the new node. All well and good.

The 10G network connection goes to the primary router and from there to the rest of the ceph nodes. One of the subnets holds my work server. My work server provides 20Tb to the ceph cluster.

On that subnet are also the wireless access points.

So the new node correctly sent packets to all the ceph nodes via the 10G interface, EXCEPT for traffic to my work server. Why? Because the 10G had a 1 hop cost, while the Wi-Fi had a 0 hop cost. By routing standards, the 200Mbit Wi-Fi was the closer, faster, connection than the 1 hop 10G connections.

When I found the connection problem and recognized the issue, I unplugged the Wi-Fi dongle from the new node and all my issues cleaned up, almost instantly.

There are a few servers that are too old. There is a need for a few more servers to get a room level redundancy. These things can be expensive.

As I’m cheap, I’ve been using older servers that accept 3.5″ disk drives. Some except 2 drives, some 6, some could accept more, but the case doesn’t.

The fix I chose was to move to some four bay NAS enclosures. This is a reasonable size that balances with the network I/O capability.

These enclosures all take the Mini-ITX motherboard.

These motherboards are nothing short of amazing. In the middle tier, they have all the things a full-size motherboard has. Some have 4 memory slots, some only 2. They come with 1, 2, 4 Ethernet ports. Some have SFP ports. Some have SATA ports. The number of SATA ports ranges from 1 to 6. Some come with PCIe slots.

Depending on what your needs are, there is a motherboard for you.

Since this was going to be a NAS, the motherboard I selected had to have 4 SATA ports, an NVMe slot, and SFP+.

Yep, this exists. They don’t exist at the price point I wanted to pay. It finally clicked with me. I can just put an SFP+ PCIe card into the machine.

Thus, I picked a motherboard with 4 SATA, 1 Ethernet, 1 USB3, 1 PCIe slot, enough memory and 2 M.2 slots.

Some NAS enclosures do not have the opening for a PCI slot, so it was important to pick a case that had the card opening.

When I got the enclosure I was impressed.

It is a sturdy, thick steel case. There is no plastic on the entire thing. There are for hot swap disk bays plus mounting space for 2 2.5″ drives. Exactly what I was looking for.

When I went to install the motherboard, I was shocked to find that the CPU cooler didn’t fit. I ordered a low profile. I’m impressed with that as well.

I get the board mounted. It looks nice. I go to close the case and the cover won’t fit on. The cover has a folded U channel that goes over the bottom rail of the case to lock the case closed.

The problem is that there isn’t enough space between the edge of the motherboard and the bottom rail for the U channel to fit.

My first real use of the right-angle die grinder. I don’t have a cut-off wheel for it, so I just ground the edge away and it worked.

Of course, I gave myself a frost burn because I was too busy to put gloves on to handle the die grinder.

Back to the worktable, the cover now goes on. I plug a wireless USB dongle into the USB 3.0 and boot. Nothing.

It took me a couple of days before I figured it out. The case came with no documentation. The front panel connector has both a USB 3 plug and a USB 3 plug. I plugged both in. You are only supposed to plug in one. Fixed.

The installation happens, I’m happy. It is fast enough, it is responsive enough. I just need to get it put in place with the fiber configured.

I take the cover off the back slot. Go to put the PCI card in.

The (many bad words) slot does not line up with the opening in the back of the case.

The open in the back is off by 0.8 inches.

I consider cutting another card opening in the back. That won’t work. The card would be half out of the side of the case.

I ordered the cutoff wheels for the die grinder, I know I’m going to need them.

I decided to cut the back opening wider. This will leave an opening that can be taped closed on the PCI side. It allows me to use the existing slot with retaining hardware. I good idea.

All I need to do is unscrew the standoffs, drill and tap four holes in the right place, and I’m done.

Except… Those standoffs are pressed into place. They don’t unscrew.

No problem. I have a set of standoffs. I’ll just cut the existing standoffs off. Drill and tap holes in the right place and use my standoffs.

Except… My standoffs are the normal length. These standoffs are a custom length. I can’t do that.

Tools to the rescue

First stop, the arbor press. It is a small 2 ton press. I have no problems pushing out the standoffs. The press also removes the bulge from removing the standoffs.

Next step, the milling machine. Using the gage pins, I found the size of the holes is 0.197-0.198. Measuring the standoffs, I get 0.208. I settled on 0.201 for the hole size. I should have gone a 64th smaller.

There is no way to clamp this thing in the vise. I do have strap clamps. The case is quickly put into position.

The first hold is located, then drilled. No issues.

Except I don’t have enough travel to reach the other three holes. I reposition the case on the table and go for it.

I go back to the arbor press to put the standoffs back in. I don’t have enough height to support the case while installing the standoffs.

Back to the mill. Square to ends of a hunk of aluminum. Punch a 3/8in hole in it. Work on the mill vise and get the standoffs put back in place.

In the middle of this, I have an alarm, fearing that I put the standoffs in the wrong place. I do a quick test fit and everything is perfect.

It takes me a good hour to put the case back together with all the case mods done. It looks good. I’m happy with how it came out.

Today is search day. I have to find the 8 meter OM-4 fiber for this NAS, and I have to find the box of screws that came with the case for the hard drives. Once I have those, this can go into production.

I know what to look for on NAS cases. I’ll be building out a few more of these boxes over the coming months. First to replace two boxes which are too old. One for the redundancy.

The world will be good, or I’ll punch it again and again until it is good.

P.S. This is filler, the article about Trump’s win in the D.C. District court was taking to long.

The amount of grief I’ve put up with to get this working buggers imagination.

To have a NTP stratum 1 server, you need to have a certain set of capabilities.

First, you need a stratum 0 device. This is an atomic clock or a GPS receiver.

You need a method to communicate with the GPS receiver.

Your clock needs to be network connected.

Each of these pieces must be done correctly with the least amount of jitter possible.

Jitter is how much a signal deviates from its target. If the jitter is zero, then we have a level of accuracy that depends solely on our reference clock.

The little GPS unit is self-contained. If it is supplied 3.3V of power, it will search for satellites and do the calculations to know what time it is and where it is.

The calculations turn out to be for someplace along the cable from the antenna to the GPS unit. Some highly accurate versions of the GPS SoC measure the length of the antenna feed and account for that in the calculations. Regardless, it is the time for a place a little offset from the actual GPS chip.

For me, that is a delay of around 10ns.

The GPS will communicate via a serial protocol. This means that we have a delay from when the message is received and when we can put our timestamp on the message. For me, that is around 140ms.

This can be discovered by tracking the time indicated by the serial GPS and the system/local clock. The local clock is synced to multiple remote NTP servers to get this number.

Unfortunately, there is about a 1ms jitter in this signal.

If I were to use a USB converter. I.e., serial to USB, that jitter goes up. I am seeing a jitter of 4 to 9 ms.

Using the serial directly is a good start.

But there is another signal that can help. That is the Pulse Per Second (PPS). We are using a 1second pulse.

IFF we can capture the time at which the pulse arrives, we can get a very accurate start of the second marker.

This requires that the hardware have a general purpose input/output(GPIO) pin available.

Most motherboards do not have exposed GPIO pins. Worse, some boards have GPIO pins, but there is no documentation on how to access them.

So the server board requires GPIO plus a method of accessing those pins.

There are two ways to discover a change of value, we can pole for it, or we can get an interrupt.

Consider you have your phone alerts silenced so you don’t get a noise every time you receive an email or message.

You have to check your phone for new messages. This is “poling”.

If somebody calls, your phone still rings. You then immediately check to see who it is and perhaps answer the phone.

This is an interrupt.

The default operation of a GPIO pin is poling driven. Even if it is generating an interrupt, that interrupt is only used to record the change of value.

What is needed is a high-performance interrupt handler. When an interrupt happens, the handler records the system clock. A user land process watches, either poling or interrupt, it doesn’t matter, for that value to change.

When it changes, the software knows that the GPS “knew” it was the start of the second when it created the pulse.

The amount of jitter is only as much time as it takes for the system to allocate a CPU and for that CPU to process the interrupt. In other words, really, really fast.

Currently, the jitter on my PPS reference clock is 300ns. Because of the many samples that have been taken, the PPS reference clock is currently running 17ns from the real time. That has been going down over the last few hours. By the time you read this, it is likely to be even less.

The PPS clock is so tight that the other clock sources hide the values, even in logarithmic form

This is an interesting graph, to me, as it indicates how the system clock is slowly being conditioned to keep more accurate time. It software currently says that the drift is -17.796271 ppm off which I think translates to 3.324ms

So how bad was this task?  More painful than I wanted it to be.

I’m fine with “dumb” computers.  I started programming on 6502s.  I’ve been bit slinging for 50 years.  Programming Arduino’s?  No problem.

Building a PC from components, installing any compatible operating system?  I do it a dozen times a week when developing.

The Raspberry Pi is a different animal.  It isn’t sold as a low-level system.  You can use it that way, but that is not how it is intended to be used. It is sold as a System On a Board (SOB) that runs a modern (Linux, Android) operating system.

This is where things get strange. When we are working with modern PCs, they have known hardware.  We boot the computer, run the OS, the OS has drivers to talk to the hardware.  Everything just works.

This is possible because PC’s have a Basic Input Output System (BIOS).  This is a low-level set of routines that are there to allow accessing certain parts of the hardware with a standard Application Protocol Interface (API).

Since every BIOS has the same API, OS vendors can use the BIOS to load enough of their software to continue booting.  The hardware is attached in known ways.  The hardware vendor supplies the drivers for their hardware.  Linux people write their drivers if needed.

So consider that SOB. It has a serial port.  The serial port is controlled by a standard UART.  That UART is programmed in a standard way.  They are all the same.

In order for that UART to work, the software needs to know where the UART is located in memory (or on the I/O bus). In addition, the pins that the UART uses have to be configured for the UART.  Most UART’s use standard pins on the GPIO header. The pins that the UART uses can be used in different modes for different things.

The problem comes from that address being different in every SOB or SOC.  A board could have one, two, or more GPIO driver chips.  It all depends on the designer.

The developers overcome this issue with what is called a “Device Tree”.

The device tree is a parsable description of devices and their locations in memory or on the I/O bus.

The board I purchased doesn’t have a supported modern OS.  The only OS that I could get to boot was released in 2016.  The OS is not really supported anymore.  The board itself was flaky. It would randomly reboot, or just power off.

The “modern” OS that should have worked didn’t even complete the boot.

In discussions with a community support person, we decided that there was hardware that was not being properly initialized in the kernel.  I.e., we had a bad Device Tree.

The replacement Banana Pi doesn’t have a supported modern OS.  It is fully supported by Arabian, which is a supported, modern OS.

When I first booted the system, it just worked.  I was thrilled.  It has continued to work properly.

Then I plugged the GPS in.  I could see it blinking.  This indicates that it has a lock and the PPS signal is being sent.

But I can’t get any input on the serial ports.

It turns out that the default device tree doesn’t activate that UART.  Once I figured that out, I had to find an overlay to the device tree to turn on the UART.

That was a pain, but it happened.

Working serial, no PPS.

With the tools on hand, I could monitor the GPIO pin and see the PPS.  But it wasn’t doing anything.

I loaded the correct kernel modules, still no PPS.

My Google Foo suggested that the device tree entry for PPS was missing.

Yep, there was no PPS overlay.

The Linux kernel documentation describes the Device Tree.  But no real examples, and nothing fully commented.

By comparing multiple sources, I finally was able to create a device tree overlay for PPS. I need to figure out how to return that DTD to the community.  The problem is, I don’t know what the hell I did.  I made it work.  I think I know what was done.  Nonetheless, it was truly a case of looking at different device tree overlays and picking out the parts that seemed to match what I needed to do.

I don’t think I’ve had this much difficulty hooking up a piece of hardware since 1983, when I was attempting to attach a DEC 10 MB hard drive to a computer that wasn’t really a DEC.

The only tasks remaining is to put everything in a case and move it to its long-term home, off the top of my computer.

I’ve known about “System On a Chip” and “System On a Board” for many years. I have one of the early embedded software development kits, including some TI chipset.

The most common style of these today is likely the Arduino class of SoC. These things are incredible.

An idea I pitched years ago, for potential military use, was a swarm of small single purpose computers that could be dropped in mass from an aircraft.

Something about the size of a quarter with one or two sensors, a mesh Wi-Fi system, GPS, and a battery. These would drop like the “helicopter” seed pods, scatter over an area, then set up a monitoring network. Using spread spectrum, low power, and burst transmissions, the network might be difficult to detect.

As the sensors detected “things”, they would report to a transmission unit, which would then send a report to home base.

This was all based on a small battery. We figured we could get these things in mass and get them ready to toss out the back of a C130 for less than $100 each.

Today, I can buy a chip that will do that, put it on a custom board with all components for less than $20 in low unit counts.

So SoC, way cool.

The other thing that has been happening is that the physical size requirements for a personal computer have gone way down. Whereas the original XT motherboard was 8.5×11 and the AT was 12×13, we are now seeing Mini-ITX at 6.7×6.7 and even Nano-ITX at 4.7×4.7.

My son found the Mini-ITX form factor computer a few years ago. A full computer that was the size of 3 boxes of .45cal. It weights less.

His computer came with 2 HDMI ports, 4 USB3.2 ports, a headphone jack, Wi-Fi, Bluetooth, and 1Gb Ethernet. It had upgradable memory and at least one M.2 port. This thing just works.

What is cool about it is that it mounts to the back of a monitor. He needs his keyboard and two power supplies.

It is what I got my lovely wife for Christmas this last year.

Can I ditch the big boxes?

Things go in circles. My primary machine lives in a case I purchased almost 20 years ago. It has a new power supply, an extra video card, and a SAS controller. It is designed to handle 9 internal drives plus the optical drive. It can connect to external SAS cabinets.

It was designed to have 36 TB of storage in a ZFS pool. It is heavy. It requires big fans. It is a workhorse.

But, I’m moving away from ZFS. It is a great system, I love it. It just does not meet my current needs. I’ve moved to ceph, a distributed system.

Ceph does not use raid technology the same way that ZFS does. Instead, it depends on having many nodes with lots of redundancy.

With ZFS, my normal was one parity drive for every 4 data drives. So a 20% overhead.

In its fast mode, ceph has a 200% overhead. For every block of data stored, it requires two copies. There are modes that are more optimized, but they all seem to have higher overhead than a raid system.

But because I don’t need to create 5 drive pools, I can do something different. More boxes with just a few drives in each.

I don’t need all that motherboard. I don’t need all that memory. I don’t even need all that much CPU.

Some of the servers I’m using only support two SATA drives, but have an NVMe slot. Still, it is a big box.

My answer was to go looking. I found a cheap ITX-Mini motherboard. It would have been cheaper off the boat, but I found it on Amazon. A used i5 CPU. It will handle an i7 and maybe an i9. A cheap 128GB NVMe, and some DDR3 ram. Toss in a CPU cooler and it is a fully functional system.

It only has a Gigabit Ethernet, but it has a PCI slot. It has four SATA ports.

The cheap case I found has four hot swap bays for SATA drives. It takes a Mini-ITX motherboard. This thing is perfect for what I require.

So yes, this can do the job. I can downsize.

What else is cool about these Mini-ITX boards?

They really are designed with specific markets in mind. I found one with 2 2.5Gb RJ45 and 2 10Gb SFP+ ports. It has GPIO ports, comes in an extruded aluminum heat sink/case. It replaced the old router and everything got faster.

There are versions with multiple RJ45 ports. Different layouts. Different CPU designs. Some come with a CPU on the board, making them a SoB. Some are AMD, some are Intel based.

What I couldn’t find is a board with four SATA ports, a PCIe slot, and GPIO.

Which brings me back to PI

The board I want needs to have hardware-driven Ethernet, GPIO pins, and serial IO. It also has to be cheap. I think I found it in the Banana PI M2 Ultra.

It has everything I need, it was very cheap, less than dinner for three at the fast food joint.

My step backward? It requires power that I can’t give it. GRRR. So it requires a dedicated power supply.

Oh well, my NTP server is almost here.

Correction(s):
I made the mistake of trusting Google’s AI answer.

Speed of light:

Sorry for that. A pepper grain is the size of a picosecond. A nanosecond is around 11.7 inches, which makes much more sense.

My mentor, Mike, had so much to teach me. Coming from University, I knew I was the best programmer on campus and better than anybody I had met.

That changed the day I met Mike. After being introduced to him, I went to my boss and said something like, “That is the most arrogant man I’ve ever met.”

Greg replied, “He’s earned it.”

When I had an opportunity to work with him, I found that yes, he was that good.

He was the first person to stretch my abilities in computer science. I will forever be thankful to him for that.

He had the privilege of meeting Admiral Grace. He was one of the many that were handed her “packet of Nanoseconds”.

This was Grace’s way of getting across to people just how fast computers were running.

In 1 ms, light will travel 299.79 meters. This is a reasonable rifle shot.

In 1 us, light will travel 0.2998 meters (299.8mm), or about 1.2 inches.

In 1 ns, light will travel 0.2998 mm. This is about the size of a single grain of ground pepper.

Just how fast?

My Cray X-MP/48 had a memory bank cycle time of 38ns. This means that light would be able to travel about 10mm or a little less than 0.5 inches.

My memory said that we had a 85ns wait time from accessing memory to having loaded that word into a register.

Your PC likely runs faster than that X/MP. It surely has more memory.

Frames of Reference

As stated above, my world was baby sitting a Cray Super Computer. We worked in nanoseconds. We were trying to optimize code to shave a few nanoseconds out of a loop.

Saving grains of pepper.

When I purchased some study lights for doing photoshoots, I didn’t buy the fancy radio controllers. Instead, I bought “slave” style lights.

With the slave style, you could tell your study light to fire when it detected the flash of another strobe.

Before I purchased these study lights, I went to Mike with concern. I had done the math.

From the moment my flash fired, a long sequence of things had to take place. The light had to travel from my strobe to the detector on the study light. There was a delay while the photoreceptor energized and “fired”. There was still more time as that signal propagated through the circuitry, and finally that light would fire.

My studio lights would be at different distances, we couldn’t even predict the sequence that they would fire.

According to my simple calculations, we could be talking as much as 2ms from the time my light fired until the last study light fired.

Mike pulled me back to the ground. My shutter speed would be set to 1/60 of a second. That is 16.6ms. If the study lights fired anytime while the shutter was open, I would get a good photo.

I was so focused on my reference frame, nanoseconds, I lost sight of the real-world application that was running in 10s of milliseconds.

pkoning Brings Reality to the Clocks

Here is the magic of GPS. It works by knowing the time and then calculating the distance to different satellites.

The more accurate the clock, the more accurate the location.

Communicating that time to an external device is where it gets interesting. The definition of NMEA tells use exactly when the second mark occurs during the transmission of the NMEA message.

Most GPS units default to transmitting at 9600 baud. Which for us is the same as 9600 bits/second. Each 8 bit byte sent requires start and stop bits. My rule of thumb is 10 bits per byte.

This means that it takes around 83ms to transmit one 80 character NMEA sentence from the GPS to the computer.

The instant when the second starts aligns with the edge of a signal of one of the characters in that sentence.

Now my issue was that I thought that the GPS unit had time that was “wrong” it was offset from the real world.

This is not the case. The real reason for the delay is in the time it takes to process the sentence. That should be fixed to the computer, not to the GPS unit.

Which brings us to PPS, or Pulse Per Second. This is a signal that indicates the start of a second. Depending on the GPS unit, this can be at ns accuracy. Even cheap units will get you sub us accuracy.

The processing time to handle the pulse is much lower than to handle a full NMEA sentence.

A PPS can be treated as the “real” time, without fear of being too far away from reality.

A couple of grains of pepper.

There are two network time protocols in use today. One is the NTP protocol, the other is PTP. I have no idea what the PTP looks like, I know that it requires hardware support.

The goal of NTP is to create a local clock that is accurate to less than 1ms from sources that have up to a hundred times that in latency.

The fact that this works at all simply amazes me.

I have 7 servers acting as an NTP cluster. That is to say, they all work to come to a consensus as to what time it is, and then each syncs to that time point.

They do this via actively querying each other every 64 seconds. The protocol knows that the referenced clock time is somewhere within the total time from query to response. Using complex statistics, it can get it much closer than just “somewhere” or “middle”.

As I am writing this, one server believes it knows the time to the network with a standard deviation of less than 500us. It has one nailed down to less than 83us.

Within the local cluster, it believes it knows the time within 50us for all the cluster members. For a few of the cluster members, they agree on the time within 3000ns (3us). That’s not bad.

So what are the problems.

The first problem is that I have a clock that claims to be very accurate, but which I know is slightly wrong.

The clock is fast by 957us with an SD of 57us. I believe it to be worse than that. The issue being that the clock is influencing the rest of the time cluster.

I did that because I had a usable fudge factor for the clock. Now I need to bring it much closer to “real time”.

To that end, I’ve reconfigured the server with the GPS clock to never use the GPS time. Instead, it will use the network servers to converge on “the time”. Once I know “the time” I will be able to adjust the GPS offset better.

The second issue is that USB injects jitter into the signal. We don’t know when the USB port received the time message from the GPS unit. Hopefully, we have a good estimate, but it is still very jittery.

It is nearly impossible to get down to 1ns or less with a clock that has a 500us jitter.

What does this mean? I need to stabilize the time signal. We do that with a PPS. This pulse tells us that the second happened on the rise or fall of the pulse. This is configurable. The second starts on the rising edge of the pulse. With a PPS input, we can reduce jitter to a few nanoseconds.

The issue still is, “What is the offset detecting the leading edge from the “real” start of the second?”

This value comes from the delay along the antenna cable and other such speed of light delays.

Which takes us to the conclusion of this article.

The GPS units I purchased came with a small ceramic antenna. The antenna is about 10 mm on a side. It has a 10 cm connector. This means the unit and the antenna are very close to each other. The antenna isn’t a great antenna.

With this taped to the inside of the window, I was picking up 3 satellites. I replaced it with a cheap, yet “real” antenna. I’m not locking on to 10 or more satellites. More locks mean better time keeping.

If you are doing this yourself, do yourself a favor and order a real antenna to go with your project.

In addition to being “real”, the cable is 3m long, giving me options on where to place it. And it is a water proof unit.

We are in the process of moving from the image above to the image below.

At least in terms of what the infrastructure looks like.

Today I decommissioned an EdgeRouter 4 which features a “fanless router with a four-core, 1 GHz MIPS64 processor, 3 1Gbit RJ45 ports, and 1G SFP port.”

When they say “MIPS64” you can think of it as being in the same class as an ARM processor. Not a problem for what it is.

The issue was that there are only 1Gb interfaces. That and I’ve come to hate the configuration language.

This has been replaced with a pfSense router running on a TopTon “thing.” I call it a thing because it is from China and intended to be rebranded. It doesn’t have a real SKU.

It is based on an N100 with 4 cores and 8 threads. 2 2.5Gb Ethernet ports, 2 10Gb SFP+ ports. It can be upgraded and has multiple extras.

Besides the hardware, this is an entirely different animal in terms of what it can do. It is first, and foremost, a firewall. Everything else it does is above and beyond.

It is running NTP with a USB GPS unit attached. It runs DHCP, DNS, HAProxy, OSPF and a few other packages. The IDS/IPS system is running in notify mode at this time. That will be changed to full functionality very shortly.

So what’s the issue? The issue is that everything changed.

On the side, as I was replacing the router, I jiggled one of the Ceph servers. Jiggling it caused it to use just a few watts more, and the power supply gave out. It is a non-standard power supply, so it will be a day or two before the replacement arrives.

When I went to plug the fiber in, the fiber was too short. This required moving slack from the other end of the fiber back towards the router to have enough length where it was needed.

Having done this, plugging in the fiber gave me a dark result. I did a bit of diagnostic testing, isolated the issue to that one piece of fiber. I ran spare fiber to a different switch that was on the correct subnet, flashy lights.

Turns out that I had to degrade the fiber from the other router to work with the EdgeRouter 4. Once I took that off, the port did light off. But that was a few steps down the road.

Now the issue is that all the Wi-Fi access points have gone dark. Seems that they are not happy. This required reinstalling the control software and moving them from the old control software instance to the new one. Once that was done, I could see the error message from the access point complaining about a bad DHCP server.

After fighting this for far too long, I finally figured out that the pseudo Cisco like router was not forwarding DHCP packets within the same VLAN. I could not make it work. So I disabled the DHCP server on the new router/firewall and moved it back to the Cisco like router. Finally, Wi-Fi for the phones and everything seems to be working.

At which point I can’t log into the Vine of Liberty.

I can see the pages, I can’t log into the admin side. It is timing out.

3 hours later, I figured out that there was a bad DNS setting on the servers. The software reaches out to an external site for multiple reasons. The DNS lookup was taking so long that the connection was dropping.

I think this is an issue that I have just resolved.

But there’s more.

Even after I got the DNS cleaned up, many servers couldn’t touch base with the external monitoring servers. Why?

Routing all looked good, until things hit the firewall. Then it stopped.

Checking the rules, everything looks good. Checking from my box, everything works. It is only these servers.

Was it routing? Nope, that was working fine.

That was one thing that just worked. When I turned down the old router, the new router distributed routing information correctly and took over instantly.

So the issue is that pfSense “just works.” That is, there are default configurations that do the right thing out of the box.

One of those things is outbound firewall rules.

Anything on the LAN network is properly filtered and works.

But what is the definition of the LAN network? It is the subnet directly connected to the LAN interface(s).

Because I knew that I would need to be able to access the routers if routing goes wrong, my computer has a direct connection to the LAN Network attached to the routers. The Wi-Fi access points live in on the same subnet. So everything for my machine and the wireless devices “just worked”

The rest of the servers are on isolating subnets. That are part of the building LAN but they are not part of the “LAN Network”.

I know this, I defined an alias that contains all the building networks.

Once I added that to the firewall rules, it just worked.

Tomorrow’s tasks include more DHCP fights and moving away from Traefik. Which means making better use of the Ingress network.

I have hundreds of dollars worth of GPS equipment. Not counting the cell phones we all carry with us.

I wanted to try to create a Stratum 0 NTP clock.

The last time I attempted this, I used a Garmin handheld GPS. Time to sync was in minutes and while the power draw as trivial, by the standards of the day, it would still burn through AA batteries.

Because you, kind readers, told me that there were cheap options, I went looking.

What I found was a GPS module that is about an inch square. For $15 I could have one delivered. It comes with a header containing VCC, GND, TXD, RXD, and PPS. I figured I could solder in the provided header then run them to a GPIO that has an attached UART.

Well, the darn things showed up a day early, and I didn’t really want to do any soldering. I plugged it in via the USB port, put it in the window. A few minutes later, it had a hard lock.

After installing gpsd and configuring, chrony I now have a system that is locked at less than 1ms accuracy, NOT using the PPS option.

That will be next week’s project. Getting that PPS signal to the motherboard.

If I had a Raspberry Pi with a good interface, not wifi, I can see that this would make a darn nice little timekeeper.

My wife read my article on passwords and “got it”. Which is nice. I was attempting to explain how password crackers use rule sets to modify input dictionaries to create more guesses from a single word list.

I decided to see how much things have advanced. To say I was shocked would be an understatement.

In 2013, the game “Battlefield” was hacked and the entire password database was captured.

This is not the major security threat you might instantly leap to, but it is bad.

Stealing Passwords

I worked in the Systems Group at my University. We were tasked with all software maintenance, installations, upgrades, and in house improvements to the operating system.

The systems group had taken the original manufacturer’s operating system and extended it to the point where it was no longer the same operating system. Having done this, we gave back all the code we had written to the manufacturer, who incorporated what they liked into their next release.

We had developed a long term backup plan. This plan was three tiered. We took daily backups of the entire file system. This was a rolling tape backup. There were 30 days of daily backups performed before the first tape was overwritten.

We also performed weekly backups. There were 52 weeks of weekly backups. So a total of 82 backup sets.

In addition to this, we did end of term backups. These were done just after the term ended. These tapes were kept.

What this meant was that if your file were to live for at least 24 hours, you would be able to recover to any particular day in the past 5 weeks of your file.

If your file were to exist over a weekend, you could recover that file to how it was on the weekend it was dumped for the past year. And if your file were to exist over the term break, it would exist for the lifetime of the storage. 9 track tapes now being dead, I’m not sure what the University did to preserve those old tapes.

In addition to these backups, we took a separate backup of the “password” file once a day. There were 30+ days of password file backups.

That is the setup. The actual story:

We used to give tours of the machine room. The operators enjoyed bragging about the quality of our backup system.

One of these tours, a little monster took one of the password backup tapes and put it in his backpack. He walked out of the machine room with that tape. Nobody noticed the missing tape for the next 30 days.

Said monster took that tape over to the engineering department, where they had their own 9 track tape drives. He read in the file.

He was presented with 10s of thousands of clear text passwords.

This had financial implications because we sold computer time.

We changed our policy to always encrypt the password file before it was written to tape. I have no idea if that encryption standard was any better than Sunday comic page ciphers.

No more Plain Text Passwords

The number of times somebody in a movie has gotten the idiot to give them somebody else’s password is astronomical. The truth is that most passwords are stored in an “encrypted” format. We don’t have access to your password.

We can reset your password, but we can’t tell you what it is because that isn’t recorded.

At the university, they were still storing passwords in plain text. They only encrypted the password when it was written to tape.

Modern systems store that password in an encrypted format. The old method was what is called “descrypt”.

The first two characters of the encrypted password is the “salt” and the rest is the DES hash of the password. This is NOT the same as encrypting your password with a secret and then being able to decrypt it with that same secret. Instead, we use your password to encrypt a given, known, piece of text. The encrypted result is what is stored.

When you provide your password, we encrypt the same text string with your password. If the resulting text matches what we have stored, you have proven you know the password.

Here are a couple of hashed passwords: SD2PFyBHY1oUY, q5M9nJsU/JSwI, sTd5NrAIMrisU, 8MbLuguRAeo92, $1$OcbNKu2y$l9faj.aCWodfonXiSlgnV0, $1$hh765lOJ$lrZ4jkCtUkG3qPBuFJQ/2., $5$2W0fdlfY.a/iXErF$xbzHcX8CfPc89vJkxsiC/BjDmqxI20Yk.Vj9OLL/6e2, and $5$HxfQ9B30d8GdmyPo$J6FWaeGKSez2cLbw3cktvaYgPvsTFaXdMzYp4yDcQjD.

These are all hashes of the same password, “hello world!”

Slow Them Down

Storing passwords in plain text is stupid. But computers are faster than you think. Thus, we want to slow down the speed at which computers can make guesses.

We do this by using a salt.

Consider the situation where you had 74,577,451,608 guesses you wanted to try. If you were to create the hash for each of those guesses, it might take you a bit of time. In the end, you would have them all. Now it is only seconds to look up the hash in a database/file and get the plaintext password used to generate that hash.

To fight this, we use the salt. The salt modifies the hashing process such that for any given password, there are many possible hashes to represent that password.

As shown above, even when using the same “hashing algorithm” we got many results.

This is to slow the guessing of passwords down.

And the results

In 2013, the game “battlefield” was cracked. They escaped with around a 1/4 million password hashes. These are not clear text, you can’t just type them into an account and get in, they are still “protected”.

I used a starting source of 184,000 known passwords. To this, I added an American and a British word list. I didn’t bother to get name lists for a total of 282,000 unique test words.

In the simplest case, with no salt applied, that is 184,000 * 282,000 different combinations to test.

In 2 minutes and 50 seconds, on my medium GPU and medium CPU, we tested 74,577,451,608 different passwords against 282,546 password hashes.

We were able to guess 7.30% of the passwords, or, 30943 passwords.

That is more than enough to make money.

And you can see how bad they can be.