Skills

This past weekend, I had the pleasure of attending, vending at, and cooking at the Northfolk Nightmarket in Phillipston, MA. This was its first year, and wow, it was amazing. I did pretty well, and I had a blast. Since this event is Viking themed (though “fantasy” Viking more than historical, they delved into the mythology of Beowulf in a day-long roving play), I decided to both dress as and cook as a Viking woman would. That meant coming up with meals that could have been served in Grylla’s mead hall. I decided to make a pork roast with apples, and a green soup. The soup was delicious, but the pork… It was divine. The following was food for about four or five people (but we were hungry from being out in the cold all day).

Ingredients:

• 1.5 lb pork loin, plain
• 4 apples, rough chopped
• 2 red onions, rough chopped
• 24 oz beer or ale (light, NOT dark)
• salt, pepper, oregano, marjoram, about 1/4 tsp each
• 1 tbsp dried rosemary

Get your fire quite hot and make a good bed of coals to cook in (alternatively, set your oven to 350° F). Over a quick flame (stove burner set to medium high), heat up some olive oil and toss in the apples and onions. Saute them until they begin to soften, but before they start to crumble. Place the pork loin over the vegetables, and sprinkle with the salt, pepper, oregano, and marjoram. Add in the beer, a little at a time so it doesn’t bubble over, until the pork is almost covered (you may need to add more beer later if you don’t cover your pot). Sprinkle the rosemary liberally over the top of the roast, and pop it over the coals for 2 hours.

Check on your pork every 30 minutes or so (or every time a patron asks you what you’re cooking and why does it smell so damn good?), turning it so that every side spends time under the liquid. If the liquid boils out, add more beer or some broth. Continue to cook until the roast is ready to fall apart when poked with a fork. If you’re cooking it in the oven, cook for 2 hours at 350°, then an hour hour or so at 250° while lidded, for the best result.

Remove the pork from the liquid and slice into coins. Using a slotted spoon, pull out the apples and onions and serve them alongside the pork, with a side of rice.

Notes:

I used old apples I’d found forgotten in our crisper drawer. They looked like apples that had been sitting around since autumn, which worked well for my event. Because of that, they were a little older, a little softer, and a little sweeter than a fresh apple. I highly recommend this, because the result was incredible. This came out moist, and absolutely bursting with the flavor of the beer and rosemary. It has a little bit of a sweet immediate taste, with a lovely savory flavor that hits you after.

If you can, I really do recommend cooking this one in cast iron over a fire. It was really easy, and it was very showy for when people came walking by. But the smell of it, and the slight background taste of smoke and ash, just really came together.

I will also say, we didn’t eat it with rice when we were at the market. We ate it with our fingers, dribbling juices into the snow and ice at our feet, and giving no f*’s. LOL… It was just so good!

That would be me.

Everything finally came together with the new system. Then I went and messed it all up.

The motherboard has a weak Ethernet. It is a 10/100 Ethernet, which is NOT a problem for a management interface. When I upgrade the box to have full redundancy, it will get a dual port fiber card.

What it does mean is that my Wi-Fi to it via a USB dongle is faster than if I were to plug it in.

Once the box was in position, I connected via Wi-Fi and finished configuration. I tested all the connectivity, and it all just worked.

At that point, I told it to join the cluster. It did with pleasure, and brought the cluster to a stop.

Did you catch my mistake? Yeah, I left that dongle in.

At the bottom of the barrel, we have 10base-T. I have some old switches in boxes that might support that. Above that is 100base-T, which is a good management speed. We can move data for upgrades and restores, but not the fastest. Some of my switches and routers do not support 100baseT.

Above that is where we start to get into “real” speeds. Gigabit Ethernet, or GigE. I’ve now moved to the next step, which is ports supporting 10G over fiber or cable, depending on the module I use. The next step-up would be 25Gbit. I’m not ready for that leap of cost.

Wi-Fi sits at around 200Mbit/s. Faster than “fast Ethernet” also known as 100base-T, but not at “real” speeds. Additionally, Wi-Fi is shared space, which means that it doesn’t always give that much.

So what happened? The Ceph(NAS) cluster is configured over an OVN logical network on 10.1.0.0/24. All Ceph nodes live on this network. Clients that consume Ceph services will also attach to this network. No issues.

When you configure an OVN node, you tell the cluster what IP address to use for tunnels back to the new node. All well and good.

The 10G network connection goes to the primary router and from there to the rest of the ceph nodes. One of the subnets holds my work server. My work server provides 20Tb to the ceph cluster.

On that subnet are also the wireless access points.

So the new node correctly sent packets to all the ceph nodes via the 10G interface, EXCEPT for traffic to my work server. Why? Because the 10G had a 1 hop cost, while the Wi-Fi had a 0 hop cost. By routing standards, the 200Mbit Wi-Fi was the closer, faster, connection than the 1 hop 10G connections.

When I found the connection problem and recognized the issue, I unplugged the Wi-Fi dongle from the new node and all my issues cleaned up, almost instantly.

It’s time to start thinking about gardening. This is not just a pleasant skill that yields tomatoes. It’s a post-apocalyptic skill that is absolutely necessary. Have you ever tried just hunting and gathering? If not, don’t bother; there’s a reason we cultivated plants. Learning to garden now, when we have ample food at the grocery store (because even a lightly stocked grocery store is ample, quite frankly), is imperative. This is not a skill you can learn after the fact. The learning curve is so sharp that it requires early adoption and constant practice.

There are lots of easy plants to grow in a new garden, and I’ll talk about them next week. This week, I want to talk about planning. First off, it’s the best part of this part of winter (the crappy, cold, damp, windy part). You get to huddle near the fireplace and look through seed catalogs (on or offline, your choice), and dream. Dreams are seeds of the mind, after all. Stage one of growing food is literally dreaming about it.

Make a rough map of your yard. This is important because there are several issues that you need to address:

• Where in your yard gets full sun? partial sun? no sun at all?
• What’s the type of soil you have (sandy, rich, damp, clay, etc)? You may have to amend your soil to grow anything, so you need to know this in advance.
• What kind of garden do you want to try (raised beds? containers? little pots? big pots? half an acre plowed by hand?), and how can you do it to best utilize the sun you get?
• What is your Plant Hardiness Zone? Find out on the USDA website. As an example, I’m 5b, which means something. More on that later.
• How much time do you want to spend on your garden? Remember it’s not just planting. You also have to factor in weeding, watering, fertilizing in some cases, weeding, helping plants with frames or structures, weeding…
• What will you eat? Don’t bother growing something you know your family won’t eat unless you have a very good reason for doing so. Plant what you’ll eat.

When you pick out your best spot (with 6+ hours of sunlight a day, if at all possible), you’ll want to sketch it out in a notebook or using a computer program. I can’t draw a straight line to save my life so I use programs online. There are several:

• The Old Farmer’s Almanac Garden Planner (7 day free trial)
• GrowVeg (this is the one I used when we had a big garden)
• myGarden Planner (okay, free, not as intuitive to use)
• VegPlotter (nice, limited free option, inexpensive if you decide to use it)

Read More

Chicken and dumplings are a staple in my house. They’re something in between a soup and a stew, and I usually get silly and call it Stoup. They’re incredibly easy to make, though it takes a bit of effort until you’ve learned the method. It’s filling, delicious, and you’ll get requests.

Ingredients for the stoup:

• 6 to 8 oz of uncooked chicken per person
• enough water to cover the chicken
• salt, pepper, oregano, thyme, and rubbed sage to taste
• 1 medium carrot, diced small
• 1 small onion, diced small
• 1 large rib of celery, diced small
• a teaspoon or so of butter, margarine, or olive oil
• white wine to deglaze the pan

Ingredients for the dumplings:

• 3 cups all-purpose flour
• 1.5 teaspoons salt
• 1 cup milk
• 1 egg

The chicken for this recipe can be done in two different ways. First, you can use bone-in chicken pieces and make it like “pulled chicken,” meaning you cook it, then remove it from the water and shred it up until it’s the size of bits you like. Second, you can use boneless, skinless chicken breast or thighs, in which case you’re going to cube your chicken into bite size pieces. Regardless of which method you choose to use, you should sprinkle the chicken with salt and pepper, then brown it in a cast iron pan (separate from the one you’ll be making the broth in). Make sure to brown all sides, but remember it doesn’t need to be cooked through. The full cooking happens in the water.

While you’re browning your chicken in batches, add enough water to a pot that it will cover your chicken when it’s added. It’s okay if you don’t get quite enough in the pot; you can add more after. You just want to get enough in there that you can start heating it up to a boil. As you finish browning chicken parts, put them into the water. Make sure there’s enough water to cover all the chicken completely, but not much more, and then lower it from a boil to a simmer. Add in your spices, about a teaspoon of each for now. You can add more later if needed.

As your chicken is simmering, dice up your carrot, onion, and celery. Add a bit of fat to the pan you cooked the chicken in, and saute your vegetables until the onions are soft and beginning to clarify. Add a tablespoon or so of a dry white wine to the pan, and stir and scrape well with a wooden spoon. All of the stuff you scrape off the bottom of the pan is “fond” and it’s what makes your stoup delicious. Add the veggies and fond to your chicken and broth.

Make your dumplings. Add the four and salt into a medium bowl. In a separate bowl or measuring cup, whisk together your milk and egg. Add the liquid to the flour, and mix until it forms a dough. This should be a soft and relatively smooth dough, with very elastic qualities. It may be a bit sticky or tacky, but it shouldn’t stick to the counter when you’re kneading.

Knead the dough on a lightly floured surface until it’s smooth and elastic. Keeping the counter lightly floured at each stage, you should then roll out your dough so it’s as thick as you like your dumplings to be. If you like your dumplings to be fat and fluffy, you’ll want to roll them out to about a half inch thick. If you want them more like noodles (my preference), roll them to under a quarter inch, basically as smooth and thin as you can manage without the dough sticking to the counter. Cut your dumplings into strips, squares, diamonds, or really any shape you like. you can use a dough cutter, or a knife, or even a rolling pizza cutter. Use a bread scraper to lift the noodles off the counter, dust them well with more flour, and let them rest and dry a bit while the rest of the food is readied.

When your chicken is thoroughly cooked (about an hour, or longer if you like), pull any boned parts out of the broth. Shred, if you like, and return the chicken to the broth. Bring the broth to a low but steady boil, and begin adding the dumplings to it a little at a time. I usually add about a handful of dumplings, then stir and let them begin to cook. This keeps them from sticking to one another. The flour on the outside of the dumplings will help thicken the gravy in your stoup, too. Cook the dumplings until they’re tender. This can take anywhere from five minutes to 20 minutes, depending on how thick they were rolled and how dry they were when you started. They should be solid throughout, with no doughy interior. Thin ones will taste like fresh noodles (which is essentially what they are).

If you find that your gravy isn’t thick enough, add a little water or cold broth to a tablespoon of cornstarch, stir until well combined, and then add a bit at a time to the simmering stoup. Repeat until you reach the consistency you prefer. Add in any spices you like, and if you want a fancier look, top each bowl with a sprinkle of fresh minced parsley.

Serve your chicken and dumplings over a pile of mashed potatoes, rice, or on their own in a bowl with a spoon.

There are a few servers that are too old. There is a need for a few more servers to get a room level redundancy. These things can be expensive.

As I’m cheap, I’ve been using older servers that accept 3.5″ disk drives. Some except 2 drives, some 6, some could accept more, but the case doesn’t.

The fix I chose was to move to some four bay NAS enclosures. This is a reasonable size that balances with the network I/O capability.

These enclosures all take the Mini-ITX motherboard.

These motherboards are nothing short of amazing. In the middle tier, they have all the things a full-size motherboard has. Some have 4 memory slots, some only 2. They come with 1, 2, 4 Ethernet ports. Some have SFP ports. Some have SATA ports. The number of SATA ports ranges from 1 to 6. Some come with PCIe slots.

Depending on what your needs are, there is a motherboard for you.

Since this was going to be a NAS, the motherboard I selected had to have 4 SATA ports, an NVMe slot, and SFP+.

Yep, this exists. They don’t exist at the price point I wanted to pay. It finally clicked with me. I can just put an SFP+ PCIe card into the machine.

Thus, I picked a motherboard with 4 SATA, 1 Ethernet, 1 USB3, 1 PCIe slot, enough memory and 2 M.2 slots.

Some NAS enclosures do not have the opening for a PCI slot, so it was important to pick a case that had the card opening.

When I got the enclosure I was impressed.

It is a sturdy, thick steel case. There is no plastic on the entire thing. There are for hot swap disk bays plus mounting space for 2 2.5″ drives. Exactly what I was looking for.

When I went to install the motherboard, I was shocked to find that the CPU cooler didn’t fit. I ordered a low profile. I’m impressed with that as well.

I get the board mounted. It looks nice. I go to close the case and the cover won’t fit on. The cover has a folded U channel that goes over the bottom rail of the case to lock the case closed.

The problem is that there isn’t enough space between the edge of the motherboard and the bottom rail for the U channel to fit.

My first real use of the right-angle die grinder. I don’t have a cut-off wheel for it, so I just ground the edge away and it worked.

Of course, I gave myself a frost burn because I was too busy to put gloves on to handle the die grinder.

Back to the worktable, the cover now goes on. I plug a wireless USB dongle into the USB 3.0 and boot. Nothing.

It took me a couple of days before I figured it out. The case came with no documentation. The front panel connector has both a USB 3 plug and a USB 3 plug. I plugged both in. You are only supposed to plug in one. Fixed.

The installation happens, I’m happy. It is fast enough, it is responsive enough. I just need to get it put in place with the fiber configured.

I take the cover off the back slot. Go to put the PCI card in.

The (many bad words) slot does not line up with the opening in the back of the case.

The open in the back is off by 0.8 inches.

I consider cutting another card opening in the back. That won’t work. The card would be half out of the side of the case.

I ordered the cutoff wheels for the die grinder, I know I’m going to need them.

I decided to cut the back opening wider. This will leave an opening that can be taped closed on the PCI side. It allows me to use the existing slot with retaining hardware. I good idea.

All I need to do is unscrew the standoffs, drill and tap four holes in the right place, and I’m done.

Except… Those standoffs are pressed into place. They don’t unscrew.

No problem. I have a set of standoffs. I’ll just cut the existing standoffs off. Drill and tap holes in the right place and use my standoffs.

Except… My standoffs are the normal length. These standoffs are a custom length. I can’t do that.

Tools to the rescue

First stop, the arbor press. It is a small 2 ton press. I have no problems pushing out the standoffs. The press also removes the bulge from removing the standoffs.

Next step, the milling machine. Using the gage pins, I found the size of the holes is 0.197-0.198. Measuring the standoffs, I get 0.208. I settled on 0.201 for the hole size. I should have gone a 64th smaller.

There is no way to clamp this thing in the vise. I do have strap clamps. The case is quickly put into position.

The first hold is located, then drilled. No issues.

Except I don’t have enough travel to reach the other three holes. I reposition the case on the table and go for it.

I go back to the arbor press to put the standoffs back in. I don’t have enough height to support the case while installing the standoffs.

Back to the mill. Square to ends of a hunk of aluminum. Punch a 3/8in hole in it. Work on the mill vise and get the standoffs put back in place.

In the middle of this, I have an alarm, fearing that I put the standoffs in the wrong place. I do a quick test fit and everything is perfect.

It takes me a good hour to put the case back together with all the case mods done. It looks good. I’m happy with how it came out.

Today is search day. I have to find the 8 meter OM-4 fiber for this NAS, and I have to find the box of screws that came with the case for the hard drives. Once I have those, this can go into production.

I know what to look for on NAS cases. I’ll be building out a few more of these boxes over the coming months. First to replace two boxes which are too old. One for the redundancy.

The world will be good, or I’ll punch it again and again until it is good.

P.S. This is filler, the article about Trump’s win in the D.C. District court was taking to long.

Hey all, I’ve been considering a variety of topics for the next few weeks. With spring rapidly coming, I could talk about gardening, and explain how to start seeds indoors, and all the stuff that goes along with making a functioning garden that will provide actual food. I could write about raising chickens (and/or other livestock, though I have less experience with non-chicken livestock) and what that takes. I could go into how one makes staples, like bread, cheese, butter, and the like. I could talk about mending and making clothing, blankets, and such.

What do you all want to hear about? My gut says go with gardening, but I also want to write what is of interest and useful to y’all.

I made this last week. It was rich, delicious, filling, and just the right thing after a long day of moving snow.

Ingredients:

• 3 lbs beef, cut into 1” cubes
• salt
• black pepper
• 1 tbsp oil
• 1 pkg bacon, cut into small pieces
• 2 large carrots, peeled, sliced on bias into large chunks
• 1 large onion, chopped
• 4 cloves garlic, minced
• 2 tbsp tomato paste
• 2 cups dry red wine
• 2 cups beef broth
• 2 bay leaves
• 1 tsp dried thyme
• 1/2 lb pearl onions
• 8 oz mushrooms, quartered
• cornstarch mixed with cold beef broth, for thickening

Preheat oven to 350°. Season beef with salt and pepper.

In an oven proof pot, heat a little oil over medium heat. Add the bacon and cook until it’s crispy, stirring occasionally. This will take about six minutes. Pull the bacon out of the pot with a slotted spoon and put it on a paper towel lined plate to drain.

Increase the heat to medium high, and then add in the beef chunks in a single layer. Don’t crowd the pot, as the beef needs to be able to move around freely. Do it in batches, removing properly seared beef and adding it to the pile of bacon.

Once all the beef is seared, remove all but a tablespoon or two of the bacon fat in the pot. Lower the heat to medium, and stir in the carrot and onion. Cook until the onions are wilted and the carrots begin to soften, which should take about five minutes. Stir in the garlic and tomato paste, and cook until fragrant. Add a splash of red wine to the pot and deglaze it, making sure to scrape any “stuff” off the bottom of the pot and incorporate it into the onion mixture.

Add in the beef and bacon, the rest of the wine, broth to cover the meat and veggies entirely, spices, onions, and mushrooms. Stir gently to incorporate, and then cover the pot with a tight fitting lid. Put the pot in the oven and cook until the beef is fork tender and the sauce is slightly thickened. This will take about an hour and a half to two hours.

Remove the pot from the oven, and set it on top of the stove. On a medium high heat, bring the liquid to a low boil. Slowly pour in a tablespoon or so of cornstarch mixed in with some beef broth, and stir the stew well. Continue to add a tablespoon or so of the cornstarch slurry until you reach the desired thickness.

Serve your Beef Bourguignon over mashed potatoes, and with a side of your favorite vegetable.

Notes:

I tend to buy larger cans of tomato paste. When I open a new can, I measure out two tablespoon “plops” onto parchment paper, and put it in the freezer. Once it’s frozen, I stick the plops into a baggie, which I can then pull from whenever I need tomato paste. If you’re using frozen tomato paste, let it simmer until it dissolves into the onion mixture before moving onto the next step.

You can thicken the liquid with a roux if you prefer, but the silkiness of a sauce made with cornstarch is hard to beat. Because cornstarch is entirely starch, you need half or less of the amount you’d need of flour, in order to get the same level of thickness. A roux of flour will tend to make a thick gravy that is dark and heavy, while a roux (or “slurry”) of cornstarch will be lighter both in color and texture. There are places for both, but I prefer the cornstarch in this recipe.

If your family are not big mushroom or onion fans, you can cook those separately in a pan and add them as a topping in each bowl, instead of cooking them in the stew. Again, this is a matter of preference.

Protecting Second Amendment Rights

I know we’re out of season, but this seemed to be a good topic. Butchering applies to all animals, and whether you’re taking something small or large, there’s a level of skill to getting it done. The video is self explanatory, and it’s pretty good. I learned things watching it, and I hope you do, too.

I wanted to talk about butchering in general, though, for those who may not want to sit through an hour long video (though I do recommend doing so when you have time, it’s VERY good). In my very strong opinion, the hardest part about harvesting an animal is killing it. Once the animal is dead, it doesn’t really matter what you do to the body. It’s not going to complain or suffer in any way. So if you can shoot or otherwise dispatch the animal, know that the butchering can go as slow as needed and no one’s going to judge you for your cuts.

The purpose of butchering is to get the meat off the animal and into usable pieces. When you’re talking about large game like deer, elk, moose, bear, and the like, you will be dealing with parts rather than a whole animal. Like in the video, your butchering will consist of taking pieces off the larger carcass, and then preparing them for freezing or otherwise preserving them for long-term storage. Smaller animals like chickens, ducks, geese, possum, squirrel, etc. are small enough to allow you to work with the whole animal, and so the process is slightly different. You can prepare the entire carcass for freezing or preserving, which can be easier (but occasionally is more tedious).

The first order of business with any animal is to remove the guts, the viscera. Generally speaking, this involves opening the stomach cavity from the anus to the ribs, and then carefully pulling everything out. In larger animals this is pretty easy except for the first cuts. I always worry I’m going to puncture something when I’m cutting around the anus, and you don’t want that because you don’t want fecal matter in what’s going to be your food. Basically, you have one long connected tube (or more correctly, set of tubes) that go from mouth to ass, and you need to remove it all. There are internal organs that can be eaten, like the heart, liver, and kidneys. Technically, brain is also edible, as well as other stuff, but the other areas are very much filters for all sorts of things. Liver and kidneys, while filters, aren’t likely to pass anything on to you (especially after both freezing and cooking), but I always avoid the brain, lungs, and other stuff. The sole exception to this is that you can use the large intestines of some animals to make sausage casing (generally you would use pig, but technically any large animal intestine will do), but it requires a lot of cleaning to make it safe. Hides on larger animals can be removed first or last, depending on how you’re storing it during processing.

Read More

The amount of grief I’ve put up with to get this working buggers imagination.

To have a NTP stratum 1 server, you need to have a certain set of capabilities.

First, you need a stratum 0 device. This is an atomic clock or a GPS receiver.

You need a method to communicate with the GPS receiver.

Your clock needs to be network connected.

Each of these pieces must be done correctly with the least amount of jitter possible.

Jitter is how much a signal deviates from its target. If the jitter is zero, then we have a level of accuracy that depends solely on our reference clock.

The little GPS unit is self-contained. If it is supplied 3.3V of power, it will search for satellites and do the calculations to know what time it is and where it is.

The calculations turn out to be for someplace along the cable from the antenna to the GPS unit. Some highly accurate versions of the GPS SoC measure the length of the antenna feed and account for that in the calculations. Regardless, it is the time for a place a little offset from the actual GPS chip.

For me, that is a delay of around 10ns.

The GPS will communicate via a serial protocol. This means that we have a delay from when the message is received and when we can put our timestamp on the message. For me, that is around 140ms.

This can be discovered by tracking the time indicated by the serial GPS and the system/local clock. The local clock is synced to multiple remote NTP servers to get this number.

Unfortunately, there is about a 1ms jitter in this signal.

If I were to use a USB converter. I.e., serial to USB, that jitter goes up. I am seeing a jitter of 4 to 9 ms.

Using the serial directly is a good start.

But there is another signal that can help. That is the Pulse Per Second (PPS). We are using a 1second pulse.

IFF we can capture the time at which the pulse arrives, we can get a very accurate start of the second marker.

This requires that the hardware have a general purpose input/output(GPIO) pin available.

Most motherboards do not have exposed GPIO pins. Worse, some boards have GPIO pins, but there is no documentation on how to access them.

So the server board requires GPIO plus a method of accessing those pins.

There are two ways to discover a change of value, we can pole for it, or we can get an interrupt.

Consider you have your phone alerts silenced so you don’t get a noise every time you receive an email or message.

You have to check your phone for new messages. This is “poling”.

If somebody calls, your phone still rings. You then immediately check to see who it is and perhaps answer the phone.

This is an interrupt.

The default operation of a GPIO pin is poling driven. Even if it is generating an interrupt, that interrupt is only used to record the change of value.

What is needed is a high-performance interrupt handler. When an interrupt happens, the handler records the system clock. A user land process watches, either poling or interrupt, it doesn’t matter, for that value to change.

When it changes, the software knows that the GPS “knew” it was the start of the second when it created the pulse.

The amount of jitter is only as much time as it takes for the system to allocate a CPU and for that CPU to process the interrupt. In other words, really, really fast.

Currently, the jitter on my PPS reference clock is 300ns. Because of the many samples that have been taken, the PPS reference clock is currently running 17ns from the real time. That has been going down over the last few hours. By the time you read this, it is likely to be even less.

The PPS clock is so tight that the other clock sources hide the values, even in logarithmic form

This is an interesting graph, to me, as it indicates how the system clock is slowly being conditioned to keep more accurate time. It software currently says that the drift is -17.796271 ppm off which I think translates to 3.324ms

So how bad was this task?  More painful than I wanted it to be.

I’m fine with “dumb” computers.  I started programming on 6502s.  I’ve been bit slinging for 50 years.  Programming Arduino’s?  No problem.

Building a PC from components, installing any compatible operating system?  I do it a dozen times a week when developing.

The Raspberry Pi is a different animal.  It isn’t sold as a low-level system.  You can use it that way, but that is not how it is intended to be used. It is sold as a System On a Board (SOB) that runs a modern (Linux, Android) operating system.

This is where things get strange. When we are working with modern PCs, they have known hardware.  We boot the computer, run the OS, the OS has drivers to talk to the hardware.  Everything just works.

This is possible because PC’s have a Basic Input Output System (BIOS).  This is a low-level set of routines that are there to allow accessing certain parts of the hardware with a standard Application Protocol Interface (API).

Since every BIOS has the same API, OS vendors can use the BIOS to load enough of their software to continue booting.  The hardware is attached in known ways.  The hardware vendor supplies the drivers for their hardware.  Linux people write their drivers if needed.

So consider that SOB. It has a serial port.  The serial port is controlled by a standard UART.  That UART is programmed in a standard way.  They are all the same.

In order for that UART to work, the software needs to know where the UART is located in memory (or on the I/O bus). In addition, the pins that the UART uses have to be configured for the UART.  Most UART’s use standard pins on the GPIO header. The pins that the UART uses can be used in different modes for different things.

The problem comes from that address being different in every SOB or SOC.  A board could have one, two, or more GPIO driver chips.  It all depends on the designer.

The developers overcome this issue with what is called a “Device Tree”.

The device tree is a parsable description of devices and their locations in memory or on the I/O bus.

The board I purchased doesn’t have a supported modern OS.  The only OS that I could get to boot was released in 2016.  The OS is not really supported anymore.  The board itself was flaky. It would randomly reboot, or just power off.

The “modern” OS that should have worked didn’t even complete the boot.

In discussions with a community support person, we decided that there was hardware that was not being properly initialized in the kernel.  I.e., we had a bad Device Tree.

The replacement Banana Pi doesn’t have a supported modern OS.  It is fully supported by Arabian, which is a supported, modern OS.

When I first booted the system, it just worked.  I was thrilled.  It has continued to work properly.

Then I plugged the GPS in.  I could see it blinking.  This indicates that it has a lock and the PPS signal is being sent.

But I can’t get any input on the serial ports.

It turns out that the default device tree doesn’t activate that UART.  Once I figured that out, I had to find an overlay to the device tree to turn on the UART.

That was a pain, but it happened.

Working serial, no PPS.

With the tools on hand, I could monitor the GPIO pin and see the PPS.  But it wasn’t doing anything.

I loaded the correct kernel modules, still no PPS.

My Google Foo suggested that the device tree entry for PPS was missing.

Yep, there was no PPS overlay.

The Linux kernel documentation describes the Device Tree.  But no real examples, and nothing fully commented.

By comparing multiple sources, I finally was able to create a device tree overlay for PPS. I need to figure out how to return that DTD to the community.  The problem is, I don’t know what the hell I did.  I made it work.  I think I know what was done.  Nonetheless, it was truly a case of looking at different device tree overlays and picking out the parts that seemed to match what I needed to do.

I don’t think I’ve had this much difficulty hooking up a piece of hardware since 1983, when I was attempting to attach a DEC 10 MB hard drive to a computer that wasn’t really a DEC.

The only tasks remaining is to put everything in a case and move it to its long-term home, off the top of my computer.