Skills

The Weekly Feast – Garlic Pull-Apart Bread

This stuff is crack. It’s worse than potato chips. You can’t just eat one. They’re so yummy that you won’t be able to help yourself. It can be made with margarine instead of butter and it turns out okay, but if you can digest dairy, use butter.

Ingredients for the dough:

  • 1/4 cup warm water
  • 1 tbsp active dry yeast
  • 1-1/2 tbsp sugar
  • 1 cup milk, warm
  • 2 large eggs, beaten
  • 1-3/4 tsp salt
  • 1 cup butter, melted
  • 5 cups bread flour
  • no-stick spray

Ingredients for the garlic butter:

  • 1/2 cup butter
  • 8 garlic cloves, grated or minced very fine
  • 1 tsp salt
  • 1/4 cup finely chopped parsley
  • 1 tbsp finely chopped rosemary
  • 1/4 cup shredded Parmesan cheese

First, gently whisk together the water, yeast, and 1/2 tablespoon of the sugar in a bowl. Let this proof until it’s bubbly and smells yeasty, at least five minutes.

To the yeast mixture, add in the milk, eggs, salt, butter, and the rest of the sugar. Beat these together with a wooden spoon or with a dough hook in your stand mixer. Slowly begin to add the flour, half a cup at a time.

Once all the flour is incorporated, knead the dough until it’s smooth and elastic. This will take 5 to 10 minutes in a stand mixer, or up to 20 minutes by hand. Always finish up your dough by hand, so you can tell when it’s ready. Place the kneaded dough into a lightly oiled bowl and turn it to cover all the dough with the oil. Cover with plastic wrap or a grocery bag with no holes, and let it rise until it’s doubled in size. This will take one to two hours.

While the dough is rising, make your garlic butter. Combine the butter, garlic, and salt in a small saucepan. Heat over a medium low heat and stir occasionally, until the butter is completely melted. Remove it from the heat, and stir in the parsley and rosemary. Reserve a tablespoon of the garlic butter for brushing on the finished bread.

Lightly grease two loaf pans (your favorite no-stick spray works great for this). Turn out the dough onto a lightly floured surface. Divide the dough into about 40 pieces. Shape each piece into a ball. Roll half the dough balls in the garlic butter, and arrange them in the bottom of the prepared loaf pans. Sprinkle with half of the Parmesan. Repeat with the rest of the dough balls. If you have any of the garlic butter left (except the reserved tablespoon), drizzle it evenly over the loaves.

Cover your pans loosely with plastic wrap or grocery bags, and let the dough rise until it has doubled in size. This will not take as long as the first rise, so plan on it being 45 minutes to an hour. Preheat your oven to 350°F during the last 20 minutes of the rise.

Uncover your loaves and put the pans in your preheated oven. Bake the bread until the tops are golden brown. If you have a thermometer, the interior should be about 200°F, which should take about 35 to 40 minutes to bake. Remove the loaves from the oven and brush the tops lightly with the reserved tablespoon of garlic butter. If necessary, reheat the butter to melt it.

Cool the loaves in the pans for five minutes, then remove the bread from the pans. Serve while still warm, with a side of pizza sauce!

Pipe Whipped Cream in Center Filling. Making Boston Banoffee Pie.

One Step Forward n Steps Back (geek)

I’ve known about “System On a Chip” and “System On a Board” for many years. I have one of the early embedded software development kits, including some TI chipset.

The most common style of these today is likely the Arduino class of SoC. These things are incredible.

An idea I pitched years ago, for potential military use, was a swarm of small single purpose computers that could be dropped in mass from an aircraft.

Something about the size of a quarter with one or two sensors, a mesh Wi-Fi system, GPS, and a battery. These would drop like the “helicopter” seed pods, scatter over an area, then set up a monitoring network. Using spread spectrum, low power, and burst transmissions, the network might be difficult to detect.

As the sensors detected “things”, they would report to a transmission unit, which would then send a report to home base.

This was all based on a small battery. We figured we could get these things in mass and get them ready to toss out the back of a C130 for less than $100 each.

Today, I can buy a chip that will do that, put it on a custom board with all components for less than $20 in low unit counts.

So SoC, way cool.

The other thing that has been happening is that the physical size requirements for a personal computer have gone way down. Whereas the original XT motherboard was 8.5×11 and the AT was 12×13, we are now seeing Mini-ITX at 6.7×6.7 and even Nano-ITX at 4.7×4.7.

My son found the Mini-ITX form factor computer a few years ago. A full computer that was the size of 3 boxes of .45cal. It weights less.

His computer came with 2 HDMI ports, 4 USB3.2 ports, a headphone jack, Wi-Fi, Bluetooth, and 1Gb Ethernet. It had upgradable memory and at least one M.2 port. This thing just works.

What is cool about it is that it mounts to the back of a monitor. He needs his keyboard and two power supplies.

It is what I got my lovely wife for Christmas this last year.

Can I ditch the big boxes?

Things go in circles. My primary machine lives in a case I purchased almost 20 years ago. It has a new power supply, an extra video card, and a SAS controller. It is designed to handle 9 internal drives plus the optical drive. It can connect to external SAS cabinets.

It was designed to have 36 TB of storage in a ZFS pool. It is heavy. It requires big fans. It is a workhorse.

But, I’m moving away from ZFS. It is a great system, I love it. It just does not meet my current needs. I’ve moved to ceph, a distributed system.

Ceph does not use raid technology the same way that ZFS does. Instead, it depends on having many nodes with lots of redundancy.

With ZFS, my normal was one parity drive for every 4 data drives. So a 20% overhead.

In its fast mode, ceph has a 200% overhead. For every block of data stored, it requires two copies. There are modes that are more optimized, but they all seem to have higher overhead than a raid system.

But because I don’t need to create 5 drive pools, I can do something different. More boxes with just a few drives in each.

I don’t need all that motherboard. I don’t need all that memory. I don’t even need all that much CPU.

Some of the servers I’m using only support two SATA drives, but have an NVMe slot. Still, it is a big box.

My answer was to go looking. I found a cheap ITX-Mini motherboard. It would have been cheaper off the boat, but I found it on Amazon. A used i5 CPU. It will handle an i7 and maybe an i9. A cheap 128GB NVMe, and some DDR3 ram. Toss in a CPU cooler and it is a fully functional system.

It only has a Gigabit Ethernet, but it has a PCI slot. It has four SATA ports.

The cheap case I found has four hot swap bays for SATA drives. It takes a Mini-ITX motherboard. This thing is perfect for what I require.

So yes, this can do the job. I can downsize.

What else is cool about these Mini-ITX boards?

They really are designed with specific markets in mind. I found one with 2 2.5Gb RJ45 and 2 10Gb SFP+ ports. It has GPIO ports, comes in an extruded aluminum heat sink/case. It replaced the old router and everything got faster.

There are versions with multiple RJ45 ports. Different layouts. Different CPU designs. Some come with a CPU on the board, making them a SoB. Some are AMD, some are Intel based.

What I couldn’t find is a board with four SATA ports, a PCIe slot, and GPIO.

Which brings me back to PI

The board I want needs to have hardware-driven Ethernet, GPIO pins, and serial IO. It also has to be cheap. I think I found it in the Banana PI M2 Ultra.

It has everything I need, it was very cheap, less than dinner for three at the fast food joint.

My step backward? It requires power that I can’t give it. GRRR. So it requires a dedicated power supply.

Oh well, my NTP server is almost here.

Prepping – Fitness

I’m writing this while I pant heavily, sitting in my chair at my desk. I’ve just finished cleaning bunny cages and exercising, and I’m dripping with sweat despite it being a mere 58*F in my room. I’m exhausted and aching. I am not fit.

I’ve seen this topic touched on a few times in the prepping world, but not really in a practical manner. Fitness is something that doesn’t really exist in the “Rule of Threes” or in common prepping documents. It does matter, though. So much so, that I feel it belongs under “three minutes without air” because right now, I’m panting. I mentioned that.

I get practical exercise every summer. I spent the weekends (2 to 3 overnights) living in a tent and cooking over a fire with cast iron. I’ve had some of our (very fit) fighters carry my cast iron pots to the table for me at the end of a long day, and they’re always astonished at the weight that I “seemingly easily” lug around all day. And it’s that weekly practice (or daily really) that makes it possible for my fat ass to haul this stuff over and around camp.

Our bodies were made to keep us alive under horrendous circumstances. The whole “fight or flight” thing is part of our basic human wiring. This means that our metabolism likes to find sugar, salt, and fat. We crave it! When you’re a Scotsman in the highlands above Edinburgh, dodging the English invaders, it makes sense. You need to find those things that will keep your body working. When you’re sitting at a desk typing emails all day, not so much. It works against us. We want to sit and do the things that stimulate our brains, but unlike even 20 years ago, pretty much everything that stimulates our brains is right here at our fingertips (with Doordash being a thing, doubly so).

It behooves each of us to get up off our duffs five to seven times a week, and move around. I don’t mean doing the dishes (though do those too). I mean exercise of the hot, sweaty, uncomfortable kind. If you’re very out of shape, going for a 20 minute walk five times a week will improve pretty much everything: mind, body, and spirit. Lifting weights for a similar amount of time will do the same thing. Ditto with using a ski machine, swimming, playing a physical game like tennis or soccer, going roller skating or ice skating, and running bases with your kids.

If you can’t do any of those things, chair exercises exist. That’s where I started, so don’t be embarrassed. You have to start somewhere, and no one needs to know. But you need to do it. This is a NEED, not a want or a desire. If you can’t pick up and go because of your lack of fitness, then the problem is 100% you. Please note, people with physical disabilities and such, who simply *cannot* do it, are exempt from this shaming. If you can’t, you can’t, and I get that. But if you can and you just don’t want to, that’s on you.

I really do get it. I don’t like getting sweaty except in one way. I don’t like it when my body aches. I don’t like being on a treadmill or bike machine. It’s boring and stupid and I hate it. But I need to do it, because if I don’t, I won’t be carrying even a half load in my pack, never mind a full load. If I can’t carry a full load, there’s not a whole lot of point in my bugging out, because I won’t get far.

So… what are you doing to make yourself more able, more fit, and more in shape?

A Little Shake of Pepper (what is a nanosecond)

Correction(s):
I made the mistake of trusting Google’s AI answer.

Speed of light:

Time Distance
1s 299,792,458m
0.001s(1ms) 299,792.458m
0.000001s(1us) 299.792m
0.000000001s(1ns) 0.2997m (299.7mm)
0.000000000001s(1ps) 0.0002997m (0.2997mm)

Sorry for that. A pepper grain is the size of a picosecond. A nanosecond is around 11.7 inches, which makes much more sense.


My mentor, Mike, had so much to teach me. Coming from University, I knew I was the best programmer on campus and better than anybody I had met.

That changed the day I met Mike. After being introduced to him, I went to my boss and said something like, “That is the most arrogant man I’ve ever met.”

Greg replied, “He’s earned it.”

When I had an opportunity to work with him, I found that yes, he was that good.

He was the first person to stretch my abilities in computer science. I will forever be thankful to him for that.

He had the privilege of meeting Admiral Grace. He was one of the many that were handed her “packet of Nanoseconds”.

This was Grace’s way of getting across to people just how fast computers were running.

In 1 ms, light will travel 299.79 meters. This is a reasonable rifle shot.

In 1 us, light will travel 0.2998 meters (299.8mm), or about 1.2 inches.

In 1 ns, light will travel 0.2998 mm. This is about the size of a single grain of ground pepper.

Just how fast?

My Cray X-MP/48 had a memory bank cycle time of 38ns. This means that light would be able to travel about 10mm or a little less than 0.5 inches.

My memory said that we had a 85ns wait time from accessing memory to having loaded that word into a register.

Your PC likely runs faster than that X/MP. It surely has more memory.

Frames of Reference

As stated above, my world was baby sitting a Cray Super Computer. We worked in nanoseconds. We were trying to optimize code to shave a few nanoseconds out of a loop.

Saving grains of pepper.

When I purchased some study lights for doing photoshoots, I didn’t buy the fancy radio controllers. Instead, I bought “slave” style lights.

With the slave style, you could tell your study light to fire when it detected the flash of another strobe.

Before I purchased these study lights, I went to Mike with concern. I had done the math.

From the moment my flash fired, a long sequence of things had to take place. The light had to travel from my strobe to the detector on the study light. There was a delay while the photoreceptor energized and “fired”. There was still more time as that signal propagated through the circuitry, and finally that light would fire.

My studio lights would be at different distances, we couldn’t even predict the sequence that they would fire.

According to my simple calculations, we could be talking as much as 2ms from the time my light fired until the last study light fired.

Mike pulled me back to the ground. My shutter speed would be set to 1/60 of a second. That is 16.6ms. If the study lights fired anytime while the shutter was open, I would get a good photo.

I was so focused on my reference frame, nanoseconds, I lost sight of the real-world application that was running in 10s of milliseconds.

pkoning Brings Reality to the Clocks

Here is the magic of GPS. It works by knowing the time and then calculating the distance to different satellites.

The more accurate the clock, the more accurate the location.

Communicating that time to an external device is where it gets interesting. The definition of NMEA tells use exactly when the second mark occurs during the transmission of the NMEA message.

Most GPS units default to transmitting at 9600 baud. Which for us is the same as 9600 bits/second. Each 8 bit byte sent requires start and stop bits. My rule of thumb is 10 bits per byte.

This means that it takes around 83ms to transmit one 80 character NMEA sentence from the GPS to the computer.

The instant when the second starts aligns with the edge of a signal of one of the characters in that sentence.

Now my issue was that I thought that the GPS unit had time that was “wrong” it was offset from the real world.

This is not the case. The real reason for the delay is in the time it takes to process the sentence. That should be fixed to the computer, not to the GPS unit.

Which brings us to PPS, or Pulse Per Second. This is a signal that indicates the start of a second. Depending on the GPS unit, this can be at ns accuracy. Even cheap units will get you sub us accuracy.

The processing time to handle the pulse is much lower than to handle a full NMEA sentence.

A PPS can be treated as the “real” time, without fear of being too far away from reality.

A couple of grains of pepper.

The Weekly Feast – Turmeric Meatballs

Meatballs are the ultimate feast food, in my opinion. There are as many ways to make them as there are cooks, and maybe more. This recipe was created based upon a video by Country Life Vlog in Azerbaijan, Turkey. While Aziza (the cook) doesn’t give you amounts or any real instructions, I pieced this together by watching her cooking the meal. It’s incredibly delicious!

Ingredients:

For the broth:

  • 6 cups water
  • 1 lb beef soup bones
  • 1 lb beef, diced
  • 1 lb pork, diced
  • 1 head of garlic, halved across the middle, paper still on it
  • 1 large onion, quartered
  • 1 hot pepper (dehydrated or fresh)
  • salt, pepper, bay leaves

For the meatballs:

  • 1 lb ground beef
  • 1 onion, minced fine
  • 1/3 cup rice, washed well
  • 1/2 tsp salt
  • 1/2 tsp black pepper
  • 1 tbsp turmeric
  • 1 tbsp dry mint (crushed fine)
  • several dates, figs, or prunes

For the rest of the recipe:

  • 3 large potatoes, peeled and split in half the long way
  • 1 can chickpeas, well rinsed
  • several threads of saffron

Make your broth first. Add the broth ingredients into a large pot and bring it to a boil. Reduce the heat and allow it to simmer for at least 3 hours, adding water as necessary to keep it from drying out. If you want to skip this part, you can use 6 cups of beef broth, and add the other ingredients to it, and let it simmer for 30 minutes before moving on.

While the broth is cooking, work on your meatballs. In a large bowl, add all the ingredients except the figs. Mix together by hand, until everything is well distributed. Let this sit, covered, for at least 30 minutes and up to 2 hours.

When the broth is ready, remove all the ingredients and strain it to be sure there’s no hot seedy surprises lurking in the bottom. Return the broth to the pot, and taste it. Add salt and pepper to taste. Bring the broth back to a simmer while you form the meatballs.

For a pound of ground beef, make 3 to 4 meatballs (yes, they are LARGE). Really manhandle the meatballs, slapping them from hand to hand to make them fairly solid. In the center of each meatball, place one fig, and form the ball around it. Set the meatballs aside.

Mix together a tablespoon of turmeric, a cup of water, and a dash of salt, and whisk to combine. Use this turmeric water to “wash” the outside of the meatballs. This adds a bit of flavor, but also smooths the outside of the meatballs to help them hold together better while cooking. As each meatball is done, place it into the simmering broth. Make certain the broth covers the meatballs most of the way. Cover the pot, and allow to simmer.

Peel and slice your potatoes in half now, and slide them into the broth around the meatballs. Add the rinsed chickpeas, a few figs, and the saffron, as well. Be careful not to squish the meatballs, as they aren’t firm yet and could fall apart. If there isn’t enough liquid at this point to cover everything, you can add a bit of beef broth or water, or even a dash of red wine, to bring it up high enough. Using a large spoon, gently nudge the meatballs to make sure all sides are getting evenly cooked, and they aren’t sticking to the bottom of the pot.
Simmer until the meatballs are cooked through and the potatoes are soft and just beginning to crumble a bit. You can check the meatballs with an instant read meat thermometer. They should register at 165°F when they are ready.

Serve up this delicious meal with a side of pickles or beets if you would like to be immersed in Turkish food culture. Alternatively, a slice of bread never goes wrong, either.

Notes:
When I made this soup, I found the broth to be so spicy that I couldn’t eat it. I used a dehydrated poblano pepper, and it was just too much for me. I’m not a big heat person. Family said that it was warm but not hot to them, so your mileage may vary! I served the broth in small bowls on its own, and then put the meatballs, potatoes, and chickpeas on a plate. That way, people could use as much or little of the broth as they wanted. This was a very hearty meal.

The Country Life Vlog video:

happy new year 2025 countdown clock on abstract glittering midnight sky with copy space, festive party invitation card concept for new years eve

Tick Tock, More Clock Stuff

There are two network time protocols in use today. One is the NTP protocol, the other is PTP. I have no idea what the PTP looks like, I know that it requires hardware support.

The goal of NTP is to create a local clock that is accurate to less than 1ms from sources that have up to a hundred times that in latency.

The fact that this works at all simply amazes me.

I have 7 servers acting as an NTP cluster. That is to say, they all work to come to a consensus as to what time it is, and then each syncs to that time point.

They do this via actively querying each other every 64 seconds. The protocol knows that the referenced clock time is somewhere within the total time from query to response. Using complex statistics, it can get it much closer than just “somewhere” or “middle”.

As I am writing this, one server believes it knows the time to the network with a standard deviation of less than 500us. It has one nailed down to less than 83us.

Within the local cluster, it believes it knows the time within 50us for all the cluster members. For a few of the cluster members, they agree on the time within 3000ns (3us). That’s not bad.

So what are the problems.

The first problem is that I have a clock that claims to be very accurate, but which I know is slightly wrong.

The clock is fast by 957us with an SD of 57us. I believe it to be worse than that. The issue being that the clock is influencing the rest of the time cluster.

I did that because I had a usable fudge factor for the clock. Now I need to bring it much closer to “real time”.

To that end, I’ve reconfigured the server with the GPS clock to never use the GPS time. Instead, it will use the network servers to converge on “the time”. Once I know “the time” I will be able to adjust the GPS offset better.

The second issue is that USB injects jitter into the signal. We don’t know when the USB port received the time message from the GPS unit. Hopefully, we have a good estimate, but it is still very jittery.

It is nearly impossible to get down to 1ns or less with a clock that has a 500us jitter.

What does this mean? I need to stabilize the time signal. We do that with a PPS. This pulse tells us that the second happened on the rise or fall of the pulse. This is configurable. The second starts on the rising edge of the pulse. With a PPS input, we can reduce jitter to a few nanoseconds.

The issue still is, “What is the offset detecting the leading edge from the “real” start of the second?”

This value comes from the delay along the antenna cable and other such speed of light delays.

Which takes us to the conclusion of this article.

The GPS units I purchased came with a small ceramic antenna. The antenna is about 10 mm on a side. It has a 10 cm connector. This means the unit and the antenna are very close to each other. The antenna isn’t a great antenna.

With this taped to the inside of the window, I was picking up 3 satellites. I replaced it with a cheap, yet “real” antenna. I’m not locking on to 10 or more satellites. More locks mean better time keeping.

If you are doing this yourself, do yourself a favor and order a real antenna to go with your project.

In addition to being “real”, the cable is 3m long, giving me options on where to place it. And it is a water proof unit.

Prepping – Disease

Mask mandates probably weren’t the best thing out there, but I’ve seen enough evidence to know that masks do help stop the transmission of many illnesses. Vaccine mandates probably aren’t the best thing out there, but a lot of long-standing vaccines are the reason we don’t have people in iron lungs anymore. Disease and illness is a problem that human beings will always have to deal with. Our methods for dealing with them change as we develop better ways of fighting disease, but the fight will always be there.

As a prepper, disease is something we need to be concerned about. There are likely going to be more pandemics in the future (because there have been many in the past, and it’s the type of pattern that doesn’t change much). It’s in our best interest to learn how to deal with the most likely diseases to bother us, should a SHTF scenario happen. So which diseases should we prepare for?

Diarrheal diseases are the first things we’re likely to see. These happen for a variety of reasons, sometimes with very little change. The food you are eating may change if a large scale emergency happens, and that can cause changes in bowel movements (in either direction, I might add, and both are bad). The “big” diarrheal diseases are cholera, typhoid, gastroenteritis, and dysentery. Changes in food and sanitation will make these four diseases something to be feared. You combat them by making sure you’ve addressed good hygiene in your emergency preparations. Be sure to have clean water to wash your hands in, if nothing else. Keep bleach in your preps so that you can sanitize things. Be prepared to use boiling water to sanitize some things. Good sanitation will take care of a lot of the things in this category, but not all. Consider keeping hydration mixes on hand in case of gastro or other diseases causing dehydration.

Nutritional diseases will eventually show up. Lack of vitamin C can cause scurvy. Vitamin D deficiency can bring on rickets and osteomalacia. Low iron and zinc can bring on all sorts of issues. Eating too much protein, or not enough protein, can cause mental confusion. Severe weight loss due to lack of food or lack of GOOD food can cause problems, too. We tend to think of losing weight as a positive thing for SHTF, but it isn’t. Rapid weight loss of that kind can bring on a bevy of health problems that we’ll be in no shape to deal with. Read More

The Weekly Feast – Oyster Soup

“In all her life Laura had never tasted anything so good as that savory, fragrant, sea-tasting hot milk, with golden dots of melted cream and black specks of pepper on its top, and the little dark canned oysters at its bottom. She sipped slowly, slowly from her spoon, to keep that taste going over her tongue as long as she could.” — from By the Shores of Silver Lake by Laura Ingalls Wilder, pp 204

Anyone who knows me, knows that I have an uncontrollable fascination with the Little House series. It was my introduction to Christianity, and the reason why I invited the minister to dinner when we moved to New England (Ma insisted it was right and proper, so therefore it was what I ought to do, yes?). I’ve been through the series so many times that I’ve had to buy new copies on several occasions, the older ones having worn out. I learned morals and ethics from them. For me, Laura and Ma and Pa and the other people there are just as real as you and me.

Several years ago (several severals of years ago), I was living on the west coast and had managed to become unemployed and rather destitute. I was scraping by on unemployment insurance payments, but it was pretty dicey. My partner D and I were approaching the Christmas and Yule season with as much joy as we could muster. After all, we had a roof over our heads, heat, and each other. It was lean, but love fills a lot of gaps.

Some kind soul had told the local fire department that we were living lean over the season, and a soft spoken gentleman brought us a hamper of food. I tried to protest, but he insisted that it was alright, we weren’t taking anything from someone else. I’ll admit, once he was gone, I tore into that box like … well, like it was Christmas morning. D and I went through the rice and pasta, a tiny canned ham, some fresh vegetables, and then at the very bottom we found the single precious can of smoked oysters.

We could have eaten that can of oysters in two seconds. We’re both in love with them, their smoky flavor, savory and oily… But I looked at him and ran to the book shelf. I pulled out “By the Shores of Silver Lake” and went skimming through it to find the New Year’s Eve scene. There it was, Laura’s description of the oyster soup Ma had made for their guests. He and I started laughing, and we recreated that soup for Christmas Eve for ourselves. It was a wonderful meal.

A while ago (before I couldn’t handle dairy anymore), I wanted to make the soup again. I remember how delicious it was way back when I was barely an adult. Tastes change, though, and I wondered if it would still be as magical. I picked up three cans of cheap smoked oysters and sacrificed some of my coffee half-and-half, and made the soup as a starter to our Yule meal last night.

Everyone enjoyed it. I made enough that I assumed there would be much in the way of leftovers, but there wasn’t. Barely a drop was left in my soup tureen when we were done! It was just as Laura described it, with the oil and butter, the salty sea taste.
Read More

It’s Late, Nerd Babble/status

We are in the process of moving from the image above to the image below.
Server room data center with rows of server racks. 3d illustration

At least in terms of what the infrastructure looks like.

Today I decommissioned an EdgeRouter 4 which features a “fanless router with a four-core, 1 GHz MIPS64 processor, 3 1Gbit RJ45 ports, and 1G SFP port.”

When they say “MIPS64” you can think of it as being in the same class as an ARM processor. Not a problem for what it is.

The issue was that there are only 1Gb interfaces. That and I’ve come to hate the configuration language.

This has been replaced with a pfSense router running on a TopTon “thing.” I call it a thing because it is from China and intended to be rebranded. It doesn’t have a real SKU.

It is based on an N100 with 4 cores and 8 threads. 2 2.5Gb Ethernet ports, 2 10Gb SFP+ ports. It can be upgraded and has multiple extras.

Besides the hardware, this is an entirely different animal in terms of what it can do. It is first, and foremost, a firewall. Everything else it does is above and beyond.

It is running NTP with a USB GPS unit attached. It runs DHCP, DNS, HAProxy, OSPF and a few other packages. The IDS/IPS system is running in notify mode at this time. That will be changed to full functionality very shortly.

So what’s the issue? The issue is that everything changed.

On the side, as I was replacing the router, I jiggled one of the Ceph servers. Jiggling it caused it to use just a few watts more, and the power supply gave out. It is a non-standard power supply, so it will be a day or two before the replacement arrives.

When I went to plug the fiber in, the fiber was too short. This required moving slack from the other end of the fiber back towards the router to have enough length where it was needed.

Having done this, plugging in the fiber gave me a dark result. I did a bit of diagnostic testing, isolated the issue to that one piece of fiber. I ran spare fiber to a different switch that was on the correct subnet, flashy lights.

Turns out that I had to degrade the fiber from the other router to work with the EdgeRouter 4. Once I took that off, the port did light off. But that was a few steps down the road.

Now the issue is that all the Wi-Fi access points have gone dark. Seems that they are not happy. This required reinstalling the control software and moving them from the old control software instance to the new one. Once that was done, I could see the error message from the access point complaining about a bad DHCP server.

After fighting this for far too long, I finally figured out that the pseudo Cisco like router was not forwarding DHCP packets within the same VLAN. I could not make it work. So I disabled the DHCP server on the new router/firewall and moved it back to the Cisco like router. Finally, Wi-Fi for the phones and everything seems to be working.

At which point I can’t log into the Vine of Liberty.

I can see the pages, I can’t log into the admin side. It is timing out.

3 hours later, I figured out that there was a bad DNS setting on the servers. The software reaches out to an external site for multiple reasons. The DNS lookup was taking so long that the connection was dropping.

I think this is an issue that I have just resolved.

But there’s more.

Even after I got the DNS cleaned up, many servers couldn’t touch base with the external monitoring servers. Why?

Routing all looked good, until things hit the firewall. Then it stopped.

Checking the rules, everything looks good. Checking from my box, everything works. It is only these servers.

Was it routing? Nope, that was working fine.

That was one thing that just worked. When I turned down the old router, the new router distributed routing information correctly and took over instantly.

So the issue is that pfSense “just works.” That is, there are default configurations that do the right thing out of the box.

One of those things is outbound firewall rules.

Anything on the LAN network is properly filtered and works.

But what is the definition of the LAN network? It is the subnet directly connected to the LAN interface(s).

Because I knew that I would need to be able to access the routers if routing goes wrong, my computer has a direct connection to the LAN Network attached to the routers. The Wi-Fi access points live in on the same subnet. So everything for my machine and the wireless devices “just worked”

The rest of the servers are on isolating subnets. That are part of the building LAN but they are not part of the “LAN Network”.

I know this, I defined an alias that contains all the building networks.

Once I added that to the firewall rules, it just worked.

Tomorrow’s tasks include more DHCP fights and moving away from Traefik. Which means making better use of the Ingress network.

Prepping – The Grey Man

We’ve used the term “grey man” a few times over the last couple of years. There’s been a bit of debate over what it is, how useful it is, and when to use it. I wanted to address a bit of that.

For me at least, the “grey man” is the person who just blends in.  You don’t notice him. It isn’t that he dresses in grey, it’s that he’s dressed just like everyone else. He walks like everyone else. He talks like everyone else.

This means that sometimes, the grey man has a gun on his hip (when it’s common and everyone else does), and sometimes it’s concealed. It means sometimes the grey man wears a camo jacket (my neighborhood, for instance, is rife with people who do this), and sometimes a golf shirt and boat shoes. Sometimes he has a “two on the top and one on the sides” and other times he has hair to his waist. It depends entirely on where you are at any given moment.

The best grey man is the one who can switch his look to match his surroundings. We see this in movies, as people like Tom Cruise drop wigs and fake mustaches into trash cans, and turn jackets inside out. In reality, it’s a lot less dramatic. It means taking off your patches when going into big cities, for instance. Wear a plain jacket instead of a camo one. Slip your side carry into your waistband carrier and out of site, rather than having it under am arm or in plain view on your hip.

The big thing that I see right now is the desire to be grey man combating with the desire to just be ourselves and fuck the Left. I think there’s something in the middle, and that it’s important to find that central position. It allows you to swing both ways, to coin a phrase. I like the jacket that Chris has, which has velcro spots for patches. They come on and off easily, and you can simply add the correct camouflage to your outfit, be that a 2A patch, an American flag, or a rainbow.

Only you know what your area is like. I can’t judge that. No one but you and your family can, honestly. I know that in my neighborhood, it’s perfectly okay to be a firearms owner, to enjoy shooting and hunting, and to engage in a variety of household preps like gardening and such. No one gives us a second glance. In Chicago, I would not do a quarter of what I do here in New Hampshire. You have to look around you, and judge how to blend in based on who you are and what you do, and where you live.