Skills

You might have heard the phrase, “He’s forgotten more than you will ever know.” When dealing with somebody who is quietly competent, that is almost always the case.

I was there at the start of the Internet. I watched our campus get X.25 networking. Later, BITNET. I watched email get dumped into the UUCP queues and see magic happen as email dropped into a black hole and reappeared where it was supposed to. The magic of ARPANET, later to be The Internet.

I was part of the team that transitioned the Internet from routing tables (Host tables) into the Domain Name System. I watched as we moved from vampire taps on 10Base2 to RFC bayonet connectors. Having to explain over and over that you can’t plug the cable into your computer, you plug the cable into a T and terminate the T. The T then connects to your computer.

The magic of 10BaseT with low-cost hubs instead of expensive switches that “real” network switches cost.

Listening to the stories of Ethernet cards costing “only” 10K because they had bought so many of them.

Today I installed another new NIC into one of my nodes. This NIC cost me $33. The SFP+ module was another $15, call it $45. This gives me a MMF fiber connection, good for up to 300 meters at 10 Gigabit Per Second.

This makes three nodes connected at 10 Gbit. 1 Node at 2.5 Gbit. The rest are still at 1.0 Gbit. When I have finished upgrading the nodes, each will have a 10 Gbit NIC. They will have either MMF LC fiber connectors or 10 Gbit RJ45 copper connectors.

The only reason for the RJ45 copper is that I need to add some more SFP+ routers with extra ports.

What I Forgot

When we installed our for 100BaseT NIC’s, we did some testing to see what the throughput was and how it affected the host computer.

What we found was that the interrupt count went through the roof, bogging the computer down. At full speed, more than 75% of the CPU was dedicated to network traffic.

The cure for this was to increase the packet size. At the time, this was a big issue. Most networking devices only accepted 1500byte Ethernet Packets. If your input packet is larger than the MTU of the egress port, then the packet becomes fragmented. There are issues with IP fragments.

A newly introduced change in the specification allowed Jumbo packets. The normal size of a Jumbo packet is 9000 bytes.

Once we did the upgrade, everything got faster. We actually had network attached drives which were faster than the physically attached drives.

When setting up a VPN, you need to set the packet size going into the VPN to be smaller than the MTU of the physical network. The VPN will encapsulate packets before they are transmitted. This makes the packet larger. If you are sending a packet through the VPN with a size of 1500, and it is going on to a physical network with an MTU of 1500, every packet of 1500 bytes will be fragmented.

I have been slowly bringing up an OVN/Open vSwitch configuration. This allows a virtual machine or a container to move from host to host, maintaining the same IP address and routing path.

I’ve done a couple of live migrations now. The perceived downtime is less than 15 seconds. There were no dropped packets during the migration. Just amazing.

The OVN setup is complex because there are many options that need to be set up, and there are tools to do all of it for you. Unfortunately, the overhead of OpenStack and learning it is something I’m not ready to do. So I’m doing each step by hand.

When my virtual machines were on the same host as the egress bridge, everything worked. If the VM was on a different host within the OVN cluster, ICMP would work, but TCP would not.

Turns out that I had not set the MTU of my physical network correctly. I’ve been slowly updating the networking configuration on all of my nodes to use jumbo packets. As soon as I did that, my cross node networking traffic started working!

Happy, happy, joy, joy.

There is more testing to perform. This might also be a fix for the firewall glitch of a few weeks ago. Once I have a couple of more nodes on the OVN cluster, I can proceed with designing and testing a redundant network design, with failover.

It was a good day. Oh, I brought another 12 TB of disk online as well.

When people begin to prep, there’s this mental thing that happens… they begin to store things in buckets. For some, it’s cat litter buckets. For others, it’s the big white buckets you can often get for free at grocery stores. They’re usually five gallon size, and they have a hard plastic exterior which is difficult for mice to chew through, and a plastic handle that’s decently rugged. The cat litter ones are usually more squarish in shape, which is great for holding ammo, candles, square tins (like Spam), and the like. The white ones are round, and are awesome for rice, wheat, lentils, coffee, etc.

For me, the five gallon bucket was too small after a while. I started getting piles of the things, and I didn’t like it. So I switched from those to Sterelite bins, the light grey ones. Those are alright, but if you stack them more than two high and they’re heavy, they will buckle under the weight. Those were replaced with good quality rigid Rubbermaid bins. The Rubbermaids last, have mouse proof (so far at least, and we’re talking ten years or more in a farm house with tons of mice) exteriors, and a decent seal at the top as well. You can stack them three high if they’re heavy, and four high if you make the top one light.

Once I reached the bin stage in my prepping, I began making single-item bins. This bin was labelled “rice” and contained countless smaller bags or boxes of rice. That one was labelled “wheat” and another was “beans.” You get the general idea. I thought this was an amazing idea! Everything was neat, labelled, easy to find… just perfect.

And then we had an emergency with a power loss of a few days. Suddenly, I had to find our emergency stores in the dark, in the basement. I had to crack 5 gallon buckets and big plastic bins to take out one or two things, and then seal them up again. It wasn’t fun. It made carrying things up the stairs more difficult. I got frustrated.

Now, I make multi-purpose bins. One bin sits in the hallway, tucked into a quiet corner. Each grocery trip, I pick something up and stash it in the bucket. I make sure each bucket has a good mix of protein (canned meat and fish, powdered eggs, peanut butter, a can of nuts, and you name it), carbs (instant potatoes, rice, pasta, flour), and fats (mostly in the form of natural fats in the cans of meat and fish, but sometimes I find canned butter and the like, and also small bottles of vegetable or olive oil). Toss in some paper plates, matches, a P-38 can opener, some fuel and one of the folding mini stoves, along with plastic forks or spoons and a couple of mags of ammo, and you’re all set. Each bucket is self-contained. I can grab ANY bucket, and know it has a bit of everything, and that it’ll be just different enough from the next bucket to ensure my family doesn’t get bored with single-flavor nutrition.

Read More

Squash soups are a favorite of mine. Because dairy and I don’t get along, I can’t indulge in my favorite chowders anymore (I don’t care what anyone says, chowder made with oatmilk is just not the same). Squash soups can be blended to give an impression of being creamy, when there’s not even a drizzle of dairy in them. Also, if you do need to add a bit of milk for flavor, oatmilk will do just fine because it’s a background thing and not the star. This is my recipe for Hubbard squash soup, and it’s really delicious. It’s great when you have to feed a crowd, because a single of these odd colored squashes is enough to feed a family of 20. Today you can get smaller ones, and indeed, my local grocery store has Hubbard squashes that are about the size of an acorn squash, but there’s something wonderful about using a huge Hubbard squash. They look like they belong in the Jurassic period. LOL!

Ingredients:

• 2 lbs squash peeled and cubed
• 3 onions diced
• 3 cups chicken or vegetable broth
• 1/2 cup protein, boiled then minced
• 2 egg yolks, beaten
• 1 tsp salt
• 2 tbsp of sugar and 1 tbsp each of cinnamon & ginger, mixed together

Add the squash and onions to your broth, and bring it to a full, rolling boil. Reduce the heat and cook until the squash is tender. If you want a “creamy” soup, cook until it’s falling apart soft, and if you prefer a more clear broth with chunks, stop when just tender to the fork.

Your protein can be pork, chicken, or even vegetarian options such as seitan or walnuts. Boil meat until it’s very soft and falling apart (think pulled pork consistency) and then shred it with a fork or mince it up into very small pieces. Stir this into the soup along with the egg yolks and salt. Stir gently, then allow to cook for a few minutes.

When putting out the soup, set the sugar and spice mixture on the side in a small bowl, with spoons. Allow people to season to their own taste.

Serve this soup as a first course to a feast, or offer it with a salad and a more substantial meat.

I make considerably more of this recipe than is called for, because it freezes well and I like having convenient lunches I can just pop into the microwave later. A large batch takes the same amount of time and effort as a small batch, so why not “go big”?

Notes:

When I make this, I use 10 cups of broth and about 5 or 6 pounds of cut up squash. I use one very large Spanish onion and one regular sized cooking onion. I also like it to be meaty and hearty, as it’s usually all we’re going to eat, so I use about 3 or 4 cups of shredded chicken. I increase the egg yolks to 3, to thicken the soup, making it much more “creamy” or “silky” in texture. I used Himalayan Pink Salt, about 1.5 tsp, and a sprinkle of pepper on the top of each served bowl of soup.

This soup is incredibly thick and hearty, much moreso than you might think.  When my kids were younger, I used to make this a lot. They  suggested that the squash was kind of lost in the broth, and that it tasted rather like chicken noodle soup. This is a plus, in my opinion, as the soup was full of all sorts of good things for the kids that they wouldn’t normally eat. I’m all for hiding the good stuff in yummy dishes!

My newest cookbook is now available!

Clay pots are probably the oldest cookware known to humans. Since not long after we harnessed fire, we’ve been learning new and improved ways to make cooking vessels, and clay was an obvious early choice. When fired, it’s hard, heat resistant, and heat retaining. It holds liquids and solids equally well. And best, clay can be pressed into any shape. This small cookbook covers five recipes from each of five different time periods: Anglo-Saxon, Viking, Medieval, 18th Century American, and modern day. Each uses a clay cooking vessel, be it pot, pipkin, tagine, or pan. They are tasty additions to any modern menu, while still touching on the history of early human cooking. M. Allyson Szabo is the author of The Re-Enactor’s Cookbook, and Cook Small, Live Large! Her interest in all things historical and kitchen oriented has served her well, and she produces cookbooks that enlighten and educate with a conversational flair. Readers at all levels of cooking can enjoy her books!

So I don’t usually toot my own horn here on the blog, but I did want to post this up because it lends itself well to prepping and how to cook during an emergency. While the recipes in the book are a bit fancier than you’re likely to make during a major emergency, the methods you’ll learn will be useful.

If you can find clay, you can make pots. Once you have pots, you can cook. You can boil water to make sure it’s safe to drink. You can wash, shave, do dishes. The book doesn’t go into how one makes pots, but there are tons of amazing YouTube videos and articles on the subject that you can tap into.

There are lots of different types of shelter that we have access to. As with most things “prepper” related, everyone has a couple of opinions, and many of them clash. Basically, the idea is that you should have access to some kind of shelter in the first three hours of an emergency. It’s a good target to aim for, but it requires a lot of forethought and/or skill to pull it off.

The first kind of shelter that most of us have is our home. If you can stay at home, you probably should. Your home has all your food, your beds, your comfort items, and you know it. If you’re the type of person who’s been prepping a while, you probably have a plan in place for how to keep your home safe from raiders or marauders, and so that makes it one of the safest places you can be. A roof with four walls and a stout door really does trump most other types of shelter.

What if your home isn’t available, though? Whether it’s because you’ve been caught out when the emergency occurred, or a flash flood has washed away your home, or a tornado has ripped it into shreds, or any other reason, home is no longer safe. Your next easiest (and possibly safest, in many cases) form of shelter is your vehicle. You can cover the windows for privacy, you can lock the doors, and it’s mobile so you can move it to a safer place if things get hairy wherever you are. Your car also has the ability to hump more goods than you can by yourself. Again, it’s a fairly warm place (at least in comparison with outside), the doors lock, and it’s yours. You can formulate a plan around your vehicle long before you ever need it.

With your home and vehicle out of the picture, what else can you do? Well, that depends a lot on your strength, ingenuity, where you are, the time of year/weather/temperature, and lots of other things.

Read More

I started my computer career with the command line, or as it is known today, the CLI.

Almost everything I do is done via CLI.

I’ve had clients that had hosts in China, Ukraine, and London. They all look the same to me because they are just another window next to the other windows on my desktop.

When programming, my hands seldom leave the keyboard. I don’t need to use the mouse to program. It is mostly done with control key sequences.

When I need to configure something, I use a text editor and edit the configuration file. If the configuration file is JSON, I use JSON CLI tools to manipulate the file. Everything is done via the command line.

Even this post is done from “the command line.” I am typing raw HTML into a simple text editor. So an aside is written as:

<div class="aside">This is the aside</div>


Which renders as

The editor also has a visual editor. What we call a “What You See Is What You Get” or WYSIWYG.

In the WYSIWYG, you type, and it is formatted close to what it will look like when presented in a web browser.

You have likely used a word processor like Microsoft Word, Apple’s old Mac Write, or the modern LibreOffice. If you’ve used Google Docs, you have used the online version of LibreOffice.

The idea is that you can look at what you type in these WYSIWYG editors and that is what it will look like when printed.

We have another term for Graphical User Interfaces, WYSIAYG, or What You See Is All You Get.

What do I mean by that? Well, if you have a GUI that performs configuration options, then only the options that are implemented in the GUI are available to you through the GUI.

The new level 3 managed switch has a web GUI. It is rather nice. You can see the status of each port. There are pleasant drop-downs to give you choices.

One of the issues I needed to deal with was to get DHCP running on it, rather than the old DHCP server.

After fumble fingering my way through the interface, I had a working configuration.

The other day, I wanted to set up network booting. I am installing Linux on so many machines, both virtual and bare-metal, that I wanted a simple method to use. Network booting seemed like the answer.

This requires setting the “next-server” and “bootfile” options in the DHCP configuration file.

There is NO place in the web GUI to do so. It is available through the CLI. Undocumented, of course.

WYSIAYG. I muddled through, I got it working. I can now do a network install anytime I want. And I can provide multiple options.

Which leads me to the root cause of this rant.

They are now building CLI tools that require CLI tools to configure them. And the CLI tools that do the configuration are not well documented because you should use the CLI management tool!

I needed to install incus on a system to configure a working OVN network! I am so frustrated right now that I could scream.

I preface this by saying that I’m not Jewish. However, having dated several Jews over the years, I can tell you that the food is INCREDIBLE. Since Yom Kippur began on Friday at sundown, I thought this delicious parve (meaning it can be eaten any time by Jews, provided they aren’t fasting) dish would be just perfect to share today. Remember that Yom Kippur is not a “happy” holiday; it’s a religious time of reflection and introspection. Mostly, people who celebrate Yom Kippur just want to be left alone for their 25 hours of attonement. After the fast, families tend to enjoy a small feast together.

Ingredients:

• 1 package puff pastry dough
• 1 large onion, diced and sauteed
• 2 (7- and- 3/4-ounce) cans salmon
• 2 medium potatoes, cooked, cooled, and drained
• 2 eggs, whisked (for egg wash)
• sesame seeds, for topping
• 1/4 teaspoon pepper
• 1 teaspoon salt

Defrost the dough per package directions. Pay attention to this, as you don’t want it to be too moist and melty, but it can’t be frozen either. Roll out the dough to 1/4″ thick, and then cut into 3-1/2″ squares.

Combine all the ingredients for the filling, and mash it very fine. You can use a food processor if you want, but it’s better to do it by hand. You want it to be fairly smooth, almost like a pate, so that it fills the pastry well. Put a heaping tablespoon of filling into the center of each square. Fold the dough over, and press to seal. You can do this “pirogi style” (fold it in half, seal with fork tines for a pretty edge) or Jewish style (fold the corners up and seal it, making it back into a square, then flip it over and put the seam side down. Put your knishes onto a parchment paper lined cookie sheet.

Brush the knishes with the egg wash, then sprinkle with sesame seeds. Bake at 350*F for 40 minutes, or until they are golden brown.

Notes:

If you celebrate Yom Kippur, these are designed to be eaten AFTER you’re done fasting. You can make them ahead of time, and then just reheat them easily enough. They can tide you over until the rest of your meal is ready! If you aren’t Jewish, these can be enjoyed anytime. As a bonus, they’re also safe to feed to observant Jewish friends.

I’ve talked about having access to fire in the past, but I haven’t really gotten in depth about how to start one. I figured that was a decent place to go, this Friday, what with all the storm disaster stuff going on. This is a really complex topic, and I’m going to include some videos so that you can see as well as read about the subject. If you have more information, please feel free to add it in the comments below!

So fire is started when three elements come together: fuel, oxygen, and heat. If you have all three, you have what you need to start or continue a fire. Take one away, and if a fire has already started, it will die out. That’s the science end of it. (CalSafe)

We have lots of ways of starting fire, whether it’s for pleasure, cooking, heat, or protection. The easy methods are things like lighters, everstrike matches (metal “matchsticks” that sit in fuel), blow torches, and that sort of thing. Generally speaking, these methods “just work” when you use them. They provide enough of the three elements that you can get a fire going. But what if you don’t have one of these easy methods? What if your fuel has run out, or your flint is used up?

There are ways to start fires that are “in the middle” when it comes to difficulty level. Regular matches are in this category (because they can easily get damp, don’t stay lit very long, and have a tendency to dissolve over time). Strike-anywhere matches are less prone to problems, and usually have much longer stems, which makes them easier to hold. Unfortunately, they aren’t literally strike anywhere, but require a rough surface of some kind to get them lit. I would put the most modern style of ferro rods into this category as well.

Then there are the methods that require a certain level of skill: flint and steel, bow drills, other types of friction styles, and the more esoteric types like using magnifying glasses. To use any of these requires practice, repetition, and education. It is important to learn at least a couple of these more difficult methods in advance of emergencies, because it’s too easy to lose all the easier methods of making fire. Read More

That nip in the evening air, the scent of rain in the distance, and the sound of falling leaves in the forest behind the house all call to me that it’s soup weather! My favorite part about soup is that you can make pretty much anything into soup if you really want to. My turkey noodle soup is a great example. On Sunday evening, we treated ourselves to a lovely turkey breast for our supper. It was much too large for us to finish, even though it wasn’t a whole turkey, and so I diligently denuded the carcass of meat and put it into the fridge for use later in the week. Then it occurred to me that I ought to make broth!

This is a great recipe to use if you’ve got leftover carcass of any kind. If you like to pick up rotisserie chicken, it’s perfect for making this kind of soup.

Ingredients for the stock/broth:

• 1 stripped carcass (turkey or chicken), with whatever skin and gristle there is
• any drippings from the cooking process
• the skin of 1 or more onion
• several crushed garlic cloves, skin on
• the peelings of 1 or more carrots
• the leafy tops of 1 or more stalks of celery
• salt and pepper to taste
• fresh herbs to taste (optional)

Read More

I’m exhausted. I’ve been pulling fiber for the last two days. All part of an infrastructure upgrade.

Normally, pulling cable in a modern datacenter is pretty easy. This is not a modern datacenter.

The original cable runs were CAT6 with RJ45 connectors. When the cables were installed, the installation had to be nondestructive. No holes in walls, no holes in floors. Hide the cables as best you can.

One of the cables we removed was to a defunct workstation. It had been run across the floor and then covered with a protective layer to keep it from getting cut or snagged. The outer insulation had been ripped away. There was bare copper showing. Fortunately, that particular workstation hasn’t been in place for a few years.

The backbone switch was mounted in the basement. Not a real issue. The people who pulled some of the last cable didn’t bother to put in any cable hangers. So it had loops just dangling.

There were drops that could not be identified. Those are now disconnected, but nobody complained, so nothing was taken offline.

I’ve found a new favorite cable organizer.

Cable Management Wire Organizer

These are reusable. They open fully and will hold many cat6 and even more fiber. They have the 3M foam double-sided tape on them. This works great against smooth, clean surfaces.

The place where they shine is that they also have a hole designed for a #6 screw. In places where there were no smooth surfaces, much less clean surfaces. The sticky held them in place long enough to drive a screw.

There are no more dangling cables.

My only hope is that there are no more configuration issues with the new switch. *caugh*DHCP*caugh*