Skills

Part of the task of making a High Availability system is to make sure there is no single point of failure.

To this end, everything is supposed to be redundant.

So let’s take the office infrastructure as a starting point. We need to have multiple compute nodes and multiple data storage systems.

Every compute node needs access to the same data storage as all the other compute nodes.

We start with a small Ceph storage cluster. There are currently a total of 5 nodes in three different rooms on three different switches. Unfortunately, they are not split out evenly. We should have 9 nodes, 3 in each room.

Each of the nodes currently breaks out as 15 TB, 8 TB, 24 TB, 11 TB, and 11 TB. There are two more nodes ready to go into production, each with 11 TB of storage.

It is currently possible to power off any of the storage nodes without effecting the storage cluster. Having more nodes would make the system more redundant.

Unfortunately, today, an entire room went down. What was the failure mode?

DHCP didn’t work. All the nodes in room-3 were moved to a new 10Gbit switch. Actual 4×2.5 2×10. The four 2.5Gbit were used to connect three nodes and one access point. One of the 10Gbit SFP+ ports was used as an uplink to the main switch.

When the DHCP leases expired, all four machines lost their IP addresses. This did not cause me to loss a network connection to them because they had static addresses on a VLAN.

What did happen is they lost the ability to talk to the LDAP server on the primary network. Because they had lost that primary network connection, no LDAP, no ability to log in.

The first order of repair was to reboot the primary router. This router serves as our DHCP server. This did not fix the issue.

Next I power cycled the three nodes. This did not fix the issue.

Next I replaced the switch with the old 1Gbit switch (4x1Gbit, 4x1Gbit with PoE). This brought everything back to life.

My current best guess is that the cat6 cable from room 3 to the main switch is questionable. The strain relief is absent and it feels floppy.

More equipment shows up soon. I’ll be pulling my first fiber in 25 years. The new switch will replace the current main switch. This is temporary.

There will be three small switches for each room. Then there will be a larger switch to replace the current main switch. The main switch will be linked with 10Gbit fiber to the 3 rooms in server rooms. The other long cables will continue to use copper.

Still, a lesson in testing.

The final configuration will be a 10Gbit backbone with OM4 fiber, the nodes will be upgraded to have 10Gbit NICs which will attach to the room switches via DAC cables. There will then be a 2.5Gbit copper network. The copper network will the default network used by devices.

The 10Gbit network will be for Ceph and Swarm traffic.

I’m looking foward to having this all done.

I’m all about prepping canned and dehydrated food in advance of any emergency. I’m more interested in learning to garden than in having 20 years of supplies on hand. My goal is to have enough food (“food” being defined as the proper ratio of fats, carbohydrates, and proteins to keep a person not just alive, but thriving) in my pantry that I can last for 18 months. That’s the longest time I would be without a garden harvest. These are good plans, but I also like to know what I can get from the world around me.

I live in New Hampshire, and there are lots of things to eat that can be foraged easily. Starting with the easiest, there are blackberries, raspberries, and blueberries pretty much all over the state. They’re largely free for the taking, and there aren’t any “look alike” poisonous versions. Eastern Hemlock is a type of fir tree that is indigenous to the area. You can tell it by its flat spines/leaves (see the picture). It contains a lot of Vit C, and makes a very nice tea. In the spring, dandelion leaves are a great source of fresh greens. While a little bitter, they’re full of vitamins and minerals, and can be added to salads and soups. Don’t forget fiddleheads, though they’re only available in the spring for a very short amount of time. Cattail (you know, the “corndog” like things growing in ponds and slow moving rivers) is also edible. Different parts of the plant are good at different times of the year.

There are many mushrooms that are edible, but all mushrooms come with a caveat. Mushrooms, all of them, are edible… but some are only edible once. Meaning, there are poisonous mushrooms, and ones which will mess with your gut in bad ways even if they don’t kill you. As an example, I found a stand of amazing looking fungi at the base of a tree in my back 40. I asked a friend of mine, who is an expert mushroom hunter, if they were edible. They looked very much like an edible mushroom I know about. She laughed when she saw the image, because the shroom in the picture is more of a “draino” style medicine. Basically, if your stomach needs to be cleaned out entirely, it’s a really good mushroom to eat. It doesn’t harm you, but it goes right through your system, taking everything with it. I’m glad I didn’t make those into mushroom soup…

Read More

I made this one this past week, and was thoroughly impressed by it. The flavors were good, though it needed a bit of tweaking to make it perfect. It was a VERY good “dump meal,” meaning all I had to do was stuff things in the crock pot and set it to high. I’ve been sick as a dog this week, and needed stuff that was simple to make. This fit the bill, and was healthy and tasty as well!

Ingredients

• 1-1/2 cups barley
• 1 lb mushrooms, sliced
• 2 lbs boneless, skinless chicken (thighs, breasts, or a combo)
• 4 cups chicken broth
• 1/4 cup lemon juice
• 1/4 cup dry white wine
• 6 garlic cloves, smashed and chopped
• 1 tsp dried thyme
• 2 tsp salt
• 1 tsp onion powder
• ½ tsp red pepper flakes
• black pepper, to taste
• 10 oz frozen sweet peas
• 1/4 cup creme fraîche, sour cream, or plain yogurt (dairy free works well)
• 2 tbsp Dijon mustard, plus more to taste
• 1/2 cup fresh tarragon or dill, minced
• Parmesan cheese

In your crock pot, combine all the ingredients from barley to the black pepper. Stir well to combine, and then cook on high for about 3 hours. Stir again, making sure to immerse the chicken in the liquid. It’s going to seem a bit soupy, but trust in the recipe. Cook for another hour on high, or switch to low to cook for up to another 3 hours.

Remove the chicken to a plate and let sit for a few minutes while you do other things. Stir in the frozen peas and let them heat for about 2 minutes.

While the peas are warming, whisk together the creme fraîche and mustard. Taste, and add more mustard to suit your palate. Fold this mixture into the barley and mushrooms in the crock pot. Now, shred or pull apart your chicken. It should fall apart easily. Return the chicken to the pot, then taste to see if you need any more salt or pepper. Stir in the chopped fresh herbs, and serve immediately.

Add the Parmesan at the table, for those who wish it.

Notes:

I found the tarragon to be a bit strong in flavor, even though I only used about a tablespoon of it. The slight liquorish flavor just overpowered the rest of the dish. I would omit this next time, and use either all dill, or a mix of dill and Italian Seasoning instead. Another suggestion would be Herbs de Provence.

Even though you’re only adding about a half cup total of “creamy” stuff to this dish, it comes out incredibly creamy tasting. I used a cashew based yogurt, because I’m not able to eat dairy, and it was delicious. I used a smooth traditional Dijon, and I think I might try a more stone ground style next time. This was great as a stand-alone, and could easily be put together in a morning before running out the door. I think it would also be nice paired with a light salad.

Following the basic rules of three, we can live without water for three days. That’s not entirely true, but it’s close enough. Water is largely what we’re made of, being between 50% and 60% of our makeup. Water keeps our brains working, flushes toxins out of our bodies and protects our muscles, helps lubricate our joints, and keeps the blood flowing through our veins. We can become dehydrated incredibly quickly, depending on our level of activity, the ambient temperature, and the humidity, so it’s important to keep drinking. A general rule of thumb is, if you’re feeling thirsty, you’re already well on the way to being badly dehydrated.

It is possible to become dangerously dehydrated within 24 hours. Dehydration is when you take in less water than your body is using up in sweat, tears, urination, etc. and it’s very serious. While you can be on day 2 or 3 of not having water and still survive it, it will be a painful survival. After that first day, you are not going to be thinking well, because your brain will not be functioning as easily. You will be sluggish. Your mouth will be dry, and possibly your nose, and skin. You become exponentially more likely to have seizures or experience hypovolemic shock (when your blood volume drops below a certain level). This all has to be kept in mind when you’re dealing with a SHTF scenario. Drinking is going to be incredibly important.

So how do you keep water on hand? That’s something you have to decide for yourself. I live in a water rich area of the world, and so I don’t store a lot of water. I keep enough on hand for 72 hours of drinking and washing (of myself and my dishes), which is about 2 to 3 gallons per person per day. Instead of keeping lots of water on hand, I keep water purification items on hand. I have the means and knowledge to get water, but I’ll need to purify it. To purify water, you have to either boil it, disinfect it, filter it, or treat it with UV light. For safety’s sake, I recommend doing at least two of the four methods, though depending on your emergency, that may change.

Read More

Chris wanted me to share my sauce from last night, so I said heck with it. I liked it enough that I’m going to post it up here now.

To start off, this was my first attempt at making a Bolognese from scratch. I’ve always used jarred sauces and then “spiffed them up” in the past. I was in the mood to try something more challenging, but not too much because I’ve been so sick this week. The consistency was good, the amount of meat to sauce was good, the mouth feel was excellent, and the taste was incredible!

Ingredients:

• 1 lb ground pork (not flavored, just plain)
• 1 celery stick, finely diced
• 1 medium carrot, finely diced
• 1 onion small, finely diced
• 2-3 garlic cloves, minced
• 1 tbsp olive oil
• 2 14.5 oz cans of crushed tomatoes or stewed tomatoes 
• 1 bay leaf
• 1 tbsp tomato paste
• 1 tbsp Worcestershire sauce
• 1 tsp dried oregano
• 1/2 cup whole milk (or oat milk for dairy free)
• salt and pepper to taste
• grated Parmesan cheese for serving

Heat your olive oil in the bottom of a Dutch oven or saucepan, and add the ground pork. Stir regularly until it’s almost cooked through, about ten minutes. While the pork is browning, mince up your vegetables. Add these to the meat, once it’s no longer pink. Continue to cook on medium heat until the onions begin to soften and clarify.

Add in your tomato paste, Worcestershire sauce, and garlic. Cook for another 3 to 4 minutes, continually stirring. Reduce the heat to the lowest setting and add in the tomatoes, bay leaf, and dried oregano. Stir it together, and partially cover the pot. Simmer this on your lowest heat setting for 45 to 60 minutes. Stir the sauce every 15 minutes, to make sure it doesn’t stick to the bottom. Technically, this can continue to simmer for as long as you like. You want it to be fairly thick, so after an hour you may want to take the lid off, and stay there, stirring, while it thickens. If you’re in a rush, you can remove any excess liquid with a spoon or small ladle.

Turn off the heat, then add in the milk. Stir it together, taste, and then add salt and pepper to taste. If the sauce has a metallic or acid tang, add a half teaspoon of brown sugar to it. While you can skip adding the milk entirely, it really does add a depth to the sauce that makes it velvety. It also helps mellow out the tomato acidity.

Notes:

The original recipe calls for crushed tomatoes, but I didn’t have those. I could have stuck my stewed tomatoes into the blender and whizzed them up, but I chose to use the tomatoes chunky. I’m glad I did. Many of them broke up during the cooking of the sauce, but the pieces of tomato really added a lovely mouth-feel, and a certain “umami” to the final sauce.

This goes perfect over any kind of pasta, but it’s best over one that has a rough texture and lots of gripping surface. Think bowties or rotini, or even radiatori.

Serve this one up with a slice of hearty french bread, or some garlic knots. Don’t bother with salad; it’s much too meaty for that.

There are three ways of making pasta shapes. You can extrude the dough through shaped holes, cutting the forms off as needed. You can roll the dough thin, then hand cut the dough to be shaped. The same can be done using a “pasta machine” which rules the dough to a known thickness. Or you can use cutters to cut sheets of pasta.

I’ve been making pasta dough for a couple of years. I stepped up my game recently by deciding to roll my dough instead of squeezing it in the pasta machine.

Since I was going to roll out the dough, I decided to get rid of my mixing bowl and do it on my board with the well method.

This is a step forward. One of the lessons I’m still learning is how small a batch can be made this way. If the batch is not large enough, there isn’t enough floor on the board to create a well with deep enough walls.

I was lucky, there were only two eggs in play, so the eggs didn’t go everywhere, just a slight panic.

Which brings us back to dough. Normal pasta dough is an egg and floor mixture. My recipe is 3/4 of flour to 1 large egg. Occasionally, it is two eggs plus a yolk. It really depends on how the dough feels.

Today, the dough was too wet. I should have added more flour, but I didn’t.

The eggs are mixed into the flour until it comes together. At that point, I switch to kneading. I knead for 10 minutes, by the clock. The sad thing is that I always check the time remaining at 2:30. I don’t know how I get there, but I do.

This egg pasta works well for rolling out or squeezing. This was turned into rigatoni.

I didn’t know I could make this by hand, I thought I would need to use an extruder for rigatoni. I am happy with my results.

It does not look like machine made pasta. The dough was a little wet. I also had some issues cutting it. Next time I will use the bicycle cutter to get nice squares.

Ally took the other half of the egg pasta and made beautiful bow tie pasta.

The other type of pasta dough is made with water and semolina flour. Bluntly? I just guessed at the amount of warm water to add. It came out nearly perfect.

This pasta dough doesn’t roll out as flat as the egg pasta, but it is used for other pasta shapes. We started with Orecchiette. Those were easy to make, but I didn’t like them. Ally loved making them, so I let her do it.

This is made by cutting a 1/2 diameter rope of dough into 1/2 long chunks. Then you roll the under your thumb.

Malloreddus is made the same way, but the pasta is pressed on a grooved board to make a wonderful pattern. This was my favorite shape to make. They go fast, they look nice, and they have a good bite to them. Plus they hold the sauce.

The Busiate was another fun shape. It uses a 1/4 in rope of dough cut into 3 inch long sections. These are twisted around a skewer and the rolled flat.

I think they look delightful, I want to practice them some more.

I’m hoping that Ally will do an article about the sauce she made for our homemade pasta. I will say it is the best I’ve had in years.

Stew is the bedrock of my winter meal plan. Part of my family dislikes soup, but is okay with something thicker. Stew is the way to go! Stew is basically a protein (chicken, beef, lamb, etc), a fat (bacon grease, olive oil, veg oil, etc), vegetables (onions, carrots, celery, etc), and a starch (rice, beans, potatoes, etc). In whatever combination you like, mix together, and you have stew. Of course there’s a bit more to it, but that’s the basic “stew recipe.”

Because stew was usually made with leftovers or scraps, the little bits that weren’t enough for another full meal, we don’t really have written stew recipes.  A good stew is different every time you  make it, because the ingredients will never be exactly the same. That is the right way to do it! So let’s make a nice, hearty chicken stew.

Ingredients

• a cut up chicken, or several thighs, bone in and skin on
• 1 or 2 onions or leeks, rough chopped
• 3 large carrots, coined
• 2 ribs of celery, diced
• 4 cloves garlic, rough chopped
• white wine (optional)
• 1 purple top turnip, diced
• 2 parsnips, coined
• 2 to 3 cups shredded cabbage
• 1 cup barley, well rinsed
• spices (salt, pepper, oregano, parsley, dill, sage, rosemary, etc.)
• oil for searing
• water to fill the pot

In a large soup or stock pot, add a drizzle of oil (olive, safflower, or vegetable) and bring up to a medium heat. Add the chicken and brown it all over, then remove it from the pot (set it aside on a plate). Add the carrots, onions, and celery, and cook until the onions soften and begin to brown very slightly. While they’re cooking, you may need to add more oil. This mixture of carrots, onions, and celery is called a mire poix, and it’s the basis of most good quality soups and stews. You want to scrape any of the browned chicken off the bottom of the pan (that’s the fond and it’s part of what makes for a rich flavor later). Add in the garlic and saute until they are fragrant, usually less than a minute.

Drizzle in some of the wine (or chicken broth if you don’t do wine), and deglaze the pan entirely. Once that’s done, add the rest of the vegetables and allow them to cook for 2 to 3 minutes. Add all the remaining ingredients, and enough water to cover everything, plus 3 cups (this is to allow for the barley to absorb some of the water – you may want to try a bit more or less, to make it the thickness you prefer). Add in some salt, pepper, and other herbs to taste, and bring to a boil, then allow to simmer for a minimum of an hour.

Check for flavor, then add more spices as necessary. The above are only suggestions, and you’re welcome to try whichever spices you like! Simmer for another hour or more, until the chicken is falling off the bone and shreds easily. You may wish to pick out the skin and bones at this point, or serve it as is. Serve this up with fresh baked bread for a hearty and healthy meal.

Notes:

You can make this recipe with diced chicken breast if you prefer. I like the added flavor and the moistness of the meat when I use bone-in, but not everyone likes bones. I almost always make this recipe with either a full cut up chicken, or with chicken legs with thighs attached. It comes out meaty and delicious!

Recipe by M. Allyson Szabo, author of The Re-Enactor’s Cookbook (available on Amazon).

Allyson had picked up a wooden box a few years ago to take to events. It was a plain wooden box with just a bit of decoration and a porcelain knob attached to the sliding top.

The knob and screw holding it are not period. The rest was pretty period. Or more precisely, it was period-20.

It looked period at 20 feet.

Over the years, it has taken a beating. This knocked the back off, pulling out the brads.

She asked me if it could be repaired. TiteBond III to the rescue. I pried the back out enough to get glue where it was needed. Clamped the entire thing closed and called it a day.

Until I decided that it could be better.

Today, we use petrochemicals to create high-quality, strong, enduring coatings for our cabinetry and furniture.

Modern finishes are long chain polymers that have all the right characteristics for a long-lasting, shiny surface.

I’ve used polyurethane finishes. They can be polished to a near mirror like finish.

But that is the modern world? What were they doing 100 years ago? 200? 300? They were using some of the same finishing methods we use today.

Read More

Are you ready for what happens after the apocalypse? I’m not talking about the zombie apocalypse here, nor am I referring to the next pandemic. The apocalypse I mean is the one that’s currently predicted: Harris winning the election. I know some of you are laughing, but please don’t. I am quite serious about this. What are your plans for if she wins?

There are so many moving parts to planning for a Harris win. We have to consider the possibility (likelihood?) of social unrest. There may be a repeat of January 6th, this time with weapons, even if Harris is a clear winner (which I think is unlikely). More importantly, we need to plan for the economic downfall of the country.

It’s time to stock up on some basic things. From now until the election, prices have eased slightly. Harris is trying to buy votes. I know money is tight, but it’s really time to go and buy up all the canned goods you use in a year. Pick up extras of all your dry goods. If you can, order a half cow and stash it in your freezer. I have a funny feeling that if she wins, the country is going to get very expensive, very fast.

I always say, prep what you eat, and eat what you prep. This isn’t for a long-term emergency, though. This is food that you expect to eat over the next year. So look at what you use on a regular basis. For me, in the canned goods department I use a ton of canned tomatoes (both regular and the type with the chilies in it). I use canned beans (black, red, chili, garbanzo, etc.) as well as dry ones. I use rice, specifically Jasmine or Basmati. My family likes sausage, and I can usually find them on sale, so I will buy extras and toss them into the freezer. Last week, I found turkey breasts on sale for 99 cents a pound, and I bought the limit.

Read More

For me, the flavors in a good soup are the taste of autumn. I adore soup weather, and with the nights getting colder, I’m looking up my soup recipes. Last week, I made a small pot of beef barley soup, and I impressed even myself. It’s easy to do, and you can even pop it in the crock pot if you want.

Ingredients:

• 2 lbs beef shin bone (with bone and marrow)
• 1/4 cup red wine (for deglazing the pan)
• olive oil (for cooking)
• 1 onion, diced
• 2 ribs celery, diced
• 1 carrot, diced
• 6 cups beef broth
• 1-1/2 cups pearl barley, rinsed
• 1 tbsp tomato paste (optional)
• bay leaf
• spices to taste (salt, pepper, thyme, oregano, sage)

In a soup pot, add a bit of the olive oil and bring to a medium heat. When the oil is shimmering, add in the beef shin bones. If you have more than one, you may need to do them one at a time. Don’t crowd the bottom of the pot. You want to sear the meat for about 2 minutes per side, then set the meat aside.

Add a bit more olive oil if necessary, then toss in the onion, celery, and carrot (this is called a mirepoix, and it’s the base of many soups and stews), and saute until the vegetables are softened but not yet brown. Use the red wine, a little at a time, to deglaze the pan. Add in a drizzle of wine, and then use a wooden spoon or spatula (it should be wooden, NOT plastic or rubber) to scrape up all the stuff off the bottom of the pot. That “stuff” is called fond, and it’s delicious.

Once the pan is deglazed, put the meat back in (still on the bone), along with the broth, bay leaf, and tomato paste. Stir to combine the ingredients. Bring it to a boil, then lower the heat and allow the soup to simmer for about an hour. Taste the broth, and then adjust for flavor by adding the spices of your choice.

Rinse your barley well, and pick out any stones you may find. Add the barley to your soup, and raise the temperature enough to keep the liquid at a steady strong simmer. Check on your soup every 15 minutes or so, to ensure nothing is burning. Add more beef broth if necessary. Continue to simmer until the barley is completely cooked and is chewy.

Lower the heat to its lowest setting, and remove the meat from the soup. Carefully trim the meat from the bone and put it back into the soup. Get rid of any gristle or fat, but make sure that the marrow from the shin bone goes back into the soup. The marrow makes your broth rich and tasty. Taste and adjust the spices as necessary, and then serve.

I like to serve this soup with fresh baked bread, still warm from the oven. If you’re not a baker, pick up a nice ciabatta at the grocery store, drizzle olive oil over it and toss on some salt and pepper, then heat in the oven at 350F for five to ten minutes.

This soup freezes very well. If you have a pressure canner, you may can it, as well. I find I never have enough left over to can up, though! A note on the tomato paste: I find it adds a nice zing to the broth, but it isn’t necessary. Some people like to put crushed tomato or stewed tomato into their beef barley soup, but I’m a purist.