Skills

I’ve talked about my Ceph cluster more than a bit. I’m sure you are bored with hearing about it.

Ceph uses two technologies to provide resilient storage. The first is by duplicating blocks, and the second is by Erasure coding.

In many modern systems, the hard drive controller allows for RAID configurations. The most commonly used RAID is RAID-0, or mirroring. Every block written is written to two different drives. If one drive fails, or if one sector fails, the data can be recovered from the other drive. This means that to store 1 GB of data, 2 GB of storage is required. In addition, the drives need to be matched in size.

Ceph wants at least 2 copies of each block. This means that to store 1 GB of data, 3 GB of storage is required.

Since duplicated data is not very efficient, different systems are used to provide the resilience required.

For RAID-5, a parity drive is added. When you have 3 or more drives, normally an odd number, one drive acts as a parity drive.

Parity is a simple method of determining if something was modified in a small data chunk. If you have a string of binary digits, 0xp110 1100 (a lowercase l in ASCII), the ‘p’ bit is the parity. We count the number of one bits in the byte and then set the p bit to make the count odd or even, depending on the agreement. If we say we are using odd parity, the value would be 0x1110 1100. There are 5 ones, which is odd.

If we were to receive 0x1111 1100, the parity would be even, telling us that what was transmitted is not what we received. A parity bit is described as single-bit detection, no correction.

Parity can get more complex, up to and including Hamming codes. A Hamming code uses multiple parity bits to create multi bit detection and one or more bit correction.

NASA uses, or used, Hamming codes for communications with distant probes. Because of limited memory on those probes, once data was transmitted, it wasn’t available to be retransmitted. NASA had to get the data right as it was received. By using Hamming codes, NASA was able to correct corrupted transmissions.

RAID-5 uses simple parity with knowledge of which device failed. Thus a RAID-5 device can handle a single drive failure.

So this interesting thing happened: the size of the drives got larger, and the size of the RAID devices got larger. The smart people claimed that with the number of drives in a RAID device, if a device failed, by the time the replacement device was in place, another drive would have failed.

They were wrong, but it is still a concern.

Ceph uses erasure coding the same way RAID uses parity drives, but erasure coding is more robust and resilient.

My Ceph cluster is set up with data pools that are simple replication pools (n=3) and erasure coded pools (k=2, m=2). Using the EC pools reduces the cost from 3x to 2x. I use EC pools for storing large amounts of data that does not change and which is not referenced often, such as tape backups.

The replication pools are used for things that are referenced frequently, where access times make a difference.

With the current system, I can handle losing a drive, a host, or a data closet without losing any data.

Which is good. I did lose a drive, I’ve been waiting to replace the dead drive until I had built out a new system. The new node was in the process of being built out when the old drive failed.

Unfortunately, I have another drive that is dying. Two dead drives is more than I want to have in the system. So I’ll be replacing the orginal dead drive today.

The other drive will get replaced next week.

Cold, wet weather practically demands chili. I love all kinds of chili. This is one I make around this time of year, because I can use fresh pumpkin rather than the canned stuff. It makes a huge difference!

Ingredients:

• 2 cans navy beans* OR 1 bag (soaked overnight)
• vegetable or chicken broth
• butter or margarine
• 1 onion, chopped
• 1 can fire roasted stewed tomatoes
• 1 small can tomato sauce
• 1 tiny can chopped green chillies
• 1 “pie” pumpkin (small), roasted and the meat scraped out
• spices: coriander, cumin, garlic flakes, rosemary, chili powder, red pepper flakes, oregano, salt, pepper

If you want to make this vegan and gluten free, dump all of the above into a crockpot, put it on high, and cook for about 6 hours. Check the beans at the 4 hour mark and every hour thereafter. Cook until the beans are soft but not too mushy. Add in extra liquid if necessary. You can add heat to it with a hot sauce, or simply sprinkle with more red pepper flakes at the table.

If you want this to be meaty, I actually recommend ground turkey or chicken rather than beef for this one. You don’t want to overpower the flavor of the pumpkin, which ground beef will do. Instead, brown up your ground turkey or chicken, then add it in with the other ingredients.

When it comes to the *beans, you don’t have to use navy beans. Feel free to use whatever type or combination of types that you like. Because I want the pumpkin to shine through, I went with navy beans. You could use little white beans, kidney beans, black beans… it’s up to you. One bag of dried beans, soaked overnight, is equivalent to 2 of the big cans of beans, rinsed and dumped into the cooker.

Serve your chili with a side of cornbread, and top it with green onions, parsley, sour cream (or a dairy free alternative), and/or cheese. Enjoy!

We’ve talked elsewhere about making sure you don’t go shopping over the next couple of weeks, because some people have made what we consider to be credible threats. So here’s the thing… if you’re not already stocked up on canned goods at this point, you’re not ready.

Going out today to pick up a few things, that’s fine. I’m picking up “milk, bread, eggs” on Monday, in the morning when I can have Chris ride shotgun (literally). If something violent happens between now and then, I’ll bake bread, buy eggs from the lady up the street, and pull out the milk powder. We have toilet paper, cleaning supplies, food (fresh, frozen, canned, and dehydrated), oil and wood for heating, gas for the stove. I just picked up a zillion candles for free, and we have 2 big jugs of oil for our decorative (and very useful) oil lamps. The world can go to hell; we’ll be fine.

If your cupboard is bare, you’re too late. If you don’t have things squirreled away “just in case,”  you’re too late. I need to hammer that into everyone’s heads. The only excuse for not having a stash, at this point, is if you just started prepping in the past couple of months. Even then, if you’ve been reading my posts, ignorance is not an excuse. You KNOW.

Head on a swivel is not enough. If shit is going down, you’re not going shopping. You’re going to tell your spouse she can’t have the milk and eggs she wants. She’s going to make unhappy noises, and your life is going to be miserable. The fact that it’ll be short lived doesn’t mean a thing when you’re actually living it.

If you want food security, you have to make it. That means long term stuff like planting a garden, knowing how to hunt and trap, understanding how to dress and prepare meat for canning or the freezer. It means having canned goods on hand, stashed in a back corner or under a bed. It means having enough toilet paper or equivalent on hand so that you don’t have to be concerned. If you truly want to be prepared, and today is a good day to be prepared, then starting now is not good enough.

I am hoping that the people calling for riots and stealing are just making noise. I am preparing for them to be telling the absolute truth. If that means I don’t go shopping this week or next, we’ll be fine. Instead of beef tips and chicken breast, we’ll be eating tongue and heart and lots of soup over the next couple of weeks. But we have the food. Do you?

I found this one on TikTok, and decided to try it!

Ingredients:

• 4 chicken thighs, skin removed (bone in or out, doesn’t matter)
• 1.5 cups cherry tomatoes
• 3/4 cup green olives (no pimento)
• 1 shallot, sliced thinly
• 8 baby potatoes, quartered
• 8 to 10 whole cloves garlic
• 2 to 4 tbsp white wine (or white wine vinegar)
• 2 to 4 tbsp good quality olive oil
• 1 large sprig fresh rosemary, minced OR 2 tbsp dried rosemary
• 1 tbsp herbs de provence
• salt, to taste

Preheat your oven to 375°F.

In an oven-safe casserole dish with high sides, place your chicken thighs evenly in a single layer. Add the cherry tomatoes (whole) and olives (also whole) around the chicken. Add the shallot in a layer over everything, and the baby potatoes above the shallot. Tuck the garlic in between the potato pieces. Drizzle the wine and olive oil over everything, and then add your spices.

Bake your Poulet Provencal in the oven for 30 minutes, then check. Your chicken should register 165°F, and the potatoes should be ready to eat. If not, return the dish to the oven for 15 more minutes, checking every 5 minutes.

Serve with a side salad, or a nice crusty French bread. Your choice!

I run into things on TikTok and Facebook that are funny, sad, embarrassing, frustrating, and the whole gamut of other emotions. This one, though, hit home. It talks about something near and dear to my heart: going hungry. You can ask any of my friends, even at my poorest moments in the past 20 years, I have ALWAYS had a full fridge, freezer, and pantry. Always. If that meant I had to visit a food bank, so be it. I’ve been hungry before, and I never intend to be there again.

Take a pause and watch…

Read More

I’ve tried drawing network maps a half-dozen times. I’ve failed. It should be simple, and I’m sure there are tools that can do it. I just don’t know them, or worse, I don’t know how to use the tools I currently have.

In simple terms, I have overlay and underlay networks. Underlay networks are actual physical networks.

An overlay network is a network that runs on top of the underlay/physical network. For example, tagged VLANs, or in my case, OVN.

OVN creates virtual private cloud. A powerful concept when working with cloud computing. Each VPC is 100% independent of every other VPC.

As an example, I have a VPC for my Ceph data plane. It is on the 10.1.0.0/24 network. I can reuse 10.1.0.0/24 on any other VPC with zero issues.

The only time there is an issue is when I need routing.

If I have a VPC with node 172.31.1.99 and a gateway of 172.31.1.1, that gateway performs network address translation before the traffic is sent to the internet. If the node at 172.31.1.99 wants to talk to the DNS server at 8.8.8.8 traffic is routed to 172.31.1.1 and from there towards the internet. The internet responds, the traffic reaches 172.31.1.1 and is forwarded to 172.31.1.99.

All good.

If I have VPC2 with a node at 192.168.99.31 and its gateway at 192.168.99.1, I can route between the two VPCs using normal routing methods by connecting VPC and VPC2. We do this by creating a connection (logical switch) that acts as a logical cable. We then attach gateway 172.31.1.1 to that network at 192.168.255.1 and the gateway at 192.168.99.31 as 192.168.255.2.

With a quick routing table entry, traffic flows between the two.

But if VPC2 was also using 172.31.1.0/24 then there is no way to send traffic to VPC. Any traffic generated would be assumed to live in that VPC. No router would become involved. And NAT will not help.

Why use an overlay network? It allows for stable virtual network, even if the underlay network is modified. Consider a node at 10.1.0.77. It has a physical address of 192.168.22.77. But because it needs to be moved to a different subnet, its physical address changes to 192.168.23.77.

Every node that had 192.168.22.77 within its configurations now needs to be updated. If the underlay is updated, it does not affect the overlay.

Back to Simple.

There are three methods for traffic to enter a VPC. The first is for a virtual machine to bind to the VPC. The second is for a router to move traffic into the VPC, sometimes from the physical network. And the final method is for a host (bare metal) machine to have a logical interface bound to the VPC.

My Ceph nodes use the last method. Each ceph node is directly attached to the Ceph VPC.

It is the gateway that is interesting. A localnet logical port can bind to a port on a host, called a chassis. When this happens, the port is given an IP address on the physical network that it binds to.

When the routing is properly configured, traffic to the VPC is routed to the logical router. This requires advertising the logical router in the router tables.

I had multiple VMs running on different host servers. They all sent traffic to the router which was bound to my primary machine. My primary machine has physical and logical difference from the rest of the host nodes.

What this meant was that traffic to the VPC was flaky.

Today, I simplified everything. I turned down the BGP insertion code. I added a single static route where it belonged. I moved the chassis to one of the “normal” chassis.

And everything just worked.

It isn’t dynamic, but it is working.

I’m happier.

When the grid fails, your calm mind might be the last light on. I know, it sounds cheesy, but it’s true. The idea of mental health in prepping is often relegated to a short oneliner: “Three months without hope.” Bam, done. Except it’s not done, and in many ways, mental health is something we ought to be paying a ton more attention to, and it’s also something we almost entirely ignore. Current prepper magazines and e-zines talk a lot about ammo, food, shelter, the Rule of Threes, types of firearms and other weapons, even first aid… but they don’t talk about what to do if someone’s mind breaks.

There are a handful of stories out there about preppers who went a little “funny” while getting anxious over expired food or having the right stuff in their go-bags. It’s definitely easy to let your anxiety run you down, especially if you’re actually IN an emergency already. Isolation is a huge contributor to mental health issues, and even if you have your family with you, bunker life or hunkering down at home is going to cause some problems. We know more about this than we should, thanks to the pandemic. With that in mind, we should be applying mental health prep to our general SHTF preparations. The topic deserves more than one line. We need to give mental health its proper focus, as an integral and continuing part of our everyday prepping activities.

The idea of a “loneliness spike” is not a new one. We know it from studying trauma after real disasters (Harvard study). We are aware of how insecurity and worry causes brain fog (Medical News Today). There are studies done on survivors of hurricanes and the prevalence of PTSD among them (National Institute of Health), where mental health issues can affect up to 30% of those who make it through. Then there’s the idea of survivor’s guilt, which is absolutely real and devastating, and something we know both from horrors like the Holocaust, and soldiers coming back from war, and even families caught in natural disasters. All of these problems are things that could and likely will happen in a SHTF scenario, and so we must begin to come up with methods to help minimize the trauma response. After all, you want the person who’s got your back to be sane… and they want the same thing.

So how do we minimize the kind of mental health problems we’re likely to see in a post-apocalypse or post-big-emergency world? Some of the most banal and stupid practices done in therapists offices around the world are the most effective. Breathing exercises (in for 4, hold for 4, out for 4, hold for 4), journaling positive things, active exercises just for the sake of doing it, avoiding reading the news (which may or may not be a problem come the apocalypse), and checking in on friends are all methods you can use to keep your panic down and your mental load a bit more healthy. Yes, the world may be falling to ruin around you, but if that’s your focus, you will go to ruin with it. The most resilient minds are the ones that can ignore “all that stuff out there” and instead focus on what can be done here and now.

Community is a big part of what keeps us on an even keel. Yes, the smallest community is your family, but at some point you have to reach beyond that. Human beings are meant to live in villages, and we need the company of others to keep us straight and sane. Whether you’re organizing a town meeting, bartering your skills for therapy (like in The Last Of Us), or having a sit down “come to Jesus” moment like an AA meeting, doesn’t really matter. It’s the doing that’s important, rather than what it is you’re doing. When paranoia is the word of the day, it’s going to be tough to reach out to people and form lasting connections, but it’s something that has to be done. If you can’t come to terms with doing it for yourself, think of doing it for the sake of your kids or your neighbors’ kids.

The big question is how to make the best SHTF armor possible, not just for the body, but for the mind as well. Group talks and forums can help, if there’s still internet to be had. But there are some really simple things you can do every day to make sure your mental health doesn’t become a liability. The first and most important thing is to get enough sleep. Obviously if you’re fighting zombies, that might get pushed back in priority… but only for a while. If you want to live rather than just survive, you’re going to have to get to a point where you can sleep on the regular. You have to get food into your body, too. Good quality food (even if it’s canned) is going to be a huge help in making you feel safe. Going for a walk, whenever it’s safe and possible, also helps, as does gardening (National Institute of Health).

So there you have it. Stock up your mental health preps right beside your beans. Be ready to surround yourself with the start of a community, and be prepared to keep building. Focus on the end goal, which is to thrive. Survival should only be your goal in the beginning. Once you’ve achieved survival, it’s time to live and thrive, to grow and heal. That’s your focus; a healthy, happy world for the children we bring into it. Don’t ignore mental health preps. We’ve seen what happens when you do (look “to the Left” for visual cues).

I have a number of Jewish friends, and I hear about brisket all the time. There was the running joke about Howard’s mother’s brisket in The Big Bang Theory. Brisket is one of those cuts of meat that is just scrumptious, if cooked right. So how do you cook it? It depends a lot on the cut you’re using. I’m going for slow cooked brisket at this point in my cooking career, because I love the convenience of the slow cooker (though I did cook this one in the oven). It’s a bit pricey (our 3lb brisket cost about $21), but it’s totally worth it. Also, it makes your whole house smell like heaven.

Ingredients:

• 3 lbs untrimmed flat-cut beef brisket
• 1/2 tbsp salt, plus more to taste
• 1 tsp freshly ground black pepper, plus more to taste
• 3 tbsp oil (olive or vegetable)
• 2 medium yellow onions, sliced
• 4 celery stalks, cut into 2-inch pieces, leaves coarsely chopped
• 6 garlic cloves, smashed
• 2 cups red wine (such as Merlot or Pinot Noir)
• 28 oz canned crushed tomatoes
• 1/4 cup ketchup
• 2 tbsp brown sugar
• 1 tsp dried thyme OR several sprigs of fresh thyme
• 3 bay leaves
• 4 medium carrots, halved lengthwise and cut on a bias into 2-inch pieces

Preheat the oven to 325°F.

Season the brisket with most of the salt and pepper. Heat the oil in a large dutch oven over medium-high heat. Brown the brisket, turning occasionally, until browned on all sides, 10 to 12 minutes. Transfer to a plate.

Place the onions, celery, and garlic in the pot, tossing it in the residual fat. Season with the remaining salt and pepper. Cook, stirring occasionally, until the onions are soft and translucent, about 5 minutes.

Add the wine, tomatoes, ketchup, brown sugar, thyme, and bay leaves, and stir to combine. Nestle the brisket down into the liquid, fat side up. You may need to move some of the onion mix to get it low enough. Most of the brisket should be covered with liquid, but it’s okay if the fat is out. Cover the pot and cook in the oven until the meat is fork-tender, about 3 hours.

Uncover the pot and add the carrots around the brisket. Cook, uncovered, until the carrots are tender (not soft) and the top of brisket is browned and crisp, 35 to 45 minutes.

Remove the brisket from the pot and let it rest while you make the sauce. Use a spoon to skim fat from the surface of the braising liquid and discard. Heat over medium-high heat and cook until the liquid is thickened to a sauce-like consistency, 5 to 10 minutes. Remove the thyme sprigs and bay leaves. Taste and season with salt and pepper as needed. When the sauce is mostly ready, cut up the brisket by slicing it against the grain. Place the sliced brisket on a deep serving platter. Pour the sauce over the brisket and serve!

For sides, you need to keep in mind that Jewish folk don’t mix dairy and meat. If you don’t care, feel free to go with whatever sides you like, but if you want to enjoy the full Jewish brisket experience, try some baked sweet potato drizzled with olive oil, a cabbage and apple slaw for a tart finish, or some hot cornbread.

Sorry this one took a couple of weeks. It’s been busy here. Things are starting to settle down, though. Of course, that also means it’s almost National Novel Writing Month, and I’m going to be writing a flurry of words (50,000+ in 30 days), but I’m not going to think about that for a bit. LOL… We left off Heinlein’s list about here:

Take orders. You need to be able to take orders because no matter how “high up” you are on any particular totem pole, at some point you’re going to run into someone who’s higher than you. This is because we’re not ever going to be experts at everything. We each spend time with people who are better at something than we are, and when those people are in charge, you must be able to do what you’re told. But as any American soldier will tell you, it isn’t that simple (even though it sort of is). Per the Uniform Code of Military Justice, soldiers are only required to obey LAWFUL orders. Our soldiers are given more latitude as to what’s lawful and what’s not, while still being held to an extremely high standard (and getting higher, thanks Pete!). All of our soldiers are expected to be thinking people. Blind adherence is not useful. But the ability to continue to take orders, even when things are tough, even when you’re shitting your pants, even when you’re scared, is absolutely necessary. That’s also true of those of us NOT soldiers, though perhaps to a slightly lower level. As non-combatants, even if we end up as guerrilla fighters, we just need to be able to follow orders at a competent level. You need to recognize when someone knows more than you do, and be able to take a back-seat for a bit.

Give orders. There will be a moment when YOU are the expert, the leader, the person in charge. It might be on purpose, and it might be by accident, but regardless, you must be prepared to give orders. More than that, you may have to give orders that you know damn well will end up with someone hurt (physically or emotionally), or worse, dead. You need to be prepared for whatever outcome happens when you give those orders.  You have to be ready to give them decisively, with authority, and with strength of belief.

Cooperate. That’s a tough one, hm? Yes, you might have to cooperate with people who don’t share your world view. You might have to work with liberals and Democrats. But it CAN be done. And you must know both how to, and when to. Sometimes, it’s just going to be an easy choice. Groups often have better survivability options than singletons. It’s a skill we’re horribly underdeveloped in, in my very strong opinion. When was the last time you reached out to someone you disagree with, to cooperate? Maybe it’s time. Practice, because it’s important. And just in case someone wants to leap to conclusions, no, this doesn’t mean you have to “give in and open the government” or anything like that. I’m talking small scale here. Neighbors. Friends of friends. Local government maybe.

Read More

This was definitely the star of last week’s fancy meal with my fellow reenactors. It’s so delicious and boozy! I used “chia eggs” because I had several people who either were vegan or couldn’t do eggs, and so I skipped the creaming part mentioned below. Even with the fake eggs, it turned out moist and amazing.

Ingredients:

• 1-1/2 cups currants
• 1/3 cup chopped candied orange peel
• 1/3 cup chopped candied lemon peel
• 1/3 cup chopped candied citron
• 3/4 cup Madeira, divided
• 1/4 cup brandy
• 3 cups all-purpose flour, sifted
• 1/2 cup slivered almonds
• 1/2 teaspoons ground nutmeg
• 1/2 teaspoons ground mace
• 3/4 cup unsalted butter, softened (or margarine)
• 1-1/2 cups sugar
• 3 eggs
• powdered sugar for top of cake

Combine the currants, orange and lemon peels, and citron in a large bowl. Add 1/2 cup of the Madeira, and stir to combine. Cover and set aside for at least 3 hours, or as long as overnight. Stir the remainder of the Madeira together with the brandy, cover, and set aside.

When ready to bake the cake, preheat the oven to 325°F. Grease and flour a 10-inch tube pan or a 12-inch springform pan (line the springform with parchment paper before greasing and flouring).

Drain the fruits in a large strainer set over a bowl, stirring occasionally to extract as much of the Madeira as possible. Add the strained Madeira to the set-aside Madeira and brandy. Combine 1/4 cup of the flour with the fruit, and mix well. Add the almonds, and set aside. Sift the remaining flour with the nutmeg and mace.

In a bowl, cream the butter until it is light. Add the sugar, 1/2 cup at a time, beating for several minutes after adding. Continue to beat for several minutes, until the mixture is light and fluffy. Alternately add the spiced flour, 1/2 cup at a time, and the Madeira and brandy, beating until smooth.

In a separate bowl, beat the eggs to form stiff peaks. By hand, gently fold them into the batter, combining lightly until well blended. By hand, fold in the fruit in thirds, mixing until well combined.

Pour the batter into the prepared pan, smoothing the top with a spatula or the back of a spoon. Bake for about 1-1/2 hours, or until a wooden skewer inserted in the center comes out clean. Set the cake on a wire rack to cool in the pan for 20 minutes. Turn it out of the pan to cool completely. Dust with powdered sugar before serving.

Notes:
To use “chia eggs” simply replace each egg with 1 tbsp chia seeds and 3 tbsp of water whisked together. Let this “chia egg” sit for 5 minutes before using it in any baking recipe where you don’t need the eggs to be fluffy. Because this was a heavy cake, the lightness wasn’t really necessary.