• Resiliency is a goal. I’m not sure if we ever actually reach it.

    In my configuration, I’ve decided that the loss of a single node should be tolerated. This means that any hardware failure that takes a node of line is considered to be within the redundancy tolerance of the data center.

    This means that while every node has at least two network interfaces, I am not going to require separate PSUs with dual NIC’s, each with two 10Gbit interfaces. Instead, each node has two 10Gbit interfaces and a management port at 1 to 2.5 gigabits RJ45 copper.

    Each node is connected to two switches. Each switch has a separate fiber, run via a separate path, back to a primary router. Those primary routers are cross connected with two fibers, via two different paths.

    Each of the primary routers has a fiber link to each of the egress points. I.e., two paths in/out of the DC.

    The NAS is a distributed system where we can lose any room and not lose access to any data. We can lose any fiber, and it will have NO effect on the NAS. We can lose any switch and not have it affect the NAS.

    We can lose any one router and not impact the NAS.

    So far, so good.

    Each compute node (hypervisor and/or swarm member) is connected to the NAS for shared disk storage. Each compute node is part of the “work” OVN network. This means that the compute nodes are isolated from the physical network design.

    Our load balancer runs as a virtual machine with two interfaces, one is an interface on the physical network. The other is on the OVN work network.

    This means that the VM can migrate to any of the hypervisors with no network disruption. Tested and verified. The hypervisor are monitored, if the load balancer becomes unavailable, they automaticity reboot the load balancer on another hypervisor.

    So what’s the issue?

    That damn Load Balancer can’t find the workers if one specific node goes down. The LB is still there. It is still responding. It just stops giving answers.

    I am so frustrated.

    So I’m going to throw some hardware at it.

    We’ll pick up a pair of routers running pfSense. pfSense will be augmented with FRR and HAProxy to provide load balancing.

    Maybe, just maybe, that will stabilize this issue.

    This is a problem I will be able to resolve, once I can spend time running diagnostics without having clients down.

  • In upgrading from copper to fiber, I’ve been exploring the different options and learning as I go. Some learning curves have been steep, others have been “relearning” what I already knew.

    One of the biggest things I needed to learn is that there are “switches” that are actually “routers”. That was mind-bending.

    The other is that the network dudes talk about VLAN and Tagged VLAN. They are different things. In the environments I’ve been working in, there are only tagged VLANs which are called “VLAN”. Same name, different meaning.

    The starting place when moving from copper to fiber is to understand what a Small Form-Factor Pluggable is. This is the magic that makes it all happen. This is standardized into SFP and SFP+. The SFP standard only supports 1G and lower speeds.

    The SFP+ supports higher speed modules. 10G, 25G, 40G and 100G are all standards I’ve seen.

    I’m only working with 10G modules, at this time.

    They have modules that are RJ45 copper that will run at slower speeds or up to 10G. The only issue is that they draw more power and run hot. Can’t touch them when running hot.

    The fix for this is to purchase a switch or router that has RJ45 Ethernet ports and at least one SFP+ port.

    I found a small, six port, switch. This comes with 4 RJ45 ports, rated at 2.5G each, and 2 SFP+ ports rated at 10G each. Cool.

    This allows me to daisy-chain them if I wanted.

    In reality, it meant that I had one host connected at 10G while the others were at 2.5G.

    I also found a L2/L3 “switch” that looks much like the switch above.

    Having done the upgrades, I started looking into upgrading the router between the outside world and the DMZ. The routers I’ve been getting to not support any crypto, so they don’t have good VPN capability, something I want.

    So I went looking. Looking for a “motherboard with SFP”. Something interesting popped. A mini ITX motherboard with 4 SFP+ ports and 4 RJ45 ports along with HDMI, VGA and the standard USB ports. It also provided space for two M.2 SSD modules, 2 DDR4 slots and two 6GByte SATA ports.

    It might not be the fastest computer on the block, but it looks like a good starting point.

    This leads me to other motherboards of the same ilk. And what I found was a bunch of these motherboards. And the port layouts all look the same. The specifications all look the same.

    What we have is a “standard” motherboard which is put in a “standard” case along with a wall wart, HDMI cable and a mounting bracket. The branding stays the same.

    I have an L2 switch that I’m going to take apart in a bit. It has a limit of 1550 byte packets, making it useless for my new network. I wonder if I will find an M.2 module in that box or something else that allows me to change the software.

    Meanwhile, that motherboard is on my wish list. I’ll load pfSense on it along with FRR and replace my current router. Giving me a considerable boost in capabilities and letting me dispense with the VyOS configuration language. Which I really don’t like.

  • This is the prime time to test your preparations. Christmas is over, but people are not settled. It’s not “usual” scheduling because kids are off school, and you may be off work. So… Go turn off your power at the main breaker.

    Why? The answer is that preparation only works if you’re actually… prepared. You cannot KNOW that you’re prepared until you test your preparations. That’s where turning off the power comes in.

    In my house, the first thing to happen if the power goes out is to locate light sources. Immediately, that might mean the flashlight on my phone, but only briefly. I want to keep the power up on the phone in case I need it as a phone. I find the flashlights, candles, oil lamps, and I get at least one lamp lit. This means I have fire, which means the world gets that much easier. With one tiny bit of fire on hand, I can start numerous others.

    When the kids were little, the next item at hand would be child wrangling. If it was daytime, the kids would be sat down near the wood stove with books appropriate to their age, or a game to play that wasn’t electronic, and told to stay out of the way. If they got in the way, they got to do “fun things” like shovel and gather wood and other stuff they hated. When they were little (under 10), it was easier to keep them busy and out from underfoot.

    As they got older, the kids were expected to do many of the “power outage” tasks on their own. It was their job to locate flashlights and solar lamps and make sure they worked. One was set to starting the fire, if it wasn’t already. The other went around and turned off all the light switches and other power hogs, so that we wouldn’t overload the circuit when power came back on. If the power was going to be out for more than 24 hours, items in the fridge were moved to our inside but unheated porch (it gets cold, but rarely below freezing) to keep them fresh. Frozen items went into the outdoor freezers, which would stay frozen for a very long time.

    After a few practice runs, we had it down to an art form. Everyone did their jobs, and within a half hour, the entire house was ready for there to be no power for however long was necessary. We had blankets over windows to hold in heat, pulled out sleeping bags so that they were ready for night, if we wanted to sleep in our beds, had easy to prepare foods on hand in case we were tired from shoveling or whatever. Everything just worked.

    The house went without power for anywhere from several hours to a few days on a number of occasions. We’ve always been fine. The wood stove sits over the water pipes in the basement, so the residual heat keeps them from freezing. We always have access to water, even if we have to go tromping to get it. We know how to make sure water is potable, too. By nightfall, we usually had everything in place, and we were all cuddled up by the wood stove, reading or talking or playing cards.

    It’s not difficult, but it is complex. There are a lot of moving parts to get figured out, and until you put them all to the test, you don’t KNOW how it’s going to work. It’s much better to do some test runs long before you actually require all this stuff to be working. Make your family a well oiled machine before the emergency happens, and the emergency won’t be catastrophic.

  • SCOTUS

    We currently have one case scheduled for the January 10th conference. The other two cases have not been rescheduled yet. I do not know what we will see in the court filings before the 10th.

    What does it mean to be conferenced?

    When a party petitions the Supreme Court for a Writ of Certiorari, they are requesting that their case be heard by the court. If the petition is submitted through the standard channels, it is processed in the “standard” way.

    If the request is submitted through the emergency docket, sometimes called the shadow docket, then a single Supreme Court Justice will evaluate the submission. They can then refer it to the court as a whole, or they can deny the request, or they can request more filings.

    Regardless, emergency or regular, cases that are referred to the court will be examined. This happens in stages.

    The first stage is the parties filing briefs on why the Court should grant cert or why they should deny cert. The parties can decline to file briefs, but they should file formal documents saying they are not filing.

    When all the briefs have been filed, the case is “Fully briefed.”

    Once the case is fully briefed, it is distributed for conference. This means that the briefs are provided to the justices for examination. The justices have their law clerks do law clerk things and provide reports. All of this is generally kept out of the public light. Nobody knows what the justices ask of their clerks, but former clerks have reported doing such things, in general terms.

    After the case is distributed, it is scheduled for a conference. The conference happens on a Friday. During the conference, only the Justices are present. No clerks, no witnesses, no experts, just the justices.

    They discuss the cases that were scheduled for conference. They can also discuss whatever else they want. They decide, not you, not I, not anybody else.

    They can also talk to each other before the conference and make decisions outside the conference.

    The conference is the formal event.

    If the Justices want, they can reschedule a case, before they discuss the case in conference. This is what happened to Snope and Ocean State Tactical.

    After the conference, the court will issue their orders. These orders will be “grant”, “deny”, or “relist”. If a petition is denied, it is over. If the petition is relisted, it means that the Justices will be discussing the petition again, in a future conference. If a petition is granted, then it is going down.

    A case that has been relisted can be relisted for any reason. Some known reasons include having time to write a statement to attach to a denial of cert, sometimes it is because they need additional information. One of the common reasons in the Robert’s court is a “suitability” phase.

    John Roberts likes to have a case relisted after the Justices have decided, internally, to grant cert. This is to give the law clerks time to thoroughly investigate the case to make sure there are no hidden issues or things that might moot the case.

    We now have three Second Amendment cases that have been distributed for conference. One of which is currently scheduled for conference on the 10th.

    Infrastructure

    We have completed most of the network upgrade. There is still a server that requires a network interface upgrade. There is still a rack that requires a switch upgrade. Not to bad.

    We need to finish running the redundant fiber network for backup purposes. Once we finish running the redundant fiber, we will upgrade the primary router and make sure everything understands multipath routing.

    All the cluster entities have been placed on virtual networks. This means that they no longer need to worry about multipath nor the physical layout of the networks. A complete separation of tasks.

    It turned out that I was having issues with my nodes having their clocks drift/skew relative to each other. This has been fixed, which leads me to want a Stratum-1 NTP server, again.

    The last time around, I used a handheld GPS unit as my clock source. It worked wonderfully. This time I’m looking at something in an IoT idea.

    I am currently researching NTP servers via Wi-Fi. If that gives good results, I might just do a Raspberry Pi Pico W and put the darn thing outside in a waterproof enclosure. I’m still investigating. I’m also attempting to avoid soldering as somebody broke my Weller soldering station.

    Christmas

    Ho Ho Ho, Merry Christmas!

    We hope you had a joyous Christmas.

    Happy New Year!

    And we wish you a happy new year! My your fortune be bright.

    Question of the week?

    Having had time to learn about what happened in western North Caroline after the hurricane, what changes in your prepping model have you made?

  • Christmas is past for another year. It was better than expected.

    Watching movies with the family was good. My wife insists on “A Christmas Story”, as it is her favorite. I picked “Red One” on a recommendation from Scott Adams on X. The final movie was “A Christmas Story Christmas”.

    This last hit a bit hard.

    Regardless, friends came through, and we were able to give back to our friends.

    My wife’s best friend’s husband passed earlier this month. We had her over for Christmas Eve dinner (tacos) and Christmas Dinner (Turkey with fixings).

    Our tradition is to go around the table and each person gives thanks for something that happened that day. Sometimes it leads to discussions, sometimes it is just a little thing, “Thank you for a dinner, I really like.”

    The goal is to stop perseverating on the bad that is happening around you, the things that are getting you down, and to acknowledge, to search for, the good that you have.

    My friend from the NVL called on Christmas Eve. That was a good talk. The only bobble was when he let his distrust of Elon slip out. We have agreed not to talk politics. We are still friends.

    My best friend died in November 2000. I don’t think I ever recovered from that day. He was not only my friend, he was my mentor.

    He was the first person I met that could program better than I could. He was a better man than I, by far.

    I found myself competing with him in programming to be better. He never competed with me. He just won. After a while, it stopped being a competition and became a lifelong friendship.

    Through Mike, I met Max. Max called me on Christmas Eve. Talking to him made me feel better. Friends can do that.

    So on this day, after you have finished with what’s under the tree, had the first of a week’s worth of leftovers, take a moment to reach out to a friend and let them know what they mean to you.

  • First off, Merry Christmas! If you’re on the blog on actual Christmas morning, go open presents! Read this later. *grin* I’ll be doing the same, because I’m writing this in advance, so that I don’t have to look at the blog today. There.

    Right… last week the Continuing Resolution was supposed to pass. All 1500 pages of it. Musk made We The People aware of it, and We The People made the decision to call our politicians to say HELL NO. Those 1500 pages were not passed. I considered that incredibly positive. And then I heard the commentary from the Left and others:

    “House Republicans have been ordered to shut down the government. And hurt the working class Americans they claim to support,” Jeffries posted on the social platform X. “You break the bipartisan agreement, you own the consequences that follow.” (1)

    “They come in and undermine us two days before the government shutdown deadline?” she asked. “That doesn’t seem like, at all, a good way to do business.” (2)

    “I thought it was a reasonably good compromise,” he added. “The problem was one or two people in our conference, instead of just opposing it, had to go out and demonize and distort things, provisions in the bill.” Republican Rep. Kevin Bacon (3)

    There are bunches of other comments that made me wince. One senator I caught a clip of basically said that because We The People chose to tank this pork-ladened CR, we were snatching disaster relief from the Carolinas and elsewhere. That one made my blood boil. Biden just sent a billion to Africa. We HAD money to take care of our own, and Biden and his people sent it elsewhere. Coming in at the last second and tacking aid onto a CR that’s meant to keep the government running is just trashy.

    There is a LOT going on with all this. The bottom line is, we want to see an end to the bloat in Washington. That means that a lot of Democrats and probably quite a few Republicans are going to get their panties in a wad. I am really okay with that. I’m okay with anything that reduces the size of government. Here’s the thing… our Founders never imagined career politicians like we have today. They couldn’t have. It’s so against their beliefs, because it essentially mimics the aristocracy in Britain. That’s exactly what they were trying to avoid.

    So… Trump and his team have come in, and they’re up to making changes. To the people who’ve suggested Trump is overstepping by attempting to lead the country now instead of waiting for January 20th, I say, “At least *someone* is attempting to run the damn country. Biden sure isn’t.”

    (more…)

  • Here is hoping that the happy man in the red suit brought you all you wish.

    More importantly, I wish you health and happiness in the coming years.

    -Chris

  • So far it is only Gray v. Jennings, but it is still early.

  • For those of the Jewish faith, I want to wish you a happy Hannukah. May your latkes be delicious, may your oil last all eight nights, and may the darkness of the year be driven away by the light of the candles. 🙂

  • We, at Vine of Liberty, wish you all a Merry Christmas and a Happy New Year.