…….

High availability is a concept that says you will have zero downtime.

Consider an old world situation. You have a server that is serving exactly one website. On that server you are running an Operating System, a database engine, a web server (Apache/Nginx), an interpreter, and a bunch of code and HTML.

In your browser (client) you type “http://www.awa-example.com”. This causes your computer to send a request to a DNS server to translate “www.awa-example.com” into an IP address. Dozens of computers working in a distributed way work to get that answer. Your browser then opens a TCP/IP connection to the address it was told to use.

When that connection finishes traveling across multiple different servers (routers) it arrives at my server. My server examines the packet and determines that it is addressed to the web server. The web server looks at the request and sees that the request is for www.awa-example.com. It looks through its configuration files and decides on which interpreter to use. It transfers the request to that interpreter.

The interpreter loads configuration files and loads the code to execute/interpret. That code runs and opens a connection to the database engine. The code makes a query to the database, the database returns a result, the code formats it and sends back a message to your browser, which displays it.

If any part of that long set of computers/servers and software fails, your browser doesn’t get an answer to display.

We have a “demark” that marks the point of responsibility. Anything outside the demark is “their” problem. Anything inside the demark, including the demark, is our issue.

What that means is that in a high availability system, there has to be at least two of everything. On the outside, we must have two “theirs” and two demarks. Linode provides us those multiple “theirs” and demarks. If one link into their data center dies, the others take up the load and everything continues as if nothing was at issue.

If we are worried about the data center, they offer data centers all around the world. We are happy with just one data center.

At their demark they send traffic to one of two “node balancers” we have purchased. These are in different racks. Each is capable of handling all traffic into our cluster. If they need to update the node balancer, they can update one, wait until it is up and running, then update the other. This is either software or hardware. They can physically turn off one rack, and we won’t even notice.

We use a cluster to support our clients. There are 6 nodes (servers) in our cluster. 4 are ours, two are theirs. Their nodes are used for the “control-plane”. This is what controls our cluster. When we tell our cluster to do something, it is the control-plane which orchestrates the other nodes.

We run two ingress pods. The node balancers send traffic to these pods directly to our nodes in a round-robin. If we need to upgrade our ingress, the cluster will create a new ingress pod, make sure it is up and running, then terminate one of the old ingress pods. It then launches another ingress pod, when that is up and running, it terminates the last old one.

There is NO downtime as this happens.

The ingress sends handles external SSL and internal traffic to services. The cluster receives traffic at the service and forwards that traffic to which ever pod is providing the service. If we run at least two pods, we will not have downtime. We set things up so that pods run on different nodes, if possible, so a node failure doesn’t take down all the pods.

Which brings us to the tail end of all of this. Our pod.

Our pod, in this case, is running WordPress. If the pods can mount a file system in ReadWriteMany then multiple pods can access the same files at the same time. WordPress has a directory of content. These are files that we upload, images, PDFs, videos, themes, and a boatload of other things. We want to have that directory accessible by all our WordPress pods.

We don’t have to worry about the database, that runs in its cluster. If one of the database engines dies, the others take over with no loss of function. We run a hot spare style of replication. We could use a multi-master version, it isn’t worth it at this time.

And this brings us to The Issue. Linode provides us with persistent volumes. This works perfectly for many situations. Unfortunately, those persistent volumes are ReadWriteOnly. This means that only one pod can access the files at a time.

Since there is only one pod, there is no redundancy. If that pod fails, the site goes down. If the node that the pod is running on fails, the pod fails, the site goes down.

On good days, when a pod fails, it is restarted and the replacement is up and running shortly thereafter. Downtime is low, but not zero.

Linode isn’t going to offer a ReadWriteMany anytime soon.

Which brings me to Ceph!

Ceph is a distributed block storage system with the ability to run a distributed file system on top of that block storage.

All I should need to do is deploy it to my cluster. Sure, if I want to buy 3 more nodes/servers and a bunch of disk space for them. Think $1000s of dollars per month.

But there is a version for Kubernetes called “rook”. It can even use persistent volumes. After a few days of fighting this on my local Kubernetes cluster, I finally got it working. All that was required was to deploy it to Linode.

12 hours of fighting and I finally got it mostly functional. Until I went to allocate block storage for Ceph. Linode doesn’t allow block persistent volumes! ARGH! I’m stopped.

Then around 2300 Monday I got it. I used the same volumes that Linode’s persistent volumes used and attached them directly to my nodes as block storage. Amazing! It works.

Today I got it all configured and running. I will be upgrading GFZ to be a HA site in the upcoming weeks.

The world is better.

There are many tropes in story telling. One of them is “Don’t judge a book by its cover”.

People misjudge others all the time. I recently watched a video of a “prank” gone sideways. A “prankster” filled a gas can with water, went into a parking lot, found somebody sitting in their vehicle, poured the water on the vehicle and then pulled a lighter.

The “joke” is to see the fear in the people’s faces when they believe they and their vehicle are about to go up in flames.

The sideways was when the “prankster” went up to a pickup with an older man inside. As he started to pour the water, the driver got out with a pistol in his hand. The “prankster” then was recorded with fear on his face. Running for his life, screaming that it was just water.

The “prankster” had misjudged his target.

Another trope is the mentor telling the younger newbie, “Listen to that old fart, he’s old in a profession where you die young.” Old doesn’t mean stupid. Old doesn’t mean without means. Old, fat and slow can still be deadly fast.

I said I never had much use for one. Never said I didn’t know how to use it. — Matthew Quigley, demonstrating his skill with Colonel Colt’s revolver”.

I normally write my daily postings the night before. The article about pulling bullets was written Friday night for Saturday morning. After I wrote the article, around 2322, I decided it was time for bed. And got distracted on the way there.

The reloading press was right there. I had warmed up the scale before starting the article. So I sat down and reloaded the bullets I had just pulled.

Per the Hornady manual, 14.5gr of Alliant 2400 under a 156gr HDY GTX. Success!

Today I did a test firing. The .357 Mag was an afterthought.

This required more digging. I cannot find the crate or ammo can that should be full of powder, nor can I find the can or crate full of 30-30 brass.

I do find the 30cal can labeled “30-30”, I open it, hoping to find cases. Nope! Instead, I found the mother load. Two boxes of bullets. Old bullets. A couple of plastic ziplock bags labeled for 30-30. A bag with a single case ready to reload.

The bottom half of the can was full of loaded, ready to use 30-30.

Using the instructions I got from the owner of boxes for ammo, I started backing them in nice white boxes. 6 boxes, 20 rounds each of 30-30. I don’t need to reload any 30-30 today!

I took the Marlin 30-30 with scope out to try to zero it. Not a lot of joy there. I need to do it again. Since I was taking the 30-30 out, I took the Ruger GP100 with .357 Mag loaded up. Fired off the cylinder. Perfection. All the cases ejected as they should. No sighs of over pressure on the primers, and I’m below max load.

The 30–30 Marlin punched my shoulder harder than I expected. It didn’t zero. I’m not sure what is going on there. My Winchesters, firing the same loads, do an outstanding job of ringing steel and punching holes in paper, exactly where I want. Iron sights and all.

Sunday I need to finish digging through the reloading room. I know there is a couple of cans of powder in there. I just need to find them.

When you make a mistake, own up, do the right thing.

I didn’t do the right thing. The right thing was to pull all the bullets for the hot loads I made. I didn’t want to because it is a bit of work, and I’m lazy. It should be safe to unload them through the R92.

Lazy, safe, reloading do not work. You have to be careful, you have to do the right thing, every time.

I have two or three powder throws. I do not use them. The consistency I was getting with them wasn’t good enough.

The two presses I have are both Lee presses. One is an OLD single stage. I like it. I don’t use it. At this point, my main complaint, using it to deprime, is no longer an issue as I use a different method. The press I do use is a Lee Turret press.

When I started using it, I used it as a turret press. Everything was all setup and I could crank out the rounds quickly and easily. First I would deprime, then clean the cases in a wet tumbler. From there, it was time to run it through the turret press.

Put the case in shell holder. Handle down to resize the case. While the handle is down, place primer in the primer cup. Handle up to insert the primer and rotate the plate to the next station. Handle down to flare the mouth, throw the powder charge. The Lee charger is semi-automatic. As the case presses into the die, it causes the charger to throw one charge. Handle up, place bullet in the case. Handle down to seat the bullet. Handle up to move to the next station. Handle down to crimp the case. Remove the cartridge, put the case in the shell holder, repeat.

Today, I use the turret press as a quick change plate. There is a universal depriming die, sizing die, mouth flare, bullet seating+crimping die. I rotate the plate to the die I want, then use it like a single stage press.

Which brings us to how I do it now. I have an automatic powder measure. You input the weight you want, press the “drop” button. The machine twists a trickle tube until the scale reads exactly the weight you wanted.

It takes just a few moments to throw the charge accurately. I have a standalone powder charger stand. I have just the base in it. This gives me a big enough target that I don’t need a funnel. I hold the case mouth against the bottom surface, pour the pan into the top and the case is charged.

The pan gets placed back on the scale for the auto-throw. Once it stabilizes, it will start throwing the next charge. While that charge is being thrown, I’m putting the case in the press, putting a bullet on and seating it along with the crimp.

Not as fast as a progressive, by far. Not as fast as just using the turret press. But it is the way I do it today.

So back to pulling bullets. I have a collet puller which hasn’t worked for me. Instead, I use an inertia puller. Put a round in the puller. Hit it on a hard surface a few times, the bullet pops out. It works, it just takes more of an effort than I like, as a lazy person.

Now that all the bullets are pulled, I can reuse the bullets and put the correct charge in place.

Thank you to everybody who told me I was being a dumb ass. I was. Your gently chiding was enough to make it clear I needed to do the right thing.

This is what I was thinking of:

Since the video, they did discover that the SLAP rounds that he was firing were not military surplus. That they were way too hot and the cause of the exploding 50 BMG rifle.

Judge Benitez is hearing this case. I believe there are a few other cases he is hearing as well. Earlier this month there was a hearing. He has now set the schedule for the trial on the merits.

This is the point at which he will hear all the evidence, decide which are “facts” and from there he will issue his opinion.

On June 30, 2023, judge Benitez set a hearing date of July 17, 2023, at 1400. He issued the following order:

The state responded with:

The state isn’t willing to tell the court, in writing, before the hearing what their arguments and evidence are. They are reserving that for the day of the hearing.
Read More

Kubernetes (k8s) is a tool for running containers on a cluster with a high level of redundancy.

Consider the case where you have a website that must be up all the time. You containerize the website. This adds a gigantic level of security but still gives you great flexibility.

The site is running just like it is supposed to, and you decide it is time to upgrade. You “rollout” the next version of the software. The new software is deployed, it starts running and is fully functional. The K8S cluster then changes some internal routing, and all traffic that used to go to the old container now goes to the new container.

When all traffic to the old container is done, the old container is terminated. Total downtime, zero.

Now it does work this way, MOST of the time. There are a few complexities to this which can cause issues. Your website consists of three parts, the “assets”, the “software”, and the “database”.

Since the database is running in a different container, there is no issue on the rollout of the website.

The software is part of the container. No issues there.

The assets, though… Those live on disk. The new website container must have access to those assets.

K8S resolves this through a Storage Method. A storage method is just away for you to describe storage that exists outside the container and which persists even if the container goes away.

The reality is that the data exists on the “node” and the container must access the node storage. Some types of storage are all network and K8S has the tools to use it straight out of the box. Other storage methods are just ways of exposing the node’s file system to the container.

Consider these three methods, SAMBA, MS Shares, and NFS. All three of these methods are pure network based. You have some sort of server that serves the files over the network. K8S attaches the file systems directly to the container. Since the actual data exists back on the file server, it is possible to have multiple nodes access that data at the same time.

Now consider an iSCSI target. There is a file server that is serving that target. That target is exported as a block device. Think “hard drive”. When you attach it to a node, it is as if you had plugged a new hard drive into that node.

Once that block device is plugged in, you can do other work with it. Not as it is. You first have to format it format c: for Windows users or mkfs /dev/sda for Linux people.

This is fine. Multiple nodes can attach the same block device. If one node formats the block device, the other nodes see it as a formatted drive that can be mounted.

Unfortunately, this will lead to data corruption. Most file systems are not designed to be modified from multiple hosts at the same time. There are only a very few file systems that are capable of running across multiple nodes at the same time.

This is the issue we are currently having with Linode. We request an iSCSI block device from Linode. This is provided for use as a “persistent volume”. When a container needs access to a particular volume, it uses a persistent volume claim.

To access that volume, K8S tells the node to mount the device. The node then allows the container to mount that part of the file system. Because the file system is single use only, the PV can only be mounted on one node at a time.

To access that PV on a different node, the current node must detach the PV. The container must move to the other node, the other node must mount the PV and only then can the container access the data again.

On Monday, a node attached the PV and then died. It did not detach the PV. K8S instantly discovered that GFZ was down and started a new container on a different node. The new node asked to attach the PV. The old node was dead, so didn’t say a damn thing. For 6 hours it sat like that until we destroyed both the new and old node and got a new new node, at which point everything started working again.

This happened because Linode forced an upgrade of our K8S, and it did not go cleanly.

On Tuesday, we upgraded K8S again. This time the new nodes all came up. Unfortunately, the node which GFZ was attempting to run on refused to attach the PV. Once Linode had looked at the issue, we kicked that node hard and it all started working again.

We need to have something that provides reliable, robust, multihost access. The answer is something called “ceph”. That was my fun for the day. Learning enough about ceph to allow us to migrate to using it for our persistent volumes.

On the wall of the living room is a display rack. My family got it for me for my birthday. It has an American Flag motif and holds four rifles and has a small cabinet at the bottom for ammo and other accouterments.

On the bottom is a Henry Golden Boy in .22LR. Squirrel gun.

Above that is a Rossi R92 in .357 magnum. Raccoon rifle.

Above that is a Winchester ’95 in 30-30. Deer rifle.

At the top sits a Henry in 45-70. Bear rifle.

The .22lr is mostly a showpiece. It is fun to run, but it is not my go to for taking squirrels. The R92 is the rifle I’m most likely to pick up when things go bump in the night. The 30-30 is too big for small game. The 45-70 is there for when it is needed.

The 45-70 is firing +P rounds. When the first round hit the 8in steel, it knocked the plate off the chains. The 30-30 is firing cast bullets, and I’ve not had any luck with the new mold. I haven’t made the time to cast, and it is too freaking hot to do casting right now. I did source 300 Hornady FTX bullets, so I’ll be loading some rounds up shortly.

Which brings me to the dumb. I’ve been shooting cast bullets out of the R92 loaded over 5 grains of TiteGroup. Nice load. In .38 SPL, 5gr is a +p load behind a 125GR HDY XTP, which is what I loaded up over the weekend. I only shot it out of the Ruger GP100. It did what it was supposed to. Felt pleasant in the hand.

Subsequently, I went to load up some .357 magnums. I checked my notes a few times, wrote down the recipe. Double-checked here and there. Did one final check and then started loading 158gr Hornady XTP over [REDACTED] grains of Alliant 2400. Loaded 50 rounds. Went out to the range and sent 5 rounds down range.

Nice punch. Recoil was there. It even left a small ache afterward. It made the steel ring and swing. Then I cleaned everything and was getting ready to shut down for the night when it occurred to me to check another number.

Yep. The amount of 2400 for a 125 gr XTP was what I put under that 158 gr XTP.

Yeah, those rounds are hot.

I checked the primers for over pressure. There might be some signs, but nothing really stands out.

Today I decided to check the loads out in the Ruger.

Now that was a kick. It RANG the steel. When it hit the box with the spatter target on it, the box jumped. Lots of energy being transferred.

I should have stopped the first time I tried to cock the hammer and the cylinder didn’t want to rotate freely. A little help, and it rotated into position for the next shot.

After we were done, I went back in and went to dump the cylinder. The cases didn’t fall out. That’s ok. I’m told it happens. I pushed on the ejector rod. Nothing. The cases didn’t even budge a little bit.

In the end, I took to tapping the ejector rod gently until the cases ejected.

The pressure was high enough to pressure from the cases to the cylinder walls. Those rounds are not going into the revolver again. They are far too hot for that hand gun.

I’m torn. What I should do is pull every round. What I want to do is just send them down range through the R92.

Do as I say, if you make a hot load and you know it. Don’t mess around, just pull them. You can save the bullets. So what if you lose a bit of powder.

That’s what I felt like, working issues yesterday. There was always another problem to fix.

It is a 122-page opinion written by somebody who was looking for a way of finding the plain text of the Second Amendment too hard to read. It is going to take time to wade through it.

My wife had her first true vacation in years last month. Her mother invited her to go to Greece with her and her husband. She had a wonderful time, but we were both ready for her to get home.

Preparing for her return, I told Miss Google, “Play love songs from the 50s and 60s.” This worked well on the drive to the airport. And I heard a song I had heard many times before and thought, “That’s the right one for Tuesday Tunes. She’ll readyreally like this and know I was thinking of her and telling her with all my heart how much I love her and missed here while she was gone.”

And of course, in the excitement of seeing her again, I lost the song. I can’t freaking remember the title. Just that it had something nice about saving kisses for her.

Instead, here’s Frank with a love song.