Chris Johnson

WordPress Issues

For years the go-to source for consistent container images has been Bitnami. Bitnami is owned by Broadcom via its acquisition of VMware in 2023.

With over 200 different containers covering every part of the container infrastructure, developers and system administrators have gone to Bitnami images for consistent interfaces and methods.

Until last month, these container images were offered free of charge to the world.

Starting on the 1st of this month, only development containers are offered free of charge. This means that if you want tagged versions of WordPress, you need to buy a subscription for the Secure Containers from Bitnami.

I looked into this. Does anybody think paying $62,000/year is a good deal? I don’t.

This is not a long-term issue; it is a pain because I have to reconfigure all my WordPress clients to use a new source for images with different mounting points.

Wire Runs

There are times when it is so simple that you can’t find instructions.

When running redundant cable or fiber, you want to make sure that there are no single points of failure. I remember the great Internet outage of the early 90s. Some good old boys in a swamp down in the south took some potshots at something hanging on telephone poles.

They managed to sever a fiber optic bundle. This cut the US Internet in half because all the redundant fibers were in that same optic bundle.

The contracts for those fiber circuits had been let to different long-haul fiber networks. Those long-haul networks were picked because they had north, south, and mid-Atlantic routes heading west. When a new dark fiber vendor came on the market, offering better service with better prices, these long-haul networks purchased their dark fiber from the same vendor.

And all the circuits ended up in a single bundle, down south.

I currently have a star configuration for my network. I’m going to be adding a ring around the outside of the star. This allows better throughput between adjacent data closets. Instead of going to the hub then out to the data closet, we will be routing closet to closet.

While the “cost” of that extra hop isn’t really measurable, it does put some limits on the total cluster bandwidth.

Say you have to move 10 Gigabytes from Node 101 to Node 102 in the same closet. We move that at 10 gigabits, so around 15 seconds to move the data.

But say you want to move the same 10 Gigabytes from Node 101 to Node 103 in a different closet. It still takes around 15 seconds to move the data.

But what if you need to move 10 gigabytes from Node 101 in closet 3 to Node 103 in closet 2, plus 10 Gigabytes from Node 101 to Node 105 in closet 1, and 10 Gigabytes from Node 101 to Node 129 in closet 0? Each movement would only take 15 seconds, but the total time to move would be 3*15, or 45 seconds.

With the extra links in place, we can go directly from Closet 3 to Closets 1 and 2. This will reduce our total time to 30 seconds.

I have the hardware to do this ring. What I don’t have is the conduit. If I use the existing conduit, it is easy. But that means if that conduit is compromised, I lose access to that closet.

I figured out that I can run some conduit along the outside of the building with much shorter runs, much easier access, and with the level of redundancy I’m looking for.

When I finish pulling fiber, I will have three paths to every node.

OSPF/Multicast

In networking, we use broadcast packets when we want to reach every node on a network segment. In the old days there was a single IPv4 address for that, which was the host portion of the IPv4 address was all ones.

It turns out that there are other reasons to want to use a broadcast packet. Often for discovery purposes. OSPF uses this method to find other routers on the network segment.

Which leads to the issue of L3 managed switches. These things are not advertised as full routers, though they can act like them. In particular, each of these switches can run an OSPF process.

This means that the network topology is quickly updated when things change.

And these low cost managed switches don’t understand how to forward OSPF packets when they are running an OSPF process.

Another thing to figure out and fix.

Government Shutdown!!!

The world is wonderful. With the government shutdown, they are less likely to FA with me.

The reality is that the government barely notices when it is “shut down.”

It looks like the Democrats played into Trump’s agenda. J.D. Vance has stated that they will start firing people very shortly if the government isn’t started back up.

The government is shut down because the Democrats and the Republicans could not reach an agreement on a continuing funding resolution.

The Republicans have put forth a CR that funds the government at current levels but for VA housing, security for federal law enforcement, and security for the Supreme Court.

There is nothing in the CR that the Democrats object to.

The Democrats have put forth a proposal that adds more than 1 trillion dollars to the budget. They want to restore funding for healthcare for people that are unwilling to meet Medicaid work requirements. They also want to provide funding to states that are showing healthcare budget shortages because the state spent state money on providing healthcare to criminal illegal aliens.

The big word is “fungible.” Let’s say you want to buy a fancy dinner for your wife. You have $1,000 in the bank. You owe $700 for the mortgage, you owe $100 for Internet, and you owe $100 for water. This leaves you only $100 for food and other things. This means you can’t afford to take your wife out for a $100 dinner. There is no money in the budget.

You reach out to your parents and ask for help with the Mortgage. They agree to pay $500 for the mortgage but tell you that it can’t be used for “fun” things.

To make sure they send the $500 directly to the mortgage holder. You send your $200, the mortgage is paid for the month. None of your parents money went to fun things like dinner out with your wife.

You now have $600 for food and other things. You allocate $250 for food, spend $100 on other things. You have $150 of your money to take your wife to dinner.

Where did the money come from for that dinner? From you or your parents.

Fungible means that you can’t send any money to states that are spending money on criminal illegal aliens. You can’t send any money to organizations that spend money on abortions.

Question of the Week

What is your opinion of this government shutdown?

Costs

Like almost everything I learn, the startup costs are quickly outweighed by the cost of the “tooling.”

My lathe and milling machine cost less than $1500. The cost of my micrometers exceeds $1000 without counting toolposts, toolholders, inserts, and a wealth of other things.

Reloading is something like that. You can get a starting kit for around $175 that has just about everything you need to start reloading. You need to add dies, reloading manual, and consumables.

Assuming you are reloading for 9mm, your dies will be between $40 and a couple of hundred. I’m currently using Redding, which prices out at $75. Manuals run $25 or so. This puts the total cost of tooling just under $300.

Your consumables will be primers at $80 per 1,000, powder at $156 for four pounds, 115gr RMR JHP at $115 per 1,000.

This puts your consumables at $351 for a total start cost of $650 or so.

Assuming you are reloading “free” brass, this gives you a startup cost of $0.60/round, which is more than range candy but less than personal defense ammo.

After your initial investment, your costs will decrease, but it will take a while before you are truly saving money.

Your next 1,000 rounds will cost you $80 for primers and $115 for bullets, giving you a cost per round of $0.195. This would bring your costs from $0.60 to $0.422 per round, including your original investment.

The place where costs really change is when you need to start reloading a different caliber. Using the same equipment, you can add a set of dies, $75, and begin reloading a different caliber.

Availability

During the great panic-demic it became nearly impossible to buy certain calibers of ammunition. Even now, it is sometimes difficult to buy certain calibers.

If you have reloading equipment, those limitations are reduced.

I, personally, reload .38 Special and .357 Magnum with cast bullets. I cast HP myself and find those rounds to be consistent in weight and go where I point them.

My LGS was only selling cowboy loads and very expensive self-defense loads. They had no brass, they had no bullets, they had no primers, and their selection of powders was limited. For the cost of a set of very nice bullet molds, I was able to create freedom seeds for my R95 and revolver.

8mm Mauser is difficult to find and expensive; I’ve got nearly 500 rounds of it for the cost of bullets and powder. The same with 30-30 and 45-70.

I believe the only firearms I cannot reload for are my shotguns. Since I only run 4 rounds a year out of them, it’s not a real issue.

This is the great power of reloading, in my opinion. You can reload for unusual calibers for the cost of a set of dies and bullets.

Quality

The big-time long-distance shooters swear by hand reloading their ammo. I believe them. When I’m reloading, I attempt to maintain that same level of quality as they do. I may or may not succeed, but it is nice to know that I’m close.

Safety

There are two big rules in reloading:

1. Don’t Shoot Other People’s Reload.
2. Your mistake can kill you.

I currently use a Franklin Armory hand primer. It works for me. I can feel each primer seat properly. Before I had the hand primer, I used two different on-press tools.

One day, I was at a private range shooting .45 Colt out of a Marlin lever action. I would pull the trigger, the hammer would fall, and there would be no bang. Not even a pop. This was a problem, but I found nothing horribly wrong. It just meant I had to fix something.

Then I switched to my 1911. Bang. Bang. Pop. STOP!

I had a squib.

This led to me pulling every bullet on that set of reloaded rounds. Luckily there were only 20 of each caliber.

Conclusion

The startup cost of reloading is not horrible. The cost grows like in every hobby. Your break-even point will be in the 4,000 round range at today’s prices.

My start was $50 for a press, four or five sets of dies, a bunch of brass, and some other stuff. I bought a pound of Hodgdon’s No. 5 powder, a Lee Reloading manual, Lee powder scoops, a box of .45 ACP bullets, and 100 primers. My total investment to start was less than $200 for my first 100 rounds, giving me breakeven on my first 100 rounds.

Your mileage may vary.

I’ve been working with Ally on how to communicate in Charlie’s Voice.

What I believe I’ve found is that he starts with people that are willing to have a conversation. Usually, the people he is speaking with have the start of their conversation with Charlie written out. They expect to have a one punch knockout.

It seldom goes that way.

The reason is that Charlie starts investigating how to connect with them or to get them to articulate what they mean by the words they are using.

This allows him to use his wealth of general knowledge to have a conversation where they are working from talking points, emotions, and opinions and Charlie is working from a fact based world view.

I’m still working on this, and we are still looking from articles by you.

Not a favorite artist, but I like the message.

Train Like You Fight, Fight Like You Train

One of the consistent doctrines of the US military is the above. And we do train that way. We train to be able to overcome, to succeed.

There are countless examples of training interfering with fighting and even more of training bringing victory to the battlefield.

The first cops into the Nashville school state that their training just took over. They moved with purpose towards the sound of gunfire. They cleared rooms along the way, rapidly with no wasted motion. When they had cleared, they took the lead and did not stop until the asshole was dead.

Compared to the school cops that ran and hid in Texas and Florida. In both cases they went into barricaded with hostages, and children died because of their cowardice and lack of training.

Another example from an earlier time: when they recovered the body of a dead cop, they found six expended cases in his pocket or pouch. Why? Because he had been trained to retain his brass when he reloaded.

Why do we train with FMJ bullets?

Cartridge Pricing

These are representative prices. I’m not suggesting that these are the best prices, but they come from the same source.
Description,Quantity,Price,Per Unit

The cost of reloading a FMJ and a JHP is the same but for the cost of the bullet.

Training Costs

If you are buying your training rounds, you are going to save nearly 60 cents per round. When you are talking about 100 or more rounds in a single training session, that is a $60 difference per 100 rounds.

That means that most people will use “range candy” instead of self defense rounds.

On the other hand, the difference between firing a reloaded FMJ vs. a JHP is 6 cents per round. Even with a range session of 200 rounds going downrange, that is only a $12 difference in cost.

For me this is a non-starter. I am working through my FMJ .45 ACP but I will no longer reload FMJ pistol ammo. It isn’t worth it to me.

I have some 9mm pistols. The Blue-Haired Fairie’s is an H&K in 9mm. The PC-9 is 9mm and takes Glock magazines, so I have a Glock.

What I didn’t have was an EDC that I actually liked. I’ve never been particularly fond of the Glocks. It might be because I was driving nails with my Kimber in .45 ACP and with the H&K, but the Glock had rounds going “that-a-way” for me. I traded it for something.

My Kahr got upgraded to a Sig P938, which I like. I just don’t like shooting it. It is so small and so light that it bites when it is fired.

That Kahr and later P938 were my EDC when I needed something small for when I was in the office. I had one box of Hornady Critical Defense for it. I fired just enough of those $1-per-round personal defense rounds to know that my FMJ reloads went to the same place.

So I looked through all the cans looking for 9mm. I found over 1,000 rounds of FMJ or copper-plated rounds. No problem there. What I didn’t find was any JHP.

This led to a big thank you to my son. Yesterday he moved every single ammo can until we found the crate with 9 mm and .45 ACP bullets.

The Ammo Hunt

My ammo cans are in crates: 4 30-cal cans per crate or 3 50-cal cans per crate. I also have some simple crates that hold bullets, brass, and other fun stuff. Each can is properly labeled.

The issue was that the crate labeled “bullets” was at the bottom of the hardest-to-reach stack of crates.

My son diligently worked his way to that stack and handed them all out to me until he got to that final crate at the bottom of the final stack. Therein he located 2,000 JHP bullets, just waiting to grow into Freedom Seeds!

I see reloading time in my future. There is no need to have bullets when I have spark buttons, powder, cases, and seeds.

As we say, shooting is a perishable skill, and I need more range time.

Conclusion

What do you think is the proper ratio of seeds to seed dispensers? My current is around 1,000 seeds per caliber or 500 per dispenser, whichever is greater.

Newcommers

EDC

Every Day Carry. It can refer to a firearm, knife, light, or IFAK. The stuff that you have with you all the time.

IFAK

Individual First Aid Kit

Spark Button

Primers

Seeds

Bullets

Freedom Seeds

A complete cartridge, ready to fire.

Seed Dispenser

A firearm

You can buy cartridges, or you can make cartridges. The process of making a cartridge is called “reloading” because, in general, we are recycling the cases of fired cartridges.

A cartridge consists of a case, normally brass, sometimes aluminum or steel, which holds all the other components. A primer, which causes a spark when struck correctly by the firing pin of a firearm. A propellant called powder, gunpowder, or smokeless powder. Gunpowder is not the same as smokeless powder; using smokeless powder when the firearm expects gunpowder or black powder causes bad things to happen. And a bullet.

We use a reloading press to prepare the case and insert the primer. We add powder to the case, then place the bullet in the mouth of the case and use the press to seat the bullet. Once done, you have a cartridge that is as good as, if not better than, factory cartridges.

Communism, an idea so good it has to use force to keep its citizens in.

The fact is that Communism requires that all dissenting voices be silenced. They are silenced with violence. People are sent to re-education camps, they are disappeared in the dead of the night, and they are grabbed off the streets, never to be seen again. All with the authority and violence of the government.

The image of people being disappeared is what the anti-American agitators are using when they say that “masked men are kidnapping people off of the street.” It invokes images of communist countries doing just that.

The difference is that these people are not being disappeared. They are being arrested. They get lawyers, or they had lawyers.

When ICE agents detain a criminal illegal alien outside a courtroom, that criminal has had due process.

Are these arrests violent? Only if the person being arrested makes them that way.

Soft Violence

When the Tea Party came into existence, I wanted to fly the Gadsden flag. I did not for fear of how it might affect my wife and children. I flew an American Flag instead and still worried.

People were silenced by fear of reprisals. This was the time of cancel culture from the left.

Voice a double-bad-think opinion, and you were canceled. Voice a bad-think opinion, and you might only be warned.

You were made to feel alone, but for a few lonely voices in the dark. You were told by every talking head that you were guilty of bad-think and would be punished if found out.

This is also the screaming and in your face, just shy of physical assault.

Physical Violence

This is where you start to see the laying on of hands. Physical beatings, shoving, and punching. Actual battery.

In Aleksandr Solzhenitsyn’s famous book The Gulag Archipelago, he talks about being stuffed into a cattle car. The violence was being dealt out not only by the guards but also by fellow prisoners.

But they weren’t “fellows” in the classic sense. Most of the prisoners were regular people stuffed in the pipeline leading to the Gulag. These were criminals, and they colluded with the guards for privileges and in exchange kept the rest of the prisoners cowed.

We see that in the criminal attacks in Seattle. Roaming gangs of violent thugs impose their beliefs with truncheons and bats. And if those don’t work, even more.

In Cabaret, the maître d` of the Kit Kat Klub kicked some Brownshirts out of the club. The cowards were not willing to do anything about it at the time. Instead, they waited until the maître d` had left the club, then beat the shit out of him.

They sent the message, “Oppose us, and bad things will happen to you.”

We see this type of thuggish behavior in the cities. We see it when conservatives set up tables to have a conversation and are driven off campus by gangs of feral louts.

And the state defends these thugs. Regular folk who stand up for themselves and others are punished by the state. Look at how the authorities treated prayer groups defending themselves vs. the antifa lice that attacked them. Look at the violence against federal buildings by antifa and then what happened when grandma walked through the Capitol.

This violence is vastly asymmetric, and it is intended to be. At every step of the process, you are told to back down. You are told that you have to restrain yourself. And if you don’t? Ask Kyle what happens.

Murder

Ask Lincoln how Democrats behave when they lose. Ask JFK, killed by a communist. Ask almost any victim of political violence in the last century how Democrats behave.

Unfortunately, you can’t. Most of those people were murdered.

The goal of the left is to create an event to prove the right is the more violent.

The problem is, we are. When we react, we do it with force and with a will to get things done.

Again, ask Kyle. When it was time, he pulled the trigger, and cos-playing degenerates died. That boy showed remarkable trigger discipline.

And this is not new.

In 1977, JoAnne Deborah Byron, AKA Assata Olugbala Shakur, was convicted of murdering a police officer.

This happy little communist worm was a member of the Black Panther Party, the Black Liberation Army and other groups. She was a murderer, a terrorist, and a bank robber. After she was convicted, she was sentenced to life + 26-33 years.

Of course the standard whining leftists blamed it all on racism.

She escaped, stayed on the lam for a while before moving to Cuba to live the life of a privileged Communist.

On September 26th, 2025, she became a good communist. Cuba has reported she died of health complications.

Pistol Optics

I have red dots on some of my AR-15s. For my wife, this is great. Put the dot where she wants the hole to be, and press the trigger. A hole appears near where the little red dot was.

All of my AR-15 red dots are co-witnessed. The little red dot shows up at the same place where my iron sight points. If the optic is off, I can see my iron sights through the glass. This means that if I have issues with the red dot being “dead,” I still have functional sights.

With the peep sight of the AR-15 platform, I have a rear sight that flips down. Some of my ARs have fixed front sights, and some have flip-up front sights. Even if they are flip-up, I leave them up.

Some pistol optics are set up so that when the optics are mounted, the rear sight is removed. This means that if your optic fails, you have no sights.

I looked into a P365-RS; this is a Sig P365 with a Romeo Sierra already mounted. The sights are co-witnessed. And the sights are the standard great Sig sights.

Holsters

I have been a fan of Alien Gear holsters for several years. I like how they fit and the positive retention they provide for my firearms.

I particularly like their Shape Shift system. This system consists of different shells, which are specific to a firearm; parts to mount to the shell for different types of retention; and a backer.

For example, you can use a positive retention where you have to use your thumb to release the firearm, or it can be retained by a spring loaded catch. Pull, and the firearm comes out of the holster. The tension on that catch can be set as you want.

The tool needed to adjust the different tensions and positions is the retaining nut, which retains the mounting system.

The shell can be used as a half shell with some backers or as a full shell with others.

For me, I like being able to move the holster to different backers: an IWB, an OWB, an appendix backer (which I haven’t used), and a shoulder rig. They have drop rigs and MOLLE rigs as well. There are many options. Those backers that attach to a belt can adjust cant easily. The belt clips are solid loops, J-Hooks or C-hooks. You choose which you want.

The C-hooks fit between your belt and your pants, so the only thing that is visible are two small black lines at the top and bottom of your belt. With the solid loop and J-hooks, you can see the “strap” of black plastic, which is the clip.

I’m looking for another holster for a P365X-Macro. They aren’t selling a shell for the Shape Shift system.

Yesterday I spoke to a rep and found out why. They are phasing the Shape Shift system out. Because the shells mount to the backers at the top of the slide, there isn’t enough room for many of the pistol mounted optics. In addition, pistols with weapon lights don’t fit well either.

According to this rep, they are working on a new system.

Is It Lawfare?

The big news yesterday was that the DoJ is attacking Trump’s political enemies, attempting to jail them.

It appears that the DoJ has found that James Comey is likely guilty of multiple crimes, so he has been indicted. We can tell this is lawfare because they are using laws that have existed for many years; they presented evidence to a grand jury, and the grand jury returned the indictment.

This is totally different from charging a political enemy with fraud for putting up collateral for a loan and then paying back the loan with interest but having a difference in opinion on the value of the collateral. It is also totally different from calling the secretary of state for a state and telling him that you need 200K votes to win the state. AND it is totally different from entering into an NDA agreement with a bimbo who is attempting to blackmail you, and then paying that bill through your lawyer.

You see, this crime is real, 18 U.S.C. § 1001 making false statements to Congress. James messed up and answered directly when asked if he authorized a leak. He should have gone with that old favorite, “I don’t recall.”

While not all the grand jury agreed that he violated 18 U.S.C. § 1001, a majority did.

All the members of the grand jury did agree that he violated 18 U.S.C. 1505, obstruction. Corruptly endeavored to influence, obstruct, and impede the Senate Judiciary Committee’s inquiry by making false and misleading statements.

The case was filed in the Eastern District of Virginia, 25-cr-00272

Networking Almost Working

The wonderful thing about Unix configuration is that you can do anything you want to a configuration file. With a GUI, you can only do what the GUI allows you to do.

My internal network uses OSPF for internal networking with a bit of BGP magic for the OVN networks. Whenever a router is presented with two choices for a route, it chooses the one with the lowest cost.

It is a little more complicated than that because a router can also adjust its priority. Priority adjustments might be used when you have links that have different dollar costs but the same bandwidth.

There are three parameters that are used to calculate the cost of a link. The algorithm calculates the sum of the costs of all the links between two nodes and takes the lowest cost.

The first method and parameter is to set the cost of the link. This is a value that ranges from 1 to 65535. No calculations are done; this is the cost of using this link.

This means that you could assign different values to different link speeds: 1 for 100G, 2 for 40G, 3 for 20G, 4 for 10G, 5 for 5G, 6 for 2.5G, 7 for 1G, and so forth. This works, but it is painful.

A faster method is to assign a reference bandwidth to use within the router. For me, I set the reference bandwidth to 10G. Since the router knows the bandwidth of each interface, it takes that reference bandwidth and divides it by the interface bandwidth.

If my interface is a 10G fiber link, that is 10G/10G, or 1. But if that particular interface is a 1G Ethernet port, then we have 10G/1G equals 10.

This is a quick way to get the correct costs assigned to every interface. I only have to specify a bandwidth for an interface if that interface is a bridge.

Nice stuff.

Because I have a link that is acting up, I was able to describe that link to the protocol. That link is 10G down and 1G up. So it gets direct traffic down, but when it is sending upstream, that takes two hops.

The only issue I still need to solve is that I have an L3 Switch that is not passing OSPF multicast packets.

Question of The Week

Who else do you want to see indicted, and Why?

One of the best tools I’ve discovered in my many years of computer work is AMANDA.

AMANDA is free software for doing backups. The gist is that you have an Amanda server. On schedule, the server contacts Amanda clients to perform disk backups, sending the data back to the server. The server then sends the data to “tapes”.

What makes the backup so nice is that it is configured for how long you want to keep live backups and then attempts to do it efficiently. My backups are generally for two years.

On the front side, you define DLEs. A DLE is a host and disk or filesystem to dump. There are other parameters, but that is the smallest DLE configuration.

Before the dump starts, the server gets an estimate for each DLE based on using no other backups, a full dump, or a partial dump or multiple partial dumps. Once it obtains this information, it creates a schedule to dump all the DLEs.

The data can be encrypted on the client or the server, is transferred to the server, sometimes to a holding disk, sometimes directly to tape. I can be compressed on the server or the client.

In the end, the data is written to disk.

Every client that I have is backed up using Amanda. It just works.

In the olden days, I configured it to dump to physical tapes. If everything fits on one tape, great. If it didn’t, I could use multi tape systems or even tape libraries. The tape size limitations were removed along the way so that DLEs can be dumped across multiple tapes.

The backups are indexed, making it easy to recover particular files from any particular date.

More importantly, the instructions for recovering bare metal from backup are written to the tape.

Today, tapes are an expensive method of doing backups. It is cheaper to backup to disk, if your disks are capable of surviving multiple failures.

Old-Time Disks

You bought a disk drive; that disk drive was allocated as a file system at a particular mount point, ignoring MS DOS stuff.

Drives got bigger; we didn’t need multiple drives for our file systems. We “partitioned” our drives and treated each partition as an individual disk drive.

The problem becomes that a disk failure is catastrophic. We have data loss.

The fix is to dump each drive/partition to tape. Then if we need to replace a drive, we reload from tape.

Somebody decided it was a good idea to have digitized images. We require bigger drives. Even the biggest drives aren’t big enough.

Solution: instead of breaking one drive into partitions, we will combine multiple physical drives to create a logical drive.

In the alternative, if we have enough space on a single drive, we can use two drives to mirror each other. Then when one fails, the other can handle the entire load until a replacement can be installed.

Still need more space. We decide that a good idea is to use a Hamming code. By grouping 3 or more drives as a single logical drive, we can use one drive as a parity drive. If any drive fails, that parity drive can be used to reconstruct the contents of the missing drive. Things slow down, but it works, until you lose a second drive.

Solution: combine RAID-5 drives with mirroring. Never mind, we are now at the point where for every gigabyte of data you need 2 or more gigabytes of storage.

Enter Ceph and other things like it. Instead of building one large disk farm, we create many smaller disk farms and join them in interesting ways.

Now data is stored across multiple drives, across multiple hosts, across multiple racks, across multiple rooms, across multiple data centers.

With Ceph and enough nodes and locations, you can have complete data centers go offline and not lose a single byte of storage.

Amazon S3

This is some of the cheapest storage going. Pennies on the gigabyte. The costs come when you are making to many access requests. But for a virtual tape drive where you are only writing (free), it is a wonderful option.

You create a bucket and put objects into your bucket. Objects can be treated as (very) large tape blocks. This just works.

At one point I had over a terabyte of backups on my Amazon S3. Which was fine until I started to get real bills for that storage.

Regardless, I had switched myself and my clients to using Amazon S3 for backups.

Everything was going well until the fall of 2018. At that time I migrated a client from Ubuntu 16.04 to 18.04 and the backups stopped working.

It was still working for me, but not for them. We went back to 16.04 and continued.

20.04 gave the same results during testing; I left the backup server at 16.04.

We were slated to try 26.04 in 8 or so months.

Ceph RGW

The Ceph RGW feature set is similar to Amazon S3. It is so similar that you need to change only a few configuration parameters to switch from Amazon S3 to Ceph RGW.

With the help of Grok, I got Ceph RGW working, and the Amazon s3cmd worked perfectly.

Then I configured Amanda to use S3 style virtual tapes to my Ceph RGW storage.

It failed.

For two days I fought this thing, then with Grok’s help I got the configuration parameters working, but things still failed.

HTTP GETs were working, but PUTs were failing. Tcpdump and a bit of debugging, and I discovered that the client, Amanda, was preparing to send a PUT command but was instead sending a GET command, which failed signature tests.

Another two days before I found the problem. libcurl was upgraded going from Ubuntu 16.04 to 18.04. The new libcurl treated setting the method options differently.

Under old curl, you set the method you wanted to use to “1,” and you got a GET, PUT, POST, or HEAD. If you set GET to 0, PUT to 1, and POST/HEAD to 0, you get a PUT.

The new libcurl seems to override these settings. This means that you can have it do GET or HEAD but no other. GET is the default if everything is zero. Because of the ordering, you might get the HEAD method to work.

This issue has existed since around 2018. It is now 2025, and the fix has been presented to Amanda at least twice; I was the latest to do so. The previous was in 2024. And it still hasn’t been fixed.

I’m running my patched version, at least that seems to be working.