Skills

Features, Issues, Bugs, and Requirements

When software is upgraded or updated, it happens for a limited set of reasons. If it is a minor update, it should be for issues, bugs or requirements.

What is an Issue? An issue is something that isn’t working correctly, or isn’t working as expected. While a Bug is something that is broken, that needs to be fixed.

A bug might be closed as “working as designed,” but that same thing might still be an issue. The design is wrong.

Requirements are things that come from outside entities that must be done. The stupid warning about a site using cookies to keep track of you is an example. The site works just fine without that warning. That warning doesn’t do anything except set a flag against the cookie that it is warning you about.

But sites that expect to interact with European Union countries need to have it to avoid legal problems.

Features are additional capabilities or methods of doing things in the program/application.

Android Cast

Here is an example of something that should be easy but wasn’t. Today there is a little icon in the top right of the screen, which is the ‘cast’ button. When that button is clicked, a list of devices is provided to cast to. You select the device, and that application will cast to your remote video device.

We use this to watch movies and videos on the big screen. For people crippled with Apple devices, this is similar to AppleTV.

When this feature was first being rolled out, that cast button was not always in the upper right corner. Occasionally it was elsewhere in the user interface. Once you found it, it worked the same way.

A nice improvement might be to remember that you prefer to cast and what device you use in a particular location. Then when you pull up your movie app and press play, it automatically connects to your remote device, and the cast begins. This would be just like your phone remembering how to connect to hundreds of different WiFi networks.

If you were used to the “remember what I did last time” model and suddenly had to do it the way every other program does, you might be irritated. Understandably. Things got more difficult, two buttons to press when before it just “did the right thing.”

Upgrades and updates are often filled with these sorts of changes, driven by requirements.

Issues and Bugs

If I’m tracking a bug, I might find that the root cause can’t be fixed without changes to the user interface. I’m forced into modifying the user interface to fix a bug that had to be fixed. Sometimes making something more difficult or requiring more steps. It is a pain in the arse, but occasionally a developer doesn’t really have a choice.

An even more common change to the user interface happens when the program was allowing you to do something in a way you should not have been. When the “loophole” is fixed, things become more difficult, but not because the developer wanted to nerf the interface, but because what you were doing should not have been happening.

Finally, the user interface might require changes because a library your application is using changes and you have no choice.

The library introduced a new requirement because their update changed the API. Now your code flow has to change.

Features

This is where things get broken easily. Introducing new features.

This is the bread and butter of development agencies. By adding new features to an existing application, you can get people to pay for the upgrade or to decide on your application over some other party’s application.

Your grocery list application might be streamlined and do exactly what you want it to do. But somebody asked for the ability to print the lists, so the “print” feature was added, which brings the designers in, who update the look to better reflect what will be printed.

Suddenly your super clean application has a bit more flash and is a bit more difficult to use.

Features often require regrouping functionality. When there was just one view, it was a single button somewhere on the screen. Now that there is a printer view and a screen view, with different options, you end up with a dialog where before you had a single button press.

Other times the feature you have been using daily without complaint is one that the developer, or more likely the application owners, don’t use and don’t know that anybody else uses. Because it works, nobody was complaining. Since nobody was complaining, it had no visibility to the people planning features.

The number of times I’ve spent hours arguing with management about deleting features or changing current functionality would boggle your mind. Most people don’t even know everything their application does, or the many ways that it can be done.

David Drake’s book The Sharp End features an out-of-shape maintenance sergeant pushed into a combat role. He and his assistant have to man a tank during a mad dash to defend the capital.

At one point the sergeant is explaining how tankers learn to fight their tank in a way that works for them. The tank has many more sensors and capabilities than the tanker uses. Those features would get in the way of those tankers. It doesn’t matter. They fight their tank and win.

As the maintenance chief, he has to know every capability, every sensor, and every way they interact with each other. Not because he will be fighting the tank, but because he doesn’t know which method the tanker is going to use, so he has to make sure everything is working perfectly.

My editor of choice is Emacs. For me, this is the winning editor for code development and writing books and such. The primary reason is that my fingers never have to leave the keyboard.

I type at over 85 WPM. To move my hands from the keyboard is to slow down. I would rather not slow down.

I use the cut, copy, and paste features all the time. Mark the start, move to the end, Ctrl W to cut, Meta W to copy, move to the location to insert, and Ctrl Y to yank (paste) the content at the pointer. For non-Emacs use, Ctrl C, Ctrl X, and Ctrl V to the rescue.

My wife does not remember a single keyboard shortcut. In the 20+ years we’ve been together, I don’t think she has ever used the cut/paste shortcuts. She always uses the mouse.

All of this is to say that the search for new features will oftentimes break things you are used to.

Pretty Before Function

Finally, sometimes the designers get involved, and how things look becomes more important than how they function.

While I will not build an application without a good designer to help, they will often insist on things that look good but are not good user experiences. Then we battle it out and I win.

This isn’t a prepping post, per se. I’m off schedule due to life being busy. I’ll try and get back on track in a week or two. Be aware that most articles through the month of November will be “canned” (ie written long in advance, probably this month) because it is National Novel Writing Month and I need to sit down and write a whole-ass book (this year it’s my 18th century cookbook) in 30 days. 50,000+ words in 30 days is not easy, and I don’t do a lot of other writing, though I may pop in to say hi. We’ll see.

So last weekend, I was up at the Fort. It was the big “Out of Time” timeline event, meaning they invited people from other eras than the Fort’s (which is 1740s through 1760s, roughly) to come and set up outside the palisade and present information on their part of history. We had someone from 13th century, quite a bit from WWI and WWII, and of course my 15th century group, The Brotherhood of the Arrow and Sword. With all my favorite reenactors there (only the Vikings were missing, as they had an event elsewhere), I asked for and received permission to plan a grand meal for everyone.

As you can see from the image to the left, it was quite the feast. I had three “removes” (we would call them courses, today). We ended up actually putting all the food onto a big table and letting people get stuff buffet style, which I totally lost control of. I really got descended upon by locusts, and that was not what I had intended. Next year will be better, with the “removes” going out on the table for people to get food from. Also, those with food allergies needed to go up first, and that didn’t happen. I learn new stuff every time I do this. 🙂

The preparations for this meal started on Friday evening. The salt cod had to go in to soak, as the water needed to be changed several times before it was put in with the turnip to cook. I think I changed the water five times? Regardless, the cod was not at all salty by the time it hit the table, and actually was quite good all mashed up with the turnip. The “pumpion soup” (squash soup) was incredibly tasty and easy, and I will be doing it again. The salad was “just” salad, but looked at tasted quite good.

Yes, I cooked a turkey over a fire. This was my first time roasting a WHOLE turkey, as in the past I’ve always disjointed it and cooked it in pieces. I wanted to put the entire turkey out, though, and so I roasted it in my largest cast iron pot. I started it breast up, flipped it after 1.5 hours, then flipped it again after 1.5 hours. It spent its last hour in the beehive bake oven, crisping up its skin to a lovely brown shade. The turkey literally fell apart, was juicy and tender, and basically disappeared within a few minutes of being put out. The ham was “braised,” which means I seared all the sides first, and then boiled it. Or rather, it simmered most of the day. The result was delicious, and it was reduced to a single meatless bone and a piece of gristle before the end of the dinner. I was so pleased that everyone liked the food!

I think I served about 40 or so people. We ate in the Great Hall, which is also where I did most of the cooking. I baked bread in the outdoor beehive oven (four loaves) and did the rest of the cooking over coals in the two fireplaces at either end of the Hall. It was nice to have all that space for cooking! Of the side dishes I made, the clear favorite was the Roots a la Creme, which are basically root veggies in a cream sauce. I have to admit, they were very delicious, and I ate the leftovers (what few there were) the next day.

The star of the show, though, was definitely the Rich Cake, from Martha Washington. I have never made a cake like this before, which is dense and full of extremely alcoholic fruit. It was moist, solid, flavorful, sweet, and indeed very rich. I will absolutely be making it again, though perhaps with a few minor adjustments. I can see what that thing could be set on a shelf and left for a month, though. There’s so much alcohol and sugar in it, nothing will touch it! LOL! Think of it as an edible Christmas cake, the kind we usually use as door stoppers.

Ratafia biscuits were the other new-to-me dessert that I attempted. They did not turn out as well as I had hoped, but they were not a clear fail, either. They were very edible, with a lovely almond undertone to them. They didn’t “loft up” as much as I’d hoped, though. I think I folded in the almond flour too roughly, perhaps. I will say, I’m never making the ratafia biscuits again without modern conveniences. Whipping the egg whites until stiff BY HAND was quite the adventure, and my arm still hurts. Totally worth it, just to prove I could do it, though. It might not have been so difficult if I hadn’t started out by creaming the butter for the Rich Cake first, then creaming the sugar into the butter for that same cake, all before moving on to whipping egg whites by hand for 25 minutes. Yes, I’m nuts. But you know what? I know I can do it, now.

Here’s the thing. A lot of people there thought I was crazy for putting this dinner on. Cooking for that many mouths, spending an entire day in prepping, cooking, plating, and all the rest. I did have help, though, with two very good friends who took the time to show me the ropes (they’re both retirees from restaurant business, and know how to do proper mise en place). Everyone loved the meal, but yes, many of them thought I was insane for not putting at least some of this stuff into the very modern oven to bake. I was determined to do this “the 18th century way” though, because I know that if I can do it that way, then doing it with modern conveniences is easy.

This is also research. The cookbook I’m about to write next month will feature all of the recipes I used in this supper, along with many others that I’ve tried in the past two years. While I tailor my recipes to be cooked in a modern kitchen, the instructions suit those who want to do it the 18th century way, as well. I have to know how it works, so that I can explain it to others. Having the opportunity to work the kitchen for a big meal that way gave me a ton of insights into how an 18th century kitchen would have run. It makes my explanations better, when I’m talking to visitors at the Fort or to the encampment of my 18th century reenactors group.

Above everything, this is prepping (see how I masterfully brought it back to prepping? Go me!). I now know without a doubt that I can cook for a large group with nothing more than my two hands and fire. Nothing can stop me now! Oh, and everyone is looking forward to next year!

Costs

Like almost everything I learn, the startup costs are quickly outweighed by the cost of the “tooling.”

My lathe and milling machine cost less than $1500. The cost of my micrometers exceeds $1000 without counting toolposts, toolholders, inserts, and a wealth of other things.

Reloading is something like that. You can get a starting kit for around $175 that has just about everything you need to start reloading. You need to add dies, reloading manual, and consumables.

Assuming you are reloading for 9mm, your dies will be between $40 and a couple of hundred. I’m currently using Redding, which prices out at $75. Manuals run $25 or so. This puts the total cost of tooling just under $300.

Your consumables will be primers at $80 per 1,000, powder at $156 for four pounds, 115gr RMR JHP at $115 per 1,000.

This puts your consumables at $351 for a total start cost of $650 or so.

Assuming you are reloading “free” brass, this gives you a startup cost of $0.60/round, which is more than range candy but less than personal defense ammo.

After your initial investment, your costs will decrease, but it will take a while before you are truly saving money.

Your next 1,000 rounds will cost you $80 for primers and $115 for bullets, giving you a cost per round of $0.195. This would bring your costs from $0.60 to $0.422 per round, including your original investment.

The place where costs really change is when you need to start reloading a different caliber. Using the same equipment, you can add a set of dies, $75, and begin reloading a different caliber.

Availability

During the great panic-demic it became nearly impossible to buy certain calibers of ammunition. Even now, it is sometimes difficult to buy certain calibers.

If you have reloading equipment, those limitations are reduced.

I, personally, reload .38 Special and .357 Magnum with cast bullets. I cast HP myself and find those rounds to be consistent in weight and go where I point them.

My LGS was only selling cowboy loads and very expensive self-defense loads. They had no brass, they had no bullets, they had no primers, and their selection of powders was limited. For the cost of a set of very nice bullet molds, I was able to create freedom seeds for my R95 and revolver.

8mm Mauser is difficult to find and expensive; I’ve got nearly 500 rounds of it for the cost of bullets and powder. The same with 30-30 and 45-70.

I believe the only firearms I cannot reload for are my shotguns. Since I only run 4 rounds a year out of them, it’s not a real issue.

This is the great power of reloading, in my opinion. You can reload for unusual calibers for the cost of a set of dies and bullets.

Quality

The big-time long-distance shooters swear by hand reloading their ammo. I believe them. When I’m reloading, I attempt to maintain that same level of quality as they do. I may or may not succeed, but it is nice to know that I’m close.

Safety

There are two big rules in reloading:

1. Don’t Shoot Other People’s Reload.
2. Your mistake can kill you.

I currently use a Franklin Armory hand primer. It works for me. I can feel each primer seat properly. Before I had the hand primer, I used two different on-press tools.

One day, I was at a private range shooting .45 Colt out of a Marlin lever action. I would pull the trigger, the hammer would fall, and there would be no bang. Not even a pop. This was a problem, but I found nothing horribly wrong. It just meant I had to fix something.

Then I switched to my 1911. Bang. Bang. Pop. STOP!

I had a squib.

This led to me pulling every bullet on that set of reloaded rounds. Luckily there were only 20 of each caliber.

Conclusion

The startup cost of reloading is not horrible. The cost grows like in every hobby. Your break-even point will be in the 4,000 round range at today’s prices.

My start was $50 for a press, four or five sets of dies, a bunch of brass, and some other stuff. I bought a pound of Hodgdon’s No. 5 powder, a Lee Reloading manual, Lee powder scoops, a box of .45 ACP bullets, and 100 primers. My total investment to start was less than $200 for my first 100 rounds, giving me breakeven on my first 100 rounds.

Your mileage may vary.

Yes, I’m a day late. I unloaded the truck Sunday, fell down, went boom. Yesterday was a lot of cleaning and recovery. So we’re a day late.

This weekend, I had the most amazing time cooking an 18th century supper for about 40 or so reenactors. I had a blast, even though it was a ton of work. We started cooking at about 9:30am, and rang the dinner bell at just after 6pm. By 8pm, I was in bed, in the dark, half asleep. LOL… But what a day! Everyone loved the food, and I made both a 13 lb turkey and an 8 lb ham, both of which were stripped like locusts in a wheat field. LOL… There were plenty of successes, and a couple of partial failures, but overall I did well. People enjoyed it, and I’ve been asked if I’m willing to repeat it next year (spoiler alert: I am!). Of the side dishes I presented, though, Roots a la Creme was probably the group favorite. It just tasted GOOD. So here is the version I made:

Ingredients:

• 4 large carrots
• 4 medium parsnips
• 1 stick of butter/margarine
• 1 tbsp fresh parsley, minced
• 3 small scallions
• 1 clove of garlic
• 1 shallot about one inch round
• A small pinch of ground cloves
• 1/4 tsp dried basil
• 1 tbsp butter
• 1 tbsp or so of flour for thickening
• 1/4 tsp salt
• 1 tsp ground pepper
• 1 cup cashew yogurt OR sour cream OR heavy whipping cream
• 1 tsp white vinegar
• vegetable broth, if needed to thin it out a bit.

Peel your carrots and parsnips and cut them in large slices. Boil them until you can stick a fork in them easily (usually 15 to 20 minutes). Drain and transfer them to a stew pan. Add the butter, parsley, scallions, garlic, shallots, cloves and basil. Over a medium heat stir well to blend the butter and seasonings with the roots.

Add flour, salt pepper and broth. Boil quickly, stirring as you do, until it thickens to a sauce. Remove from heat and reduce your heat to medium. Add the yogurt to the roots. Stir over medium heat until well blended and smooth. Take care not to boil or scorch this. If your sauce breaks, take a few tablespoons out, add a bit of white vinegar to it, and whisk well. Slowly add that back to the sauce, and it should fix it. You can add broth if it’s too thick.

Serve.

Train Like You Fight, Fight Like You Train

One of the consistent doctrines of the US military is the above. And we do train that way. We train to be able to overcome, to succeed.

There are countless examples of training interfering with fighting and even more of training bringing victory to the battlefield.

The first cops into the Nashville school state that their training just took over. They moved with purpose towards the sound of gunfire. They cleared rooms along the way, rapidly with no wasted motion. When they had cleared, they took the lead and did not stop until the asshole was dead.

Compared to the school cops that ran and hid in Texas and Florida. In both cases they went into barricaded with hostages, and children died because of their cowardice and lack of training.

Another example from an earlier time: when they recovered the body of a dead cop, they found six expended cases in his pocket or pouch. Why? Because he had been trained to retain his brass when he reloaded.

Why do we train with FMJ bullets?

Cartridge Pricing

These are representative prices. I’m not suggesting that these are the best prices, but they come from the same source.
Description,Quantity,Price,Per Unit

The cost of reloading a FMJ and a JHP is the same but for the cost of the bullet.

Training Costs

If you are buying your training rounds, you are going to save nearly 60 cents per round. When you are talking about 100 or more rounds in a single training session, that is a $60 difference per 100 rounds.

That means that most people will use “range candy” instead of self defense rounds.

On the other hand, the difference between firing a reloaded FMJ vs. a JHP is 6 cents per round. Even with a range session of 200 rounds going downrange, that is only a $12 difference in cost.

For me this is a non-starter. I am working through my FMJ .45 ACP but I will no longer reload FMJ pistol ammo. It isn’t worth it to me.

I have some 9mm pistols. The Blue-Haired Fairie’s is an H&K in 9mm. The PC-9 is 9mm and takes Glock magazines, so I have a Glock.

What I didn’t have was an EDC that I actually liked. I’ve never been particularly fond of the Glocks. It might be because I was driving nails with my Kimber in .45 ACP and with the H&K, but the Glock had rounds going “that-a-way” for me. I traded it for something.

My Kahr got upgraded to a Sig P938, which I like. I just don’t like shooting it. It is so small and so light that it bites when it is fired.

That Kahr and later P938 were my EDC when I needed something small for when I was in the office. I had one box of Hornady Critical Defense for it. I fired just enough of those $1-per-round personal defense rounds to know that my FMJ reloads went to the same place.

So I looked through all the cans looking for 9mm. I found over 1,000 rounds of FMJ or copper-plated rounds. No problem there. What I didn’t find was any JHP.

This led to a big thank you to my son. Yesterday he moved every single ammo can until we found the crate with 9 mm and .45 ACP bullets.

The Ammo Hunt

My ammo cans are in crates: 4 30-cal cans per crate or 3 50-cal cans per crate. I also have some simple crates that hold bullets, brass, and other fun stuff. Each can is properly labeled.

The issue was that the crate labeled “bullets” was at the bottom of the hardest-to-reach stack of crates.

My son diligently worked his way to that stack and handed them all out to me until he got to that final crate at the bottom of the final stack. Therein he located 2,000 JHP bullets, just waiting to grow into Freedom Seeds!

I see reloading time in my future. There is no need to have bullets when I have spark buttons, powder, cases, and seeds.

As we say, shooting is a perishable skill, and I need more range time.

Conclusion

What do you think is the proper ratio of seeds to seed dispensers? My current is around 1,000 seeds per caliber or 500 per dispenser, whichever is greater.

Newcommers

EDC

Every Day Carry. It can refer to a firearm, knife, light, or IFAK. The stuff that you have with you all the time.

IFAK

Individual First Aid Kit

Spark Button

Primers

Seeds

Bullets

Freedom Seeds

A complete cartridge, ready to fire.

Seed Dispenser

A firearm

You can buy cartridges, or you can make cartridges. The process of making a cartridge is called “reloading” because, in general, we are recycling the cases of fired cartridges.

A cartridge consists of a case, normally brass, sometimes aluminum or steel, which holds all the other components. A primer, which causes a spark when struck correctly by the firing pin of a firearm. A propellant called powder, gunpowder, or smokeless powder. Gunpowder is not the same as smokeless powder; using smokeless powder when the firearm expects gunpowder or black powder causes bad things to happen. And a bullet.

We use a reloading press to prepare the case and insert the primer. We add powder to the case, then place the bullet in the mouth of the case and use the press to seat the bullet. Once done, you have a cartridge that is as good as, if not better than, factory cartridges.

“A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship, design a building, write a sonnet, balance accounts, build a wall, set a bone, comfort the dying, take orders, give orders, cooperate, act alone, solve equations, analyze a new problem, pitch manure, program a computer, cook a tasty meal, fight efficiently, die gallantly. Specialization is for insects.” ― Robert A. Heinlein

I’ve probably used this quote before on Vine, and I have purposefully used it elsewhere many times. I use it as a checklist of things to know how to do, to teach my friends and family, to help my children learn about. More than just a checklist, though, it is a treatise on logic. This is the first of two posts on the list of things Heinlein expects of a human being.

Change a diaper. This one relates to family, and to chosen family and friends. You need to know enough childcare that you can change a diaper. More than that, you need to know that a man spending time with his children is not babysitting them, because they are his children. He is parenting, not babysitting. Same with mom. You need to know how you’ll react if a kid sasses you (because it’ll happen, trust me), what to do if one escapes your grasp and runs pell mell for the roadway, and what medications everyone in your intimate personal circle is taking in case you need to tell a paramedic or a doctor.

Plan an invasion. You need to understand the basics of chess and other strategy games. You may not be the best strategist out there, but you need to know how to fight back if there’s a problem. This is literally why we have a Constitution and a Bill of Rights. This is why free speech and gun rights are at the top of the list of protected God-given rights. All the weaponry in the world does you no good if you have no idea how to use it. All the weaponry int he world does you no good if you’re not WILLING to use it, if you have to. Planning an invasion means you have to know all sorts of things about guard rotation, building maps, routes in and out, security, physical barriers, targets, and a whole lot more. You have to be able to think your way through it, even if your plan is simplistic.

Butcher a hog. You must understand where your food comes from. Vegetables come out of the ground, and most of the ones we grow today require a lot of care. Animals have to be cared for, occasionally medicated, fed, watered, exercised, and loved in order to make them good food. They also have to have a clean death. And when it comes time to do the actual butchering, you don’t need to know special cuts and such, or be the most efficient, but you have to be willing to get your hands dirty and dig in as needed. You have to know that, at the point of butchering, the animal no longer cares what you do to its carcass, and your job is to make the most out of that animal’s sacrifice to feed your family.

Read More

One of the best tools I’ve discovered in my many years of computer work is AMANDA.

AMANDA is free software for doing backups. The gist is that you have an Amanda server. On schedule, the server contacts Amanda clients to perform disk backups, sending the data back to the server. The server then sends the data to “tapes”.

What makes the backup so nice is that it is configured for how long you want to keep live backups and then attempts to do it efficiently. My backups are generally for two years.

On the front side, you define DLEs. A DLE is a host and disk or filesystem to dump. There are other parameters, but that is the smallest DLE configuration.

Before the dump starts, the server gets an estimate for each DLE based on using no other backups, a full dump, or a partial dump or multiple partial dumps. Once it obtains this information, it creates a schedule to dump all the DLEs.

The data can be encrypted on the client or the server, is transferred to the server, sometimes to a holding disk, sometimes directly to tape. I can be compressed on the server or the client.

In the end, the data is written to disk.

Every client that I have is backed up using Amanda. It just works.

In the olden days, I configured it to dump to physical tapes. If everything fits on one tape, great. If it didn’t, I could use multi tape systems or even tape libraries. The tape size limitations were removed along the way so that DLEs can be dumped across multiple tapes.

The backups are indexed, making it easy to recover particular files from any particular date.

More importantly, the instructions for recovering bare metal from backup are written to the tape.

Today, tapes are an expensive method of doing backups. It is cheaper to backup to disk, if your disks are capable of surviving multiple failures.

Old-Time Disks

You bought a disk drive; that disk drive was allocated as a file system at a particular mount point, ignoring MS DOS stuff.

Drives got bigger; we didn’t need multiple drives for our file systems. We “partitioned” our drives and treated each partition as an individual disk drive.

The problem becomes that a disk failure is catastrophic. We have data loss.

The fix is to dump each drive/partition to tape. Then if we need to replace a drive, we reload from tape.

Somebody decided it was a good idea to have digitized images. We require bigger drives. Even the biggest drives aren’t big enough.

Solution: instead of breaking one drive into partitions, we will combine multiple physical drives to create a logical drive.

In the alternative, if we have enough space on a single drive, we can use two drives to mirror each other. Then when one fails, the other can handle the entire load until a replacement can be installed.

Still need more space. We decide that a good idea is to use a Hamming code. By grouping 3 or more drives as a single logical drive, we can use one drive as a parity drive. If any drive fails, that parity drive can be used to reconstruct the contents of the missing drive. Things slow down, but it works, until you lose a second drive.

Solution: combine RAID-5 drives with mirroring. Never mind, we are now at the point where for every gigabyte of data you need 2 or more gigabytes of storage.

Enter Ceph and other things like it. Instead of building one large disk farm, we create many smaller disk farms and join them in interesting ways.

Now data is stored across multiple drives, across multiple hosts, across multiple racks, across multiple rooms, across multiple data centers.

With Ceph and enough nodes and locations, you can have complete data centers go offline and not lose a single byte of storage.

Amazon S3

This is some of the cheapest storage going. Pennies on the gigabyte. The costs come when you are making to many access requests. But for a virtual tape drive where you are only writing (free), it is a wonderful option.

You create a bucket and put objects into your bucket. Objects can be treated as (very) large tape blocks. This just works.

At one point I had over a terabyte of backups on my Amazon S3. Which was fine until I started to get real bills for that storage.

Regardless, I had switched myself and my clients to using Amazon S3 for backups.

Everything was going well until the fall of 2018. At that time I migrated a client from Ubuntu 16.04 to 18.04 and the backups stopped working.

It was still working for me, but not for them. We went back to 16.04 and continued.

20.04 gave the same results during testing; I left the backup server at 16.04.

We were slated to try 26.04 in 8 or so months.

Ceph RGW

The Ceph RGW feature set is similar to Amazon S3. It is so similar that you need to change only a few configuration parameters to switch from Amazon S3 to Ceph RGW.

With the help of Grok, I got Ceph RGW working, and the Amazon s3cmd worked perfectly.

Then I configured Amanda to use S3 style virtual tapes to my Ceph RGW storage.

It failed.

For two days I fought this thing, then with Grok’s help I got the configuration parameters working, but things still failed.

HTTP GETs were working, but PUTs were failing. Tcpdump and a bit of debugging, and I discovered that the client, Amanda, was preparing to send a PUT command but was instead sending a GET command, which failed signature tests.

Another two days before I found the problem. libcurl was upgraded going from Ubuntu 16.04 to 18.04. The new libcurl treated setting the method options differently.

Under old curl, you set the method you wanted to use to “1,” and you got a GET, PUT, POST, or HEAD. If you set GET to 0, PUT to 1, and POST/HEAD to 0, you get a PUT.

The new libcurl seems to override these settings. This means that you can have it do GET or HEAD but no other. GET is the default if everything is zero. Because of the ordering, you might get the HEAD method to work.

This issue has existed since around 2018. It is now 2025, and the fix has been presented to Amanda at least twice; I was the latest to do so. The previous was in 2024. And it still hasn’t been fixed.

I’m running my patched version, at least that seems to be working.

I love tomato soup. I love roasted tomato soup and plain tomato soup. I love it thick and thin, and with a variety of toppings and additions. This is my newest version of tomato soup and I’ve fallen in love with it.

Ingredients:

• 8 oz bread, cut into 1″ squares
• 2 tbsp olive oil
• ¾ tsp oregano
• ¼ tsp garlic powder
• salt, to taste
• 1 tbsp olive oil
• 1 small yellow onion, chopped
• 3 cloves garlic, minced
• ¼ cup tomato paste
• 1 (28-oz) can whole peeled tomatoes
• 2 cups chicken or vegetable broth
• 2 (15-oz) cans cannellini beans, rinsed, drained, divided
• 1 tsp oregano
• pinch of crushed red pepper flakes
• salt and fresh ground pepper to taste
• 3 cups packed baby spinach
• ½ cup grated Parmesan, plus more for serving
• ½ cup heavy cream

Preheat oven to 350°F. On a baking sheet, toss bread, oil, oregano, garlic powder, and a large pinch of salt. Spread in an even layer. Bake the croutons, shaking the pan halfway through, until golden and crispy, 15 to 18 minutes.

In a large pot over medium heat, bring the oil up to temperature. Add in the onion and cook, stirring until softened, about 8 minutes. Add the garlic and continue to cook, stirring, until fragrant, about 1 minute more. Add in the tomato paste and cook, stirring, until the onions are coated and the paste slightly darkens, about 2 minutes.

Stir in the tomatoes, broth, and 1 cup of beans until combined. Add oregano and red pepper flakes; season with salt and black pepper. Bring to a boil, then reduce heat to medium-low and bring to a simmer. Cook, stirring occasionally, until slightly reduced and flavors have melded, about 20 minutes.

Remove the pot from the heat and carefully puree with an immersion blender until smooth. Alternatively, transfer soup to a standard blender and blend, stopping to allow steam to escape very carefully every 10 seconds, until smooth. Return the soup to medium-low heat. Add in the spinach, Parmesan, cream, and remaining beans. Bring to a simmer and cook, stirring occasionally, until spinach is wilted and cheese is melted and incorporated, about 10 minutes more.

Divide the soup among bowls. Top with croutons and more Parmesan.

Notes:

To make this vegan, use vegetable broth. Add most of the beans to the soup and blend them in, and only add a few unblended ones for mouth feel at the end. Cannellini beans make the soup much more creamy feeling. Instead of a half cup of heavy cream, sub in a half to a cup of Forager plain unsweetened yogurt. This will give the creaminess the soup demands, without watering it down or having to resort to using a roux to thicken it (undesirable with this type of soup). Instead of actual Parmesan, use one of the vegan options such as Violife brand.

This tastes incredible when served with grilled cheese (or vegan grilled “cheeze”) sandwiches!