Server room data center with rows of server racks. 3d illustration

Working Network?

We’ll be back to regular postings tomorrow. Tuesday Tunes, then “Envy” and “Show Me Your Papers.”

In the best of all worlds, every server would have two interfaces (NIC) with two ports cross-connected to two switches.

Each switch would be cross-connected to their upstream switch and so forth until you reach your redundant gateways.

The problem with this setup is that you can end up with loops that will take that part of the network down.

This means that switches need to be smart enough to keep that from happening. And you have to configure your network to allow for all those cross connections.

This means that a room will have three subnets. Primary 1, Primary 2, and Management. The switch needs to have routing capability. Then you use a routing protocol like OSPF to make magic happen.

OSPF sends out routing information often. Timers are measured in seconds, not minutes. This allows every router/switch to pick the best path at that instant.

Last week I determined that there was a problem with uplink speeds to the network. I was getting maybe a hundredth of what I should have been observing. This became obvious when I was attempting some bandwidth tests of client configurations.

So we start the process of elimination. The first thing to eliminate is the provider network. They could not provide a remote speed test because I was using my router; therefore, it required a technician to be dispatched.

The goal was to eliminate any questions regarding their side of the “demarc,” or demarcation.

The demarc is the separation of responsibility between the provider and the customer. Everything on their side of the demarc is their responsibility; everything on my side is my responsibility.

In my installation the demarc is the Optical Network Terminator (ONT). The ONT is a fiber modem; it’s not really, but it works for our purposes.

The provider prefers to provide the ONT, the router/access point/switch/VoIP gizmo. If they provide that gizmo, they move the demarc to that gizmo.

My issue is that the gizmo in question is always home quality, never better. My router is just a router. It has two 2.5 Gbit Ethernet ports and two 10 Gbit SFP ports. It has a 60 Gbit internal bus and provides DHCP, DNS, Proxy, port forwarding, and many other professional features. My access points are on a controller that monitors the power usage and adjusts the power of each access point to produce the best coverage. This allows me to have multiple Wireless Access Points (WAP) within the house and grounds that don’t interfere with each other.

Better yet, as you move from place to place, your device will seamlessly transfer to different WAPs as needed, without getting new IP addresses.

Regardless, my equipment is much better than what they provide.

We tested from the ONT and got good upload and download speeds. A step in the correct direction.

The next test was from the router. This gave me acceptable speeds, much better than 0.260 Gbit.

This left SFP modules, SFP sockets, and fiber. Being lazy, I start with SFP modules. Switching modules doesn’t help. Next I swapped the cables. Finally, I swapped the router ports.

Nothing fixed it. I currently believe that the SFP port/slot on the router has an issue.

Since I had a second switch, I could have used that; instead, I decided on a network upgrade.

The new switch was configured. It was attached to the second SFP port on the router. All the management networks and the DMZ network were attached to it. Removed from primary (router1). Then router2 was cross connected with router1 via a Direct Attach Connector (DAC). A DAC is a wire with module connectors on both ends. They are cheaper than fiber plus two modules and will run at very high speeds. They are the connector of choice for SFP to SFP within a meter or so with no sharp bends.

It took way too much time to get everything configured correctly. Mostly because I just didn’t have all the pieces correctly configured.

My network map was wrong. I had 192.168.99.x/24 allocated to P2P connections. It isn’t allocated to P2P. It is allocated to a remote subnet that is part of the local network via a VPN. Oops!

One that was taken care of, I had working connections between the gateway and both routers. But the routers would not talk to each other.

Turns out that I had the DAC plugged into the wrong slot. Once I had it in the correct slot with the correct media type, everything just started working.

I now have a cleaner network, with more options, and better bandwidth than ever before. It is working as I anticipated. It just took way too long to accomplish.

For Sale Real Estate Sign in Front of New House.

You’re Greedy For Owning A House!

I ran into this tripe a little more than a year ago. This week it is showing up everywhere.

People are whining on X and Reddit about how hard it is today. How the “boomers” took all the wealth. How they don’t want you to be able to afford a home.

My parents grew up during the Great Depression and WWII. My maternal grandfather tried to volunteer for the army but was not allowed, as he was a critical worker in railroads. My paternal grandfather was busy being a scientist for Goodyear. Family history says that he helped develop the tires that were using synthetic rubber.

My grandparents worked hard to earn what they held. They died owning a houses.

My maternal grandparents bought their house in the 30s or 40s. Grandpa built the garage himself. He did all the work around the house. Before he started working for the railroad, the SooLine, he was a machinist and a carpenter/woodworker. His hands were calloused from working his entire life.

He bought his house for under $2k, it is currently valued at $145k.

My grandfather also worked most of his life. He was a scientist who worked with cotton. His house was worth much more when my grandmother passed because of California. Likely near enough to a million dollars for a little house.

When my parents started, Dad was an ensign in the Navy. It wasn’t until a LT that they could afford their first house.

They purchased an OLD house in Norfolk. Dad took the test to be a certified electrician because he needed to rewire that house. They couldn’t afford to have somebody else do any work. Dad fixed that house up.

When he was transferred, they couldn’t sell the house for what they had in it, so they rented it out until they could. That allowed them to buy another house. A little nicer, a little better.

It wasn’t until I was in 6th grade that I had a room to myself. There was no “spare” bedroom. It wasn’t until high school that the house was big enough for there to be a shared office for Mom and Dad.

That house now lists for $550k. I do not know how much they purchased it for.

When they passed, their house was worth around $360k.

Their houses and their wealth went up as they invested money they had earned and saved.

My first house was a “three”-bedroom, one-bath house. With 870 sq. feet. I could barely afford it with my $35k/year salary. I think we paid around $50k for it. It now sells for $242k. I quote three because it was actually two bedrooms with an extra room tacked on the back, through the second bedroom. So it was three bedrooms and a laundry room, craft room, kids room.

I stuffed 5 kids and a wife into that house, and we made it work.

But here’s the thing: the house I thought I could afford was a $30k fixer-upper. I was going into that house knowing I would have to rip up every floor and put new floors back in. That every wall had to be stripped and painted. And likely, I would have to redo the roof. And I expected to do all the work myself.

I made no money on that house. The bank repossessed it because I was unable to make child support payments AND house payments.

I have a house today because my wife makes good money and I get bursts of money. We were able to afford it only because my parents helped and the house had been foreclosed.

It is worth almost 4 times what we paid for it. Not a bad investment.

It took me 30 years of work to be able to buy this house. I’ve never gone hungry, but I’ve eaten freezer scrapings more than once.

But today I’m told I had it easy. That my parents had it easy. That I’m greedy because I bought this house that could have been used by a large growing family,

One person complained that he couldn’t find a starter home. His definition of a starter home put him at $500K.

My house is not a “starter home.” It is my grow-old home. And it isn’t $500k.

There are 7 houses for sale within 2 miles of my town for less than $150k.

The most significant difference, in my opinion, is what luxuries we “must” have today.

My child dropped her phone in the lake. She has a new phone. My phone bill, for the family, is $250 per month. We shouldn’t be paying that much for phones. But I like having my phone. I like being able to read books in the dark. I like my Google Maps and Android Auto.

So I pay for my kids and family to have cell phones. And good ones.

I think we spend nearly $100/month on streaming services. I have not added it up, but that sounds about right.

There is money for servers, internet, VoIP service, and a dozen other things.

Our electric bill is high. People run heaters when it is cold and fans or AC when it is hot.

All these things add up.

When I was young, going out to eat was a special thing. Today, my kids eat out 4 out of 7 days of the week. The amount of money spent on drinks from Dunkin or McDonald’s blows my mind.

I feel bad for spending $35/month on good coffee. My wife will spend $2 for an iced tea from McDonald’s multiple times per month. It all just adds up.

To put it in perspective, I’ve seen my wife order takeout for us and the kids and spend nearly 10% of a mortgage payment. Taking the entire family out for dinner can easily hit 20%.

Do I feel bad for the people who are struggling to make ends meet? Yes, I do. I’ve been there. I fought through that. I drive a 15 year-old truck so I don’t have to pay $500/month on a car payment. Are they making the same types of sacrifices? Do they make the sacrifices my parents made?

Now they are not.

Stop whining, stop blaming me, get up and go do.

Network Maps

There was a time when I would stand up at a whiteboard and sketch an entire campus network from memory, including every network subnet, router, and switch.

Today, not only can I no longer hold all of that in my head, my whiteboards no longer exist.

In the first office I rented, I installed floor-to-ceiling whiteboards on all walls. I could write or draw on any surface.

I can remember walking into Max’s office with an idea, asking for permission to erase his whiteboard, and then drawing out or describing the idea or project. Maybe 30 minutes of drawing and discussing.

What surprised me was asking to erase my chicken scratches months later and being told, “No,” because they were still using it.

Regardless, today I need to draw serious network maps.

I have multiple routers between multiple subnets. Managed and unmanaged switches. Gateways and VPNs. I have an entire virtual network layered over the top of all of that to make different services appear to be on the same subnet.

Not to mention the virtual private cloud(s) that I run, the internal, non-routing networks.

It is just to much for me to do in my head.

Oh, here’s one that’s currently messing with me. I have a VPC. It has multiple gateways allowing access residing on different chassis in different subnets. I can’t figure out how to make it work today. Even though it was working yesterday.

I’ll be messing with networks for the next week to get things stabalized.

Prepping – Skills Tree

Everyone who wants to survive upcoming emergencies (long or short) needs to have a skills tree. This is a list of skills that are in your household, as well as any that are overlapped. All good prepper families have lots of overlap, because we know that if one person succumbs (to a virus, to a gunshot wound, to dropping a tree on themselves…), someone else has to take up the slack. When you make your skills tree, be sure to list EVERYTHING that people can do, because you never know what’s going to be needed. If you find holes, you need to figure out a way to fill them. That means bringing someone into your plans, training yourself or someone else up in the missing skill, or figuring out ways to not need it. But plans need to be made.

Basic Skills

These are skills everyone should know. If someone in your group doesn’t know these, educate them, and fast.

  • cooking over a fire
  • gardening (basic, ie you can identify a pea and know how to operate a watering can, etc)
  • sewing (basic, ie you can sew on a button or patch something roughly)
  • triage (everyone should know how to tell how serious an injury is, even if they can’t treat it)
  • shelter building
  • filtering water to make it potable
  • building a fire
  • basic first aid (specifically, treating gunshot wounds, burns, and breaks, because they’re the most likely injuries you’ll encounter)
  • self defence (pick your weapon)
  • basic strategy
  • how to wash clothes with no power
  • where to find basic vitamins (ie you can drink pine needle tea to get Vitamin C)
  • how to find dry firewood
  • how to go to the bathroom when you have no indoor plumbing
  • how to care for a newborn and its mother
  • how to read maps (both standard ones and topographical)
  • how to use a compass
  • general problem solving/logic skills
  • record keeping (write down what you do and how it’s done, for future generations)

Read More

Friday feedback banner, a man with a phone writing reviews

Friday Feedback

Can I call Network Support, Please?

I’m in network hell again. The internal network keeps getting better, but I’m a cheap SOB, so I don’t buy Cisco $50 SFP modules. I buy 4 for $50.

This has downsides, one of which is when it breaks, I’m responsible.

So here’s the story in short. Node 3 <=> SW1, Node 4 <-> SW1, S1 <=> R1, R1 <=> N129, R1 <=> N5, R1 <-> GW, GW <-> THE WORLD.

I have measured bandwidth between N3 and N5 at nearly 10G, as expected, both directions. I have measured bandwidth of nearly 2G between N129 and N3 and N5, as expected in both directions. I have measured bandwidth between N4 and N3, N5 and N129 at 1G, bi-directional.

The measured speed from the WORLD to the ONT is 1 Gbit, bi-directional. The measured speed between GW and WORLD is 480 Mbit bi-directional. This is because the GW is CPU starved at that point. It is a router, not a compute engine.

The measured throughput from the GW to N129 and N5 is 1 Gbit, as expected.

The measured throughput from N129 and N5 to the GW is < 30 Mbit. This is messed up. I'm working backwards. ONT to GW replaced ethernet cable. GW to R1 I've replaced the fiber modules on both ends, next step is to replace the fiber itself. That's where I'm stuck. I've even power cycled R1.

Techs vs Tech Support

The Fidium/Consolidated Communications tech was on site yesterday. He quickly found the ONT. Measured its performance, declared it needed to be replaced. Took a look at the equipment he could see and the fiber runs, went and moved the house to a different splitter. We are now on the primary splitter instead of one a few steps down. There are only 4 drops on this splitter instead of 10+.

When he was finished, I ran my speed tests again. 1G down, 0.040G Up. A major improvement, still not good.

Here are some observations. English was his primary and likely his only language. He understood that he was speaking with somebody who knew networks.

After we decided that something else was wrong, we decided to test with his laptop. Before he did any testing, he tested his dongle. He made sure his laptop was capable of 1G testing. That dongle was not. He went back to his truck to get one that did support 1G.

We tested, and he was getting the same numbers I was.

I’ve ordered a USB-C to Ethernet dongle for the laptop so I can connect directly to the GW to do testing. That’s a different question.

He called tech support. Those people, tech support for the techs, could not handle him just plugging in his laptop to test.

I introduced the tech to 8.8.8.8 and showed him how to verify he was online.

Because Tech Support couldn’t figure it out, he had to install a company router for them to be able to test. Tech Support then tried to force me to use their router. Their router has no SFP ports. No, thank you.

New Client

The new client project is winding down. After their people said it couldn’t be done for over a month, I do have the new server running. Management and their customers are happy. Most of the people using the new server are happy.

Their IT people are not happy with me. He managed to make enough visible mistakes that management noticed. Not that they hadn’t noticed before, they just didn’t know what they didn’t know.

Now comes the difficult task for me: turning a one-time project into long-term client support.

TDS

I look at the world of today versus the world of 12 months ago, and my life is better. I might not be pulling as much money as I want, but the economy is moving in the correct direction.

If I see a headline with “Trump!” in it, I know another TDS sufferer is about to tell me why their life is horrible because of Trump.

People being told to buy soda and snacks with their money and not mine? That’s evil.

The federal government not paying for people’s solar systems with my money? That’s evil too.

Removing criminal aliens from our country? Think of the children!

There was a thread on Reddit complaining that solar is going to die in Vermont because the solar subsidy is being stopped.

I’m sorry, solar in the Northeast is not as viable as solar in Florida or other sunny places. If you think it is such a great deal, buy it with your money, not mine.

Go read Dive Medic’s blog over at https://areaocho.com for his personal experience with solar power. It is a win for him. It is a nice concept for me. But it requires doing the math. And using your money.

The National Guard Can Be Used for Law Enforcement IF

It is for a good cause, like protecting President Biden from those horrible, evil, red-hatted people. They should have been called in to stop the most horrific, violent, attack on the government of the United States ever. January 6th, when 10s of thousands of the most heavily armed people in the world, left all their guns at home.

Activating the National Guard to help law enforcement officials in the District of Columbia? That’s forbidden. That’s misuse of the military.

Remember, the Second Amendment only protects the rights of the Militia, which is the National Guard, until Trump uses them, in which case they are the Military and it is a violation of the Posse Comitatus Act. (End sarcasm).

There is nothing that the Trump administration can do that the left won’t cry about.

It’s Not Sharp

I think I’ve mastered the art of woodworking with hand tools. All cutting implements are dull, by definition. Sharpen them before you start.

Seriously, almost every time I’ve had issues, it is because I have not done enough sharpening or flattening.

That includes such things as flattening your planes. I have a Stanley #4 that needs more work to make its sole flat. It is almost there, but I got tired of trying to make it flat. I’m in a position to work on it again.

I picked up a wooden fore plane. This is really a jack plane, but it works.

The first step was to examine it. The sole looked ok, until I realized that there were little drips from whatever was used to finish it.

It took a good 30 minutes on 320 grit sandpaper to make the sole flat. It took longer than that to grind a good edge on the iron. It was so bad that the iron got to hot to hold, even with my slow speed grinder.

I need to spend a bit more time, but I believe this will be a real workhorse.

Now all I need is a day that isn’t so hot and muggy.

The same with the 26″ handsaw I found. Once it is properly sharpened, its 8 TPI will just rip through wood. The long length of the panel will make it another workhorse.

All I need to do is finish my Jointer’s Box. OH, I realized I had been staring at a Jointer’s box at the Fort without seeing it. I have to spend the time to examine it more closely.

Question of the week

My standard in interacting with the police is “I don’t answer questions.” Which I fail at, so I work harder at succeeding.

If police were to roll up in my yard, I would be telling them to leave.

With the advent of federal law enforcement hitting the streets of DC, there are reports coming out that LEOs are walking up on people sitting on their stoops to demand ID and justification.

Where do you stand on the idea of the cops demanding ID of people sitting on the steps of their own homes?

Fiber optic technician performs repairs on cabinet to restore network connection to customers impacted by failure

Consolidated Communications, outsourced Customer Service

Two years ago I switched from the worst Internet provider to the second worst Internet provider in my area. Fidium by Consolidated Communications.

Until yesterday, when I called, I got an American-based tech support person. They were all at what I consider tier II or better. Their knowledge base was good, and they treated me with respect.

Yesterday Ally pointed out that my voice communications had gotten bad. My new client is complaining about upload speeds to their server. I’m seeing 28 KB/s upload speeds.

Before I go yelling at the client about their network, I verify my own.

Download speed: 1095.4 Mbps down. This is precisely what I pay for. 0.260 Mbps up. This is not the 1 Gbit I am paying for.

To put this into units that highlight how bad this is. I pay for 1.000000 Gigabits up and down. I got 1.09500 down and 0.000260 up.

This is an issue that needs to be resolved. I power cycle the firewall and the ONT. No change.

I call Fidium via VoIP; the representative, speaking with a strong accent, can’t hear or understand me.

I hang up, take my cell off WiFi calling, and call, making my way through the prompts to reach customer support again.

The representative I reach is also not in the US. I authenticate to her. Tell her that I have asymmetrical speeds. Give her the download and upload speeds.

“That’s the plan you’re paying for.”

“No, it is not. I’m paying for 1G up/down symetrical.”

“You called technical support. You need to talk to sales.”

“I am only getting 0.26 megabits up. This is not my plan.”

“You need to talk to sales.”

“Let me talk to somebody competent!”

More arguments from her that I’m getting exactly what I am supposed to be getting

“Let me change that, let me talk to your supervisor now.”

10 plus minutes of waiting before she comes back on and asks, “What are the speeds you are getting again?”

I tell her, and she finally starts to work towards a resolution.

I ask her where is the supervisor that I asked to speak to, “He’s in a meeting and can’t respond.”

She gets me an appointment for a technician. For today at 0800-0900. 23 hours from the time I put in the call.

I tell her this is unacceptable. She refuses to do anything. I tell her to yank her supervisor out of the meeting.

When I finally get to talk to him, I use some emotional blackmail. “My VoIP is down. This means I’ve lost e911 capabilities at my site. Tomorrow is to long. I need somebody here today.”

I’ll update this posting if they actually did get the issue fixed in a timely fashion.

Damn, I miss US-based support.

FBEL- Baking Bread

There are all sorts of old timey skills that are useful in a SHTF scenario, but knowing how to bake bread will be high on the list. At one time in history (ie anytime prior to 1930), pretty much everyone had at least a vague idea of how to bake bread. If they hadn’t done it themselves, they at least had witnessed it being made. The modern grocery store killed the last of the bakers, though. Baking, even the “bougie” artisan stuff, is done largely by machine these days. Why bake when you can just pick up a couple of loaves at the store? And store loaves last for weeks and weeks, while fresh bread goes moldy after only a few days.

When you have access to store bought bread, that’s fine. But what if you don’t? What if … oh, say a pandemic happens, and all the grocery stores are out of EVERYTHING, and you can’t find store bread? The bottom line is you need to know how to make your own.

There are simple loaves, no knead recipes that come together quickly with a minimum of mess. There are complex loaves that require multiple rises and tons of work. I tend to go for bread somewhere in the middle. Two rises to develop the gluten and make for a lovely, crusty bread that will hold together as sandwich exterior. A nice mix of flours. Standard yeast. It’s not a difficult loaf, if you know how to bake.

On Sunday, I held a class for six people at the Fort at No. 4, where I taught them how to make bread. From scratch. In a wood fired bake oven. We had a real range of students. One was a reenactor from the current iteration of Roger’s Rangers, who simply hadn’t learned to bake in the beehive and really wanted some help. One was a complete bread virgin but the price was right and how exciting to get to hang out in a fort and cook bread? The two couples were doing the lessons as a sort of “date day” thing, and were at varying levels of having attempted bread. One of the guys was a baker at a big company, but had never made bread with nothing more than a wooden spoon and a bowl.

We started out the day by adding yeast to warm water, and feeding it a bit of sugar. I explained that we were fermenting the yeast, letting it become active. While it isn’t necessary with dry active yeast (which is what we were using), it’s a good habit to get into. While our yeast was waking up, we went out to start the fire in the beehive oven. Everyone brought some wood, and I had already split kindling and had scraps from Chris’s day in the workshop on Saturday, so the fire started up quickly. Once it was loaded up and roaring, we were off to the kitchen again.

We slowly added our flour into the water and yeast mixture, then stirred with a wooden spoon. Once we had most of the flour in and the dough was forming, it got turned out onto the table, and we started the kneading process. Everyone has sore shoulders this morning (except me), because when kneading entirely by hand, it takes about 20 minutes. They all had various problems with their dough as we kneaded, and I was able to explain a variety of possible failure points. Everyone ended up with a decent “silky smooth” ball of dough, and we set that to rest.

Read More

A depressed, stressed woman putting her face on a pillow, mental problem and health care concept

Words Are Not Deeds

Or to put it in the jingo of my youth, sticks and stones may break your bones, but words will never hurt you.

In Cincinnati, a white man and a white woman were beaten badly. This wasn’t a one-on-one beatdown; it was a gang of feral hood rats attempting to murder them.

Because they are black, they are not to blame. The white folks are. They said something that justified the beatdown.

It is the white victims that need to be charged and arrested. They are responsible for 6 felonies.