High Availability Services

Comments

11 responses to “High Availability Services”

1. 

   curby

   2024-08-25

   what I find frustrating with this modern day life- EVERY single app, program, thingy, whiz bang gets “updated” and the new “update” SUX. it works sssslllooowwwweeerrrr IF it works at all. it requires 10x the “touches” on the touch screen than before. it “buffers” 400 times more often.. and speaking of touch screen- whose idea was it to put every control in these new vehicles on a flippin touch screen???? distracted driving!! distracted driving!!! they scream. then they put everything on a tv screen you have to squint at to fukkin use! glad its only in my company work van… I will NEVER buy a new vehicle for personal use… maybe yall could enlighten some of us knuckle dragging neanderthals..

   1

   Reply
2. 

   Slow Joe Crow

   2024-08-25

   Then you have to think of edge cases. When the hurricane hit NYC a few years ago a lot of supposedly HA sites went down because the backup generators were in the basement, which was flooded. A similar story is the company that wanted to be “green” so they used bio diesel for their generators and 6 months later it had gelled and the generators wouldn’t run.
   Some of our clients run firewalls and servers in HA pairs and a second ISP. a lot don’t because they can survive an outage

   Reply
   1. 

      curby

      2024-08-25

      I got talked into “bio diesel “ for my furnace one winter. 10% bio fuel. after 5 nozzles and 3 filters on the tank in a month and a half I told them to bring me real fuel… internet and “modern “ wifi will go away around here in any minor or major event. I can get along without it..my wife not so much

      Reply
3. 

   Dan

   2024-08-25

   Corporate bean counters who have the final say in how jobs are done want network redundancy. But they don’t want to PAY for network redundancy. When this narrow minded penny pinching mentality meets real world conditions the inevitable result is loss of service. Having been a network admin for a hospital system for more than a decade….as a “second hat”….my primary responsibility being imaging patients….I saw this short sighted insanity routinely. They wanted everything done as cheaply as possible. And when the inevitable failures happened compromising patient care they would be livid wanting to know who’s fault it was. And they would become positively apoplectic when told it was THEIR fault for refusing to budget and plan for failures.

   1

   Reply
   1. 

      curby

      2024-08-26

      the bean counters have infected most every large businesses… the “dei” word is anylist…usually a 20 something college “educated” moron. proof of concept is one of the new big words. I have taught myself to just do my job and find ways to work around thier lunacy..

      1

      Reply
   2. 

      pkoning

      2024-08-26

      The bean counter thing may be how Boeing ended up with MCAS without adequate redundancy. From what I read when those 737s first drilled themselves into the ground, that system has a single angle-of-attack sensor feeding its algorithms.
      I’m not an aeronautical engineer, nor even a pilot (just a skydiver), but I looked at this and said to myself: hm, a control system whose specific purpose is to push the nose down (i.e., to aim the aircraft towards the ground), that’s a safety-of-flight critical system. An SoC critical system with non fault tolerant sensor inputs? Surely you’re kidding me…

      Reply
4. 

   pkoning

   2024-08-26

   I’ve worked on building network and storage systems for decades, and HA is a big piece of the design effort. For networks, it’s redundant boxes (routers and switches) more wires, and careful design of the protocols and algorithms so they detect outages and route around them quickly enough that applications don’t fail. That last part can be tricky. For one thing, it means that the system has to react, guaranteed, within N seconds. It’s not good enough to be fast enough on average, if some of the time it takes longer to reroute than it takes for applications to die. (We liked to joke that the HA deadline is “less time than it takes the customer to get p**d off and call Customer Support”.)

   For storage, a big piece is RAID — multiple disk drives that together store the data. A classic setup is RAID-1, a.k.a., “mirroring” which is just what it sounds like. More efficient storage-wise but slower are RAID-5 and RAID-6 which use one or two extra drives per group of data drives. The hard part for all these is the software for the error case. The “happy path” is easy, but it’s only about 10% of the total work.

   For storage systems especially, I learned that “hot swap” is a big deal, and not an easy one. You build the hardware with redundant components (disk drives, power supplies, line cards). But instead of requiring a system shutdown to replace a failed part, which defeats the point of being redundant, you can pull out the bad part and plug in a replacement while the system keeps running. That requires careful design both at the electrical level and in the software, but it’s doable and it makes for systems that can keep on going for years on end.

   Reply
5. 

   Stuck In Pugetopia

   2024-08-26

   Main and backup path fibers run into building via the same conduit, and the two dirctions on a SONET ring on the same pole, are both referred to as ‘redumbdancy..’

   Reply
   1. 

      pkoning

      2024-08-27

      A variation appeared in the telephone system of the nearby town of Hollis, NH. At the time it was run by a local guy, who also sold fuel oil. Their SONET “ring” was only connected to the outside world at one end, in other words it was installed with the first fault already in place.
      So when a squirrel chewed through that one connection, the entire town disappeared from view, including 911 service. I think it took a day to fix the fault; it’s not clear that the underlying mistake was ever corrected.o

      I learned a bit about fault tolerance design from a Navy guy, who talked to me about FDDI networks aboard ship (normally two rings, with single fault tolerance). The Navy setup ran one ring on the port side and one on the starboard side. I asked why; the answer “so a torpedo won’t take out both of them. Oh. “fault tolerance” takes on an entirely different meaning when enemy action is involved, something I as a civilian wasn’t used to thinking about. Then again, in studying encryption you do learn to think of “data errors” created intentionally and maliciously rather than random things resulting from noise.

      Reply
      1. 

         curby

         2024-08-27

         the town that i live in is running telephone and internet on buried lines that were put in in 1947….good times..

         Reply
      2. 

         Stuck in Pugetopia

         2024-08-29

         One of our local offices in a neighboring town dumped completely off of our network. What is really nuts is that I -know- it had two physically separate fiber paths, and even a private microwave T1 circuit for a last ditch connection.

         Well, the T1 was up so we could at least telnet into the routers and stuff. Lo and behold ALL the fiber paths through that com room were down, including SEPARATE fiber paths for the SCADA system as well as the biz net..

         Ok, beat feet to that office, and on the way there, start looking for car wrecks, fires, etc, that could get that group of cables to fail SOMEWHERE on the path.

         Nothing.

         Finally start pulling out the 19″ rack mount fiber fusion splice trays, and out of the 24 fibers in those trays, I think four were still intact.

         All of the broken ones had at least two chews through them, resulting in a pile of 1/2″ to 1″ fiber segments scattered in the trays, interspersed with numerous rat turds.

         Never found the rat. (I think it was a small pack rat as I’d seen them out in the linemans’ warehouse caught in the traps.)

         I hope the little bastard at least got a bad stomach ache..

         Reply