How To Securely Update A Device

There are two basic types of devices, an embedded system, and a general system.

There are embedded systems everywhere. Your smart TV. The Chromecast you have attached to your “dumb” TV. Your coffee maker, your washing machine or drier.

These are a few of the embedded systems you use every day. There are some that are “critical” systems. Your car, likely, has an embedded system. If that system were to be modified, it could cause “bad things” to happen.

Medical devices are also considered to be critical systems. The computers that control your IV medical drip, even the automatic blood pressure machines or any of the machines that are used for monitoring are critical.

While the blood O2 monitor, attached to your finger, might not seem critical, if the values it is reporting are in error, your health care professional (doctor) could miss diagnose something.

Other examples of critical systems include: many military computer systems, voting systems, systems processing classified information, alarm systems, spacecraft control systems.

Your computer and laptop are not normally considered to be embedded systems, yet they have a part that is embedded. That is the BIOS on your computer and laptops.

Your phone and tablets are a sort of hybrid, where a large part is embedded, but there is an easy way to add other software.

Update A System

For many people, updates just happen. You sit down at your computer, and it tells you to not power it off or reset because it is in the middle of an update. At some point, it completes and things have “changed”.

For others, like my systems, the system will tell me of updates, I will choose to install them or not, as I wish.

Most updates are distributed as “binary” updates. These binary updates will have complete replacements for libraries (DLLs) or programs or data sets. They might also have scripts to make configuration changes.

Why in the world would I want to install a binary that I just downloaded from the Internet?

That involves trust. If you thought you had trust issues, it’s going to get worse.

I trust Canonical, these are the people who produce the Ubuntu version of Linux. Most of my servers and desktop computers run some version of Ubuntu. (yes, I run many other versions as well, but…)

Canonical promises me that they will only send out software that is “good”. So if I download a package from their website, it will be “safe” and “good.” Well, sort of.

How do I know that their website has not been compromised? What if I’m downloading the package from a mirror site that is not directly controlled by Canonical? What if I’m working on a server in China, and I don’t trust anything that is pulled from nominal sources?

The simple answer is digital signatures.

Digital Signatures

In 1977, Rivest, Shamir, and Adleman published their paper on “Public Key Encryption.” This paper is the basis of most, nearly all, secure communications on the Internet.

The concept was simple. They realized that there were a couple of difficult math problems that could be used in interesting ways.

The hard math problem they used was factoring large, nearly prime, numbers.

Consider the number 75. If we start brute force factoring, we find that 2 does not go into 75, 3 does, and 5 does, twice. Giving use factors of 3*5*5.

Now consider 79. It is not divisible by 2, 3, 5, 7, or 11, making it a prime number.

75 required us to test 5 numbers to completely factor the number. For 79 we had to test 5 numbers as well, but that was only because it was such a small number.

But, what if we were trying to factor 6557 vs. 7047? They are nearly the same size. Which is easier to factor?

We find that 3 goes into 7047 5 times. Leaving 29, a prime number. That is fairly easy to do. It is factoring that can be done by hand. We have to test 22 numbers before we find the first (of two) factors in 6557.

This is the primary idea behind RSA. The Wikipedia page has all the “hard” math laid out for you.

The major discovery by RSA was that they could generate two keys, such that you could encrypt a message with one key but had to decrypt it with the other key. Knowing one key did not allow you to solve for the missing key.

So we have an ability to do asymmetric cryptography. We, arbitrarily, mark one of the keys “private” and the other key “public”.

We publish the “public” key so that everybody has access to it.

If I want to send you a secret message, I encrypt it with your public key and send it to you. You are the only person with your private key, so you are the only person that can decode the message.

You can also use your private key to encrypt a message. You can then publish your encrypted message. Everybody who has your public key can decrypt your message. But only you could have created that message because you are the only person who has your private key.

But what sort of message would you be sending me? How about a checksum?

A cryptographically secure checksum, is a number that is calculated from the bits of a digital source (file). The characteristics are such that changing even one bit of the file will result in a different checksum. The part that makes it cryptographically secure, is that it is extremely difficult to modify the file in such a way as to generate the same checksum.

The old checksum method was MD5, it is now SHA256 or bigger.

Publishing A Package

Part of the publishers’ job is to distribute their public key, far and wide. You can get that public key from a trusted source, and you can verify that the public key belongs to the publisher because the publisher had their key “signed” by some other trusted source.

This is called a “web of trust.” where you assign your trust value to any signer. What that trust level represents is how much you trust them to never sign an unverified key.

Today we use certain trusted sources that are “easy” to verify.

The publisher then “signs” the package. They do this by calculating a cryptographical secure checksum for the package and then encrypting that checksum with their private key.

When I download the package, I calculate the same checksum by using the same algorithm, SHA256. I then decrypt the publishers’ checksum, using their public key. If my checksum and their checksum match, then I can trust that the package is unmodified from what the publisher published.

This does not mean that the publisher didn’t insert malicious code into the package. Nor does it mean that the author of the package didn’t insert malicious code. It is that I trust the publisher to have done due diligence to protect from a bad actor author, and I trust the publisher to not have injected malicious code.

I don’t trust Microsoft, I do trust Canonical.

Installing a Package

When I want to install a package, I tell my “package manager” to get the most up-to-date list of packages, along with their signatures. I then tell my package manager to download the package, verify the signature on the package, and if the signature matches, to install the package.

For me, that is as simple as apt install -y global-thermonuclear-war and when that command completes, I can play wargames.

If I need to update installed programs, I would type apt upgrade -y, which would update any installed packages that needed updating.

If I wished to, I could configure my computer to automatically do updates, daily. Or to just do security updates. Or, as I have done, just inform me if there are packages that require updating.

Microsoft doesn’t give you those options. When Microsoft decides it is time for you to update your packages, Microsoft pushes the update and your computer gleefully obeys its master and installs the software packages.

This means, in a very practical sense, that if the US Government wanted to spy on your computer, they could have Microsoft push out the spy software and your Microsoft-based computer would install the spy software and happily start reporting you to the government.

Embedded Systems Updates

Some embedded systems expect to be connected to the Internet. They phone home regularly to check for any updates. Then, on the orders of their Master, they install those updates.

This could be a new version of the OS for your iPhone or Android. It could be new software for your Smart TV. Or it could be an update to improve the reliability of your networking equipment.

Some embedded systems are protected from network updates. My router is an example of that, sort of. There is no method to directly update the router. Instead, I have to download the update, verify it myself, then I upload the update to the router via secure means.

Other embedded systems require that updates be provided via a physical device.

One of the nice things about requiring manual intervention to upgrade/update your software, is that it gives us time to see what happens to early adopters.

If the release comes out on Monday and I wait until Friday to do the update, there have been 5 days for the early adaptors to stumble into the landmines.

Secure System Updates

Let’s say that I have a critical device with embedded software. A vote tabulating system. That system is not connected to the Internet. It is isolated.

Because it is secured, we want to make sure that gaining physical access to the system does not defeat all security.

Bluntly, if somebody gains physical access to your computer, they have access to everything on your computer.

A properly secured PC should have USB ports disabled for mass storage. The BIOS needs to be configured to stop people from booting from DVD or USB devices. The BIOS must be password protected, to stop any unauthorized people from accessing the BIOS.

Many years ago, I was helping to perform a security audit at a company that did collections for credit card debt. It was a call center type of place, about 20 people working with one manager.

We had to access the physical security and the security of the devices. One of the tests we did was to plug in a USB device to see if we could move data to the USB device.

What we found was medium quality physical security. The system security was pretty good. Then I went to plug the USB test device into the managers’ computer.

He tried to stop me, got overridden by my boss. Sure enough, his USB ports were open.

The company failed their security audit.

With a secured system, we will often build the package ourselves, from the source code. This allows us to verify what changes have been made since the last time we audited the source code. It means that there is nothing just hidden from sight.

Linux does not use the same programs as the original UNIX. UNIX had a C compiler called the portable C compiler. It was “easy” to port to new hardware.

One of the programs that is built on a UNIX system is “login”. This program handles the username/password authentication process to allow people to log in.

Because this is a critical part of the security for the system, the code has been examined many times, by many people. The source code for login was clean.

What we didn’t know, and this is rumor, was that there was a modification to the portable C compiler to allow it to detect when it was compiling “login”. When it was compiling “login” the compiler would inject a backdoor.

Nobody was checking the compiler for that sort of nasty behavior.

Regardless, when dealing with a secure system, we will often build the package ourselves.

Once the package is built, we transfer it to a USB device. That USB device will have come from a trusted, random source. I.e., somebody purchased it from the PX. It would arrive in its bubble wrap. It would be fully erased three times. Every byte would be written with all zeros. Then all ones, and finally with random numbers.

We would then transfer the package to the USB device and then go to the secure computer. Sneaker net in use.

Somebody else would reboot the secured system to enter BIOS and enable a USB port. Once the port was enabled, we would transfer the package.

We would check the signatures again. (We would have signed the package.). And if everything was good, we would install the package.

Side Note

When the CrowdStrike outage took place, it was because of automatic updates from 1000s of devices that were internet connected.

Interesting tidbit. The vote tabulating computers in Maricopa Country, AZ, are reported to have failed at the same time as the CrowdStrike outage began.

It is interesting because those servers were not connected to the Internet, according to county officials.