20
$\begingroup$

It dawned on me that the tragic STS-107 disaster of (my favorite shuttle) Columbia was after more than 20 years of service.

Computers in 1981 were of course significantly inferior to what I had in my phone in 2003. I'm picturing our old 1980s Tandy 1000 with 640k of memory still operating an incredible space craft in 2003.

I'm not sure what software was in the Shuttle Orbiters, but was it updated? And were there rigorous tests to avoid software crashes?

$\endgroup$
37
$\begingroup$

"Rigorous tests" doesn't begin to describe the process used to make sure there are no bugs in the Shuttle software. This massive article details how the process works.
A few salient points:

  • The Shuttle software consists of ca. 420,000 lines. The total bug count hovers around 1. At one point around 1996, they built 11 versions of the code with a total of 17 bugs. Commercial programs of similar complexity would have thousands of bugs.
  • The process of writing the software is beyond meticulous. They needed an update to the GPS software (6300 lines of code). Before any work was done on the code, they wrote a 2,500 page specification detailing the changes they needed.
  • Every change to the code is documented.

... the history of the code itself — with every line annotated, showing every time it was changed, why it was changed, when it was changed, what the purpose of the change was, what specifications documents detail the change. Everything that happens to the program is recorded in its master history. The genealogy of every line of code — the reason it is the way it is — is instantly available to everyone.

  • Every bug is analyzed exhaustively:

For every one of those errors, the database records when the error was discovered; what set of commands revealed the error; who discovered it; what activity was going on when it was discovered — testing, training, or flight. It tracks how the error was introduced into the program; how the error managed to slip past the filters set up at every stage to catch errors — why wasn't it caught during design? during development inspections? during verification? Finally, the database records how the error was corrected, and whether similar errors might have slipped through the same holes.

  • Everything the software group does is structured in processes, and every error is a reason to examine and improve these processes:

How do they write the right stuff?

The answer is, it's the process. The group's most important creation is not the perfect software they write — it's the process they invented that writes the perfect software.
...
Don't just fix the mistakes — fix whatever permitted the mistake in the first place. The process is so pervasive, it gets the blame for any error — if there is a flaw in the software, there must be something wrong with the way its being written, something that can be corrected. Any error not found at the planning stage has slipped through at least some checks. Why? Is there something wrong with the inspection process? Does a question need to be added to a checklist?

So, Shuttle software is written to the highest standards in the world. And that's just one layer of the system NASA devised to prevent the control computers from causing trouble.

The Shuttle is controlled by 5 AP-101 General Purpose Computers. After an upgrade in 1991, they had 1 Mb of memory and ran at 1.4 MIPS. But they are built like a tank: each one weighs 64 pounds. It has been designed to run for years without crashing. Its memory circuits can detect memory cells becoming corrupted due to radiation, and will prevent the corrupt data from being used.

A software change typically goes through about nine months of in-house simulator testing and then another six months of testing in a unique NASA lab before it is accepted for flight. The results of the strenuous testing regimen? Well, it has been 24 years since the last time a software problem required an on-orbit fix during a mission. In the last 12 years, only three software errors have appeared during a flight. But perhaps the most meaningful statistic is that a software error has never endangered the crew, shuttle or a mission's success.

Each of the 5 GPCs can control the Shuttle on its own if necessary.

Five IBM computers — four of which were arranged in a redundant configuration, with a fifth computer acting as a backup unit — allowed early Shuttle missions to continue even if multiple failures were experienced. The computers cross-checked each other more than 500 times a second.

The 4 redundant computers run the PASS (Primary Avionics Software System). The fifth computer runs an entirely separate program, the BFS (Backup Flight Control System). It has the same functions as the PASS, but is written and maintained by a different group. This adds another layer of security: a bug cannot affect all 5 computers at the same time.

On the Shuttle, four identical AP-101Bs would function simultaneously as a quadruple-redundant set during critical mission phases such as ascent and reentry, processing the same information, derived from completely separate data buses, in precise synchronization. If a conflict arose among the four primary computers, the majority would rule, voting the conflicting unit out of the loop. None of the computers, singly or en masse, could turn off any other—that step was left to the crew.

To ensure that the BFS was as independent as possible, NASA contracted with Rockwell to write it, and even different development environments and configuration management systems were specified.

The software is written in the HAL/S language, which was specifically designed for aerospace use.

The three key principles in designing the language were reliability, efficiency, and machine-independence. The language is designed to allow aerospace-related tasks (such as vector/matrix arithmetic) to be accomplished in a way that is easily understandable by people who have spaceflight knowledge, but may not necessarily have proficiency with computer programming.

HAL/S was designed not to include some constructs that are thought to be the cause of errors. For instance, there is no support for dynamic memory allocation. The language provides special support for real-time execution environments.

Of course this doesn't come cheap:

In an industry where the average line of code cost the government (at the time of the report) approximately \$50 (written, documented, and tested), the Primary Avionics System Software cost NASA slightly over \$1,000 per line. A total of \$500 million was paid to IBM for the initial development and support of PASS.

If you want to know more:

$\endgroup$
  • $\begingroup$ Very thorough!! $\endgroup$ – andy256 May 31 '15 at 8:40
  • 6
    $\begingroup$ Can we fork it on GitHub? $\endgroup$ – Antzi May 31 '15 at 12:35
  • 3
    $\begingroup$ It might be worthwhile to add that this level of rigor had a very high cost: One line of code per programmer per day. $\endgroup$ – David Hammen May 31 '15 at 15:57
  • 1
    $\begingroup$ This answer addresses the "rigorous tests" part of the question, but I don't really see how it addresses the other part (which is also the title question) of how often the software was updated (it only states that the software did receive changes). $\endgroup$ – a CVn Nov 27 '15 at 9:46
  • 3
    $\begingroup$ I worked on the PASS software from '89-95-ish. The answer is accurate. The process was certified as CMM-5 (optimizing). The team I was on had weekly process improvement meetings. Everything was reviewed, any issues/bugs were put through root cause analysis. Checklists were modified to catch similar errors in the future. No interactive debuggers. Compiling and test runs were batch. Builds were made once week. There was always a long queue to setup the test scenario and run it. I've never seen any other development environment that was similar. The SW development process is proprietary. $\endgroup$ – JohnP May 18 '18 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.