36
$\begingroup$

The Mars Climate Orbiter failed in 1999 due to:

ground-based computer software which produced output in non-SI units of pound (force)-seconds (lbf·s) instead of the SI units of newton-seconds (N·s) specified in the contract between NASA and Lockheed. The spacecraft encountered Mars on a trajectory that brought it too close to the planet, causing it to pass through the upper atmosphere and disintegrate.

I find this software error to be absolutely shocking. How was it not found prior to the launch? I would think some basic integration testing would have caught the error. And were there no simulations that indicated the spacecraft would have flown on an incorrect path based on the faulty numbers reported by the software?

$\endgroup$
  • 13
    $\begingroup$ Yeah, you would think these things would be caught in preflight simulations www-users.math.umn.edu/~arnold/disasters/ariane5rep.html $\endgroup$ – Organic Marble Feb 20 '17 at 17:04
  • 11
    $\begingroup$ Or, before a product is released: Pentium FDIV bug. $\endgroup$ – Makyen Feb 20 '17 at 23:54
  • 4
    $\begingroup$ They could also have just used <code>boost::units </code> but everyone makes mistakes, and when there is a long enough chain of mistakes, this leads to disasters. There is no way to do bug free software. $\endgroup$ – PlasmaHH Feb 21 '17 at 9:19
  • 3
    $\begingroup$ @PlasmaHH And the more complicated systems and libraries you use, the higher the chance the problem will be in the libraries. How does boost::units perform under hard radiation? :D $\endgroup$ – Luaan Feb 21 '17 at 9:32
  • 11
    $\begingroup$ @Luaan: very well, as it is a zero overhead component that only exists at compile time. You mess up the units, it doesn't compile. If it compiles, its the same machine code. It is also useful to understand how such a library woks and what can absolutely not happen as a result of using it. $\endgroup$ – PlasmaHH Feb 21 '17 at 9:35
47
$\begingroup$

NASA formed a board to investigate the loss of the spacecraft and reached some high level conclusions. The board cited a number of contributing factors, which I have filtered to include the ones most relevant to the question:

  • errors went undetected within ground-based computer models of how small thruster firings on the spacecraft were predicted and then carried out on the spacecraft during its interplanetary trip to Mars
  • the systems engineering function within the project that is supposed to track and double-check all interconnected aspects of the mission was not robust enough, exacerbated by the first-time handover of a Mars-bound spacecraft from a group that constructed it and launched it to a new, multi-mission operations team
  • some communications channels among project engineering groups were too informal
  • the small mission navigation team was oversubscribed and its work did not receive peer review by independent experts
  • personnel were not trained sufficiently in areas such as the relationship between the operation of the mission and its detailed navigational characteristics, or the process of filing formal anomaly reports
  • the process to verify and validate certain engineering requirements and technical interfaces between some project groups, and between the project and its prime mission contractor, was inadequate

Also in the high level report is this quote:

These contributing causes include inadequate consideration of the entire mission and its post-launch operation as a total system, inconsistent communications and training within the project, and lack of complete end-to-end verification of navigation software and related computer models.

It sounds like it was a failure of management and quality control at multiple levels. The entire report is also available if you'd like some light bedtime reading.

The Mars Climate Orbiter was one of the probes in administrator Goldin's Faster Better Cheaper (FBC) initiative, which forced tight budgets and very short timelines on projects, which has been controversial ever since as there were several spacecraft failures attributed to failures of management and engineering due to the initiative. The Harvard Business Review has a great article summarizing what went wrong:

In shifting to FBC from a slow, reliable, but costly approach to development, NASA forced its project managers to invent radically new processes and procedures. FBC imposed on them budget, schedule, and weight constraints that could not be met using NASA’s traditional approaches to spacecraft development. “The attitude was ‘The book’s not working. So throw out the book, try something different, and then write a new book,’” one NASA manager explained. Implicit in this approach was the need for project managers to learn from the organization’s collective experiences, adopt what worked, and jettison what didn’t. Unfortunately, NASA undermined this learning process in several ways.

First, with the launch of each FBC mission, NASA demanded ever faster development times and even lower costs. But because it typically takes more than four years for a small spacecraft to go from drawing board to completed mission, managers were forced to meet the tougher demands on new projects while earlier projects were still in progress. So they couldn’t capture all the potential lessons from one mission before moving to the next. In short, NASA was raising the bar before seeing if project managers could clear it where it was. By the time the organization realized it had set the bar too high—around the time the first FBC missions began to fail—the project pipeline was full of missions that were potentially compromised. It’s no surprise that later FBC missions failed more frequently than earlier ones did.

Second, NASA didn’t realize that because the FBC initiative depended so much on shared learning, it would require a more aggressive and systematic approach to knowledge management. Although NASA had implemented a “lessons learned” database in 1995, a 2001 survey found that only one-quarter of its managers contributed to it. A similar number of managers were unaware the system even existed. Furthermore, while “red team reviews”—periodic progress reviews conducted by NASA’s most experienced managers—proved invaluable in early FBC projects, NASA conducted fewer of these assessments in later missions. As a consequence, the transfer of learning across the organization suffered.

Finally, NASA fell prey to “superstitious learning”—the assumption that there is more to be gleaned from failed missions than from successful ones. In the challenging climate of space exploration, however, the difference between what makes one mission succeed and another fail can be subtle. There is no reason to believe that success indicates a flawless process while failure is the result of egregious bad practice. For example, as many mistakes could have been made in the celebrated 1997 Pathfinder mission as were made in the failed 1999 Polar Lander mission. But NASA will never know. By not conducting detailed postmortems on its successful missions, the space agency missed the opportunity to identify problems (and solutions) that might have helped avoid later failures.

A summary of this would be that, while there was nothing bad about trying to speed up the pace of missions and cut costs the way it was implemented forced people to cut corners. The strategy depended on shared learning, with newer projects re-using older project's code, equipment, knowledge and lessons learnt but the agency did not put adequate tools in place to do this nor did it foster a sharing culture. Lessons from earlier projects weren't learned because the earlier projects weren't completed before the later projects were started. They didn't review successes as top management didn't think there was anything to be learned. There were several spacecraft failures which are attributed to this strategy backfiring, leading to the phrase "Faster Better Cheaper - you can have any 2 you like", however there were many notable successes from the strategy including Mars Pathfinder and the NEAR asteroid rendezvous and in the end all of the 10 FBC spacecraft cost about the same as Cassini.

$\endgroup$
  • 6
    $\begingroup$ I worked on Mars Climate Orbiter. The reasons listed in the bullet points for this answer ring true to me. It's worth noting that the navigation required to get the vehicle to Mars worked fine; it was the final orbit approach and entry that failed. That part of the mission is also the most difficult to simulate accurately, particularly in an environment with a number of geographically diverse players. $\endgroup$ – RickNZ Oct 7 '18 at 22:36
  • 3
    $\begingroup$ That's interesting @RickNZ, thanks for sharing it. It must have been a bad day when that happened! $\endgroup$ – GdD Oct 9 '18 at 7:48
25
$\begingroup$

There were a good number of chances to catch the error after launch, which is what most of the reports on the mission focus on. To look specifically at what testing was done before launch this paper from the American Astronautical Society has a decent overview, starting on page 6: The failures of the Mars Climate Orbiter and Mars Polar Lander: a perspective from the people involved

  1. They were modifying code from a previous mission, which had the unit conversion factor but buried it in the equations and didn't comment it to make the conversion obvious.
  2. Requirement and code walkthroughs failed to notice the missing conversion factor because it wasn't obvious in the original code and they didn't understand the previous equation to the degree necessary to spot the lack.
  3. Formal software acceptance testing used a "truth" file produced by manually calculating the equation that was coded, not a data file from an independent source (likely because the navigation team wasn't brought onto the project until very late in development as a cost-saving measure).
  4. Integration testing consisted of making sure the file was produced and could be moved to the right server: they didn't do anything with the data in the file.

What a lot of this came down to was that they were reusing code from a previous mission, and it had gone through all the validation there, so management (who were under pressure to minimize costs) assumed the modifications were low risk. They did enough testing to feel comfortable, but they didn't have experts on hand to do independent testing so it didn't buy them much.

As an aside, this failure to actually do meaningful integration testing meant that at launch the data files produced by the small forces module (the software with the bug) were in the wrong format and couldn't actually be used. For the first few months of the mission the navigation team was applying/calculating the results of the angular momentum desaturation firings manually based on emails between them and the contractor.

$\endgroup$
  • 3
    $\begingroup$ "calculating ... manually" Wow, to think they might have found the bug before it became critical. Such sadness. $\endgroup$ – KalleMP Feb 21 '17 at 10:30
  • 2
    $\begingroup$ Unit conversion errors could be made near-impossible by proper programming techniques. Supporting them was one design goal for Ada. Ada allows to create sub types from all built-in types. Different sub types cannot directly be assigned to each other. So one would have one type for feet and another one for meters; both keep the built-in operators, but any conversion must be explicit. I thought that Ada is in use in space and aeronautics? $\endgroup$ – Peter A. Schneider Feb 21 '17 at 17:18
  • 3
    $\begingroup$ @PeterA.Schneider Many projects have shifted to C. The rationale for such a shift is definitely far beyond the scope of a comment. It could actually serve as a heck of a SE question! $\endgroup$ – Cort Ammon Feb 21 '17 at 17:30
3
$\begingroup$

Big organisations have big cracks for things to fall down. NASA has the additional problems of:

  • There not being much precedence for most of the things they do.

  • They only getting one shot at it most of the time.

In such case the big ticket items, are typically well taken care of. If it's clear what the question is and there is the potential for get it wrong there are resources to solve the problem and none says go until an you can prove it will work.

However if its not clear what the question to ask should be, possibly because is bleeding obvious, if none is checking it, all the resources in the world won't help.

$\endgroup$
-3
$\begingroup$

I worked on the Midas and Samos projects in the 50's and & 60's, and on the Space Shuttle until it's demise in the 80's. On the former, quality assurance was comprehensive, every nut and solder joint was inspected. There essentially was no QA on the Shuttle, technicians I was told would inspect their own work! Which would save lots of time and money.

As in most businesses, it's about money. As long as the profits exceeded the losses, it is considered acceptable.

$\endgroup$
  • 1
    $\begingroup$ Your information on the Shuttle disagrees with e.g. space.stackexchange.com/questions/9260/… - which says the Shuttle flight software is among the most rigorously reviewed in the world. Do you have any additional information? $\endgroup$ – Hobbes Feb 21 '17 at 17:43
  • 1
    $\begingroup$ Also, the Shuttle's "demise" was not in the 80s, it was in 2011. $\endgroup$ – Organic Marble Feb 21 '17 at 18:47
  • 1
    $\begingroup$ Also, you're talking about Shuttle hardware, not MCO software. $\endgroup$ – Nathan Tuggy Feb 21 '17 at 18:56
  • $\begingroup$ @Hobbes I think the OP is talking about more the mechanical and regular service things, not about the software. And his experience is mainly from the pre-Challenger era, from which it is known that inspection/testing wasn't so strong as after the catastrophe. $\endgroup$ – user259412 Oct 6 '18 at 19:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.