In 1997, NASA remotely fixed a bug that caused priority inversion on their Mars Pathfinder. How did they go about doing this? What kind of communication protocols are used? How do they update the source for an operating system, compile it, and run it from a remote location? This might be simpler than I thought, but to me this seems like quite the feat!

Story of the bugfix here: http://research.microsoft.com/en-us/um/people/mbj/mars_pathfinder/authoritative_account.html

The author said to email him and he would provide details, but this was almost 20 years ago. Curious to see if anyone else knows how this worked.


Specifically, for the Mars Pathfinder priority inversion problem, this is explained in detail in Mars Pathfinder: Priority Inversion Problem, Report for the Seminar Series on Software Failures, Risat Mahmud Pathan, Chalmers University of Technology (PDF). I'm reproducing here an excerpt that's most relevant to your question, and I'd recommend reading the rest of it from the source:

2.2 How was it debugged?

The software on Mars Pathfinder had several debug features. One of these tools is atrace/log facility. The feature remained in the software in the final version of the design because the engineers at JPL have the philosophy that "test what you fly and fly what you test". So, they did not remove the debug facility; it was there in Mars. After the problem occurred on Mars, they run the same set of recorded activities (sent by Pathfinder before resetting) over and over again in the lab and within three weeks they were able to reproduce the error in the replica at JPL. The priority inversion problem was obvious. The solution is to enable priority inheritance by setting the mutex flag for the select() calls of ASI/MET to “on”. However, the fix is not so obvious for several reasons:

Concern 1: Setting the mutex flag is a global option and thus applicable to all mutex. Enabling it for ASI/MET would enable it for other tasks. How would this change the behavior of the rest of the system?

Concern 2: The priority inheritance option was deliberately “off” by Wind River1 for optimum performance. How will performance degrade if we turn priority inheritance “on”?

Concern 3: Would the select() mechanism become incorrect if priority inheritance was enabled?

Wind River concluded that the performance impact would be minimal and that the behavior of select() would not change. The JPL engineers tested, analyzed and concluded that changing the flag on a global basis had no adverse impact. So, they decided to patch the software in Mars by enabling the priority inheritance option.

2.3 How the patch was uploaded?

VxWorks contained a C language interpreter to execute statements on the fly during debugging. The JPL engineers decided to launch the spacecraft with this feature still enabled. A short C program was uploaded to the spacecraft, which when interpreted, changed the values of the mutex flag for priority inheritance from false to true. No more system reset occurred!

Quoted source also lists several references worth reading, but for a broad overview of the problem and its solution I'd recommend Tom Durkin, What the Media Couldn’t Tell You About Mars Pathfinder, Robot Science and Technology Magazine, Issue 1, 1998 (PDF) that includes explanations by Glenn E. Reeves (JPL).

In first revision of my answer, I answered by my mistake for Spirit (Mars Exploration Rover, or MER-A) rover's FLASH problem, so if you're interested in how such problems are fixed in more general sense, then that sub-answer (is that the word?) should give you a clue that it really depends on each specific hardware and the nature of the problem. Pathfinder used COTS (Commercial, Off-The-Shelf) hardware with IBM's radiation hardened RAD6000 processor and Wind River vxWorks RTOS. Other Mars landers and rovers, unless they're of same series like Spirit MER-A and Opportunity MER-B, might use different hardware and OS combination. And while RAD6000 (Pathfinder, Mars Polar Lander, MER-A & MER-B, Phoenix Polar Lander) and vxWorks RTOS (Pathfinder, Spirit, Opportunity, Curiosity) seem to be a rather popular processor and OS combination on Mars, they're not the only ones either (Sojourner rover that Pathfinder deployed used a custom cyclic executive OS not too unlike today's Arduino platform, Curiosity uses RAD750 PowerPC,...).

Communications channel to send these updates to hardware on Mars is NASA's Deep Space Network (DSN) that uses Mars orbiters to relay communications to and from rovers, landers et al. on the ground of the red planet (you can observe its status at DSN Now). For distant hardware where communications delay could be dozens of minutes long, commands are usually transmitted in sessions for a whole Martian day (sol) or more in advance and, when necessary (like with changes to the FSW - flight software), first uploaded and tested on replica that stays at home. Exact network protocols could be different, from radio frequency to packet size and structure (today, it's mostly in X-band DTN - Delay-Tolerant Networking), but all would be programmed to listen with their communication subsystem on specific frequencies for newly issued commands and schedule them according to priority flags within received data. Some hardware on Mars (like Mars Science Laboratory, better known as Curiosity) uses dual bus platform for redundancy and fault tolerance, but since that increases size, mass, power consumption, and so on, smaller ones don't.

1 Wind River systems supplied the VxWorks Real-Time Operating Systems for Mars Pathfinder mission.

  • $\begingroup$ I think technically the RS6000 is the model line, and Power X is the CPU. (Power 3, 4, 5, 6, 7, or 8). I just worked on an AS400 with Power8 I believe. That is a kickass CPU. I had an RS6000 with a Power 3 chip in it on my desk many moons ago. $\endgroup$
    – geoffc
    May 22 '15 at 1:19
  • $\begingroup$ @geoffc Updated. ;) I was working more with Alpha and SPARC powered workstations and servers back then. Also kickass, there's a lot to miss from those times. $\endgroup$
    – TildalWave
    May 22 '15 at 1:33
  • $\begingroup$ "Pathfinder used COTS (Commercial, Off-The-Shelf) hardware with IBM's radiation hardened RAD6000 processor and Wind River vxWorks RTOS" Where you know that from? $\endgroup$ Jul 19 '16 at 12:55
  • $\begingroup$ The first link is 404 $\endgroup$
    – usernumber
    Oct 13 at 16:43

Generally space probes have several copies of the operating system. The one being run, and one it can swap too, and often a third that is a safe mode/fallback that is simplest and allows for remote reboots.

They would upload the new code to one of the backup memory locations, then swap over to run on it once it was complete.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.