What was the first unplanned "over-the-air" software update of a spacecraft?

Question

From time-to-time spacecraft have had to be rebooted, in one case Voyager 2 mutinied and had to be reprogrammed and in another Opportunity had it's memory "hacked".

As far as I know, deep space spacecraft have always been designed with some capability to receive and store instructions and execute them at a later time, both "bus-like" for propulsion and navigation, and "payload-like" for science payloads, but the first Earth orbiting spacecraft were so primitive that they didn't have anything close to memory or a computer (see also first transistor and last tube (valve) in space).

Question: What was the first spacecraft to receive an unplanned "over-the-air" software update or reprogramming of some kind in space during a mission?

This should be something that was not anticipated or expected, even though a provision for such an eventuality existed or was discovered in a pinch.

If there are different "firsts" for deep-space and Earth orbit missions, it would be good to hear about both.

Answer 1

February through August, multiple times, 1969.

(TL;DR)

Q:

What was the first unplanned "over-the-air" software update of a spacecraft?

A:

Mariner 6 and 7 probes, launched in 1969 to observe Mars, differed to the earlier probes in having new computers that could be reprogrammed. This allowed, for the first time, changes to be made to the missions that were unplanned prior to launch.

• So, reprogramming, whether planned or unplanned prior to launch, would have been impossible prior to the CCS computers launching on the Mariner probes. So the earliest date for a OTA software update to any spacecraft, at all, would be these two probes in 1969.

(This also led to the reprogrammable and backup computers being used on the Viking missions, which in turn used identical hardware on the 1977 Voyager missions with software improvements allowing the mutiny of Voyager 2 to be subdued.)

Mariner 6 and Mariner 7 were identical spacecraft launched on February 24, 1969 and March 27, 1969 respectively, and their missions were entirely devoted to the flyby study of Mars.

In transit to Mars, likely due to a battery rupture, contact was temporarily lost with Mariner 7 on July 30. After a 7-hour silence, contact was restored, but it soon became evident that the instrument responsible for reporting the orientation of the television cameras had been damaged and was no longer functioning. Without this information the Mariner 7 cameras could not be pointed properly and, with the Mars encounter close at hand, a solution was needed quickly.

On August 1, manual calibration by ground crews brought Mars into the view of Mariner 7 cameras and, on August 2, Mariner 7 began to relay far encounter images of Mars. The restoration of the Mariner 7 imaging system was a prime example of the expertise being developed by mission operators during these early interplanetary missions and the event stood as a testament to the importance of having a reprogammable computer on the spacecraft.

https://nssdc.gsfc.nasa.gov/planetary/mars/mariner.html

The new Central Computer and Sequencer (CCS) system, used for the first time on this mission, which allows extremely flexible spacecraft operation using in-flight reprogramming of the computer memory by radio command .

Previous ...missions employed a relatively simple fixed-sequence device having a series of events preprogrammed and hard-wired prior to launch. The only events that could be changed after launch were the turn durations, midcourse velocity increment, and the time of midcourse.

The difference with this new system was ..although this flight program is loaded into its l28-word memory prior to launch, the CCS may be completely reprogrammed in flight by radio command.

During the flight of the Mariner VI and Mariner VII spacecraft, the CCS has been reprogrammed many times by ground command. As of June 17, 1969, the Mariner VI had received 575 radio commands and the Mariner VII had received 217.

The flexibility allowed by the reprogrammable CCS permitted the operations team to carry out many sequences that had not been previously planned. It has also enabled alternative approaches when problems in other spacecraft subsystems have occurred.

As an example, calibration and testing on the Mariner 1969 spare television system indicated that the automatic aperture control had been positioned in a manner that could cause excessive brightness of some of the early near encounter television pictures. This was a result of abrupt brightness changes sensed as the television field of view swept across the line separating the darkness of space and the bright limb of the planet. The CCS was reprogrammed by ground command to cause an automatic sequence that would hold the camera aperture control at a minimum- gain position through several pictures, after which it would go into an automatic compensating mode.

This is only one example of many instances where the CCS has been reprogrammed to adjust for performance variations so that the spacecraft is able to carry out a nominal mission in the event of failure of the command system occurring at a later time.

Discussion at the end of the paper:

Q. You mention that the magnitude of flexibility increased the efficiency. What do you mean by efficiency in this connection? Was it calculated or measured?

A. The improvement in the efficiency of the Mariner 1969 mission is one example: during the time after closest approach to the planet, approximately 300 commands were sent to the spacecraft to reprogram the digital computer on board, which allowed the spacecraft to perform a series of manoeuvres which completely map the sky in the ultra violet.

This was a mission which had not been designed into the spacecraft prior to launch, but was made possible by our being able to reprogram the control system from the ground. Mission improvements were made by reprogramming the digital computer.

https://www.sciencedirect.com/science/article/pii/S1474667017687755

In-flight reprogramming, begun when the programmable sequencers flew on Mariners, and brought to a state of high quality on Mariner X, was a nearly routine task by the time of Voyager's launch in 1977. Both the CCS and Flight Data System computer have been reprogrammed extensively.

https://history.nasa.gov/computers/Ch6-2.html

On an end note, not the first but certainly one that came to my mind immediately, aside from Viking and Voyager (through looking at Viking I noticed Mariner..), for reprogramming was the International Sun-Earth Explorer-3 (ISEE-3) satellite - reprogrammed to become ICE.

https://en.wikipedia.org/wiki/International_Cometary_Explorer

and finally, a video:

Mariner 6 and 7 Programs Missions to Mars, 1969 HACL Film 00033

At 8:58 it mentions the commands being sent to Mariner.

and lastly, this reminds me of Voyager, but in reverse:

https://meh.com/forum/topics/building-the-plane-on-the-way-up

When the Voyager probes were launched with Reed-Solomon encoders on board, no Reed-Solomon decoders existed on Earth.

Also, as an aside:

The Voyagers original control and analysis software was written in Fortran 5.

NASA Study on Flight Software Complexity

Growth in Code Size:

1969 Mariner-6 (30)
1975 Viking (5K)
1977 Voyager (3K)
1989 Galileo (8K)
1990 Cassini (120K)
1997 Pathfinder (175K)
1999 DS1 (349K)
2003 SIRTF/Spitzer (554K)
2004 MER (555K)
2005 MRO (545K)
1968 Apollo (8.5K)
1980 Shuttle(470K)
1989 ISS (1.5M)

https://www.nasa.gov/pdf/418878main_FSWC_Final_Report.pdf

Answer 2

Near loss of SOHO
There are three "firsts" that I think are relevant to this question. The first is the near loss of SOHO on June 24, 1998. You can read the details on the Wikipedia page but the summary is the flight operations team was performing some normal attitude control tests when the spacecraft sensors lost sight of the Sun causing the spacecraft to panic. The entire effort ended after nearly a year and the flight operations team had to design, write, and implement a new attitude control system that did not depend upon the now functionless gyroscopes. This last part, I think, satisfies your "unplanned" aspect of a software update since I cannot foresee them thinking about how to control the spacecraft without the gyroscopes during the initial design phase of the mission.

Loss of STEREO-B
The second unplanned problem occurred during the loss of STEREO-B. While the Wikipedia page doesn't detail exactly what happened, the real issue was that APL was performing some attitude maneuvers (on October 1, 2014) in anticipation of the spacecraft going behind the Sun (relative to Earth). See the spacecraft had to "flip over" before passing behind the Sun so that upon exiting the solar limb the high gain antenna would point toward Earth. The team started performing maneuvers (note the spacecraft were over one astronomical unit away, i.e., more than 8 light minutes) and suddenly realized that something was wrong. They quickly determined that the spacecraft had gone into an uncontrolled spin but they did not elevate the issue to an emergency (i.e., this would have allowed them to use the Deep Space Network (DSN) immediately and likely would have recovered the spacecraft). The spacecraft couldn't reorient on its own as it was spinning too fast, so it went into a safe mode waiting for help. Its spin axis would remain roughly fixed relative to its initial spacecraft-Sun line, so over time the solar panels would no longer be illuminated and it would die. After four years NASA eventually gave up trying to recover the spacecraft. In this case, the "unplanned" part was all the efforts to try and despin a spacecraft when communication is itermittent and then trying to recover the spacecraft after it came out from behind the Sun.

Interesting side note: The flight operations folks did not initially realize they would need to flip the spacecraft after they passed behind the Sun as that was well beyond the planned mission lifetime, among other things.

Double SEU on Wind
The third unplanned problem occurred in late October 2014 when the Wind spacecraft had two simultaneous single-event upsets (SEUs) in its command and data handling (C&DH) hardware. The flight operations team, along with DSN, had to manually acquire the spacecraft (which I still find amazing) by manually adjusting the orientation of one of the 34 meter dishes at DSN. Once that was done, they had to diagnose and correct the issue using the real time telemetry stream only. This resulted in at least two of the flight ops engineers having to dig up the old three-ring binders of the spacecraft command code and then translating it in real time (i.e., they looked at the three-ring binders while typing) to the current software used by DSN. They then had to completely rebuild and rewrite the entire spacecraft command tables (SCTs) from scratch (some are updated and changed every few days during the usual downlink intervals, but not all). This was elevated to an emergency immediately and the spacecraft was recovered to an operational status within ~11 days (old spacecraft take a lot of time to re-upload entire SCTs and we were only given ~3-4 hours per day after it was established we could contact the spacecraft at will). The spacecraft was back to full science operations after another ~20 days or so. In this case, the "unplanned" part was the recreating of the SCTs, diagnosing a an apparently unresponsive spacecraft using only real time telemetry in a beacon mode, and using the second C&DH processor until the first could be reset and properly re-programmed.

Interesting side note: The second command and attitude processor (CAP2) had not been used until this anomaly occurred. The CAPs contain the error encoder that ensures the received bits at Earth are the correct ones (e.g., Reed–Solomon encoder). The Reed–Solomon encoder on CAP1 failed back in 1997 but it still had a convolution encoder so it was still good. It was not known prior to this, but upon using CAP2, the flight ops folks discovered that the convolution encoder had failed. The design of the encoders allowed one to electronically bypass the Reed–Solomon encoder but not the convolution encoder, thus we realized we could not rely on CAP2 for operations and had to recover CAP1 fully. The error rate without an encoder was found to be only 1 in 10,000 bits (or 1 bit in every ~1250 bytes), but that still problematic if it's a bit that informs code where to point in a binary file when looking for data. So CAP1 recovery started about a month after operations had been fully restored using CAP2 and CAP1 was reinstated as the primary about a month after that (i.e., roughly Jan. 30, 2015).

I am positive there are other, more interesting examples but these were the first I could think of off the top of my head.