What type of data did New Horizons send to Earth in order to diagnose the switch to Safe Mode issue?

Question

Recently, the New Horizons spacecraft on its way to Pluto suffered a "hard-to-detect timing flaw in the spacecraft command sequence." Does that mean that it was a multi-threading error or something like that? What kind of data would be sent back to Earth to help the NASA diagnose the issue? Would they receive a stack trace? I imagine that the enormous distance would cause some unreliability in the transmission besides the 4.5 hour transmission time, so I would guess that the data that gets sent back and forth needs to be rather minimal.

Answer 1

The first clue was in which frequency the spacecraft switched to, so they learned a great deal just from reestablishing communications lock. Then, it appears that the necessary data was in the normal housekeeping data/logfiles, as no special data collection is mentioned.

Multithreading was not the issue and in fact, the software never crashed. It was simply asked to do two computationally intensive things at once: (1) compress science data and (2) copy a command load to non-volatile memory. Doing so used more CPU percentage than the system expected, and the fault management systems decided that the processor was misbehaving, and forced a switch to the backup computer. A stack trace would not have been helpful in this case.

Source: Washington Post

Answer 2

New Horizons carries two computer systems. As the New Horizons team posted on July 4:

The mission operations center at the Johns Hopkins University Applied Physics Laboratory, Laurel, Maryland, lost contact with the unmanned spacecraft -- now 10 days from arrival at Pluto -- at 1:54 p.m. EDT, and regained communications with New Horizons at 3:15 p.m. EDT, through NASA’s Deep Space Network.

During that time the autonomous autopilot on board the spacecraft recognized a problem and – as it’s programmed to do in such a situation – switched from the main to the backup computer. The autopilot placed the spacecraft in “safe mode,” and commanded the backup computer to reinitiate communication with Earth. New Horizons then began to transmit telemetry to help engineers diagnose the problem.

On July 5, they posted this, and explained the problem:

The investigation into the anomaly that caused New Horizons to enter "safe mode" on July 4 has concluded that no hardware or software fault occurred on the spacecraft. The underlying cause of the incident was a hard-to-detect timing flaw in the spacecraft command sequence that occurred during an operation to prepare for the close flyby. No similar operations are planned for the remainder of the Pluto encounter.

"I'm pleased that our mission team quickly identified the problem and assured the health of the spacecraft," said Jim Green, NASA's Director of Planetary Science. "Now - with Pluto in our sights - we're on the verge of returning to normal operations and going for the gold."

They mentioned nothing about needing to download logs, just "telemetry", and given the communications lag and rapidity that they diagnosed the issue, I doubt they downloaded much data. I know little about the technical details, but I would be absolutely shocked if they didn't have a simulator or duplicate MIPS R3000 CPU on Earth.

I presume they simply replicated the series of commands they sent to the probe to their "virtual New Horizons" computer on Earth, and realized the error of their timing glitch. The error, while they say was hard to detect, is perhaps (very simplified) something along the lines of:

1. Command A executed.
2. Probe requires X seconds to reorient.
3. Command B executed Y seconds later.

Y < X. Command B Fails. Computer detects failure, switches to backup, phones home.