How are Falcon-9 telemetry data transmission errors handled?

Question

I understand the rocket constantly transmits the telemetry information to an engineering facility on the surface. I also understand it is normal for a transmission channel to create errors in the transmitted data. I know there are redundancy-based data error correction algorithms in place. My question is, what happens if an error is so big that it cannot be corrected by the algorithm. Does the failed packet gets re-requested from the rocket and transmitted again? Or is it considered irrevocably lost?

Answer 1

This is likely asked in reference to a Tweet by Elon Musk about the CRS-7 failure. Text of the Tweet is:

Cause still unknown after several thousand engineering-hours of review. Now parsing data with a hex editor to recover final milliseconds.

On Reddit, someone posted this very excellent explanation. Not a great source, but best explanation I have seen to date. Specifically, this node:

That's not how this stuff works at all. It's all transmitted CCSDS, typically. Each packet of 1024 bytes typically has it's own error correction code added, in addition to all of the other required features (frame sync markers, virtual channel encoding, etc).

Having had to do this same analysis myself many, many times, I'm guessing that the data is corrupted, particularly the frame sync markers (0x1ACFFC1D... when you do this for a living, yes that's right off the top of your head). When that happens, the Reed-Solomon correction can't really be applied because the frame could have bit slipped and that would make things worse.

So they have to go through each part of the stream in a hex editor manually offsetting things one bit at a time looking for reasonable packet contents (which is almost impossible, as the payload is PN encoded...), and then if they see that, they can then pull of the RS correction symbols and do an error correction, then decode the corrected version again. Then on to the next Kb of data... by hand. You can imagine why this takes a while.

On a Nasaspaceflight.com forum there was an interesting comment on telemetry that is worth pointing at:

Since we're talking about telemetry, I'll throw a bit of my knowledge into the ring.

I build systems that process ground and flight telemetry for spacecraft and instrument avionics (not launch vehicles).

I know nothing about SpaceX' implementation, so I won't even try to give anything specific about their vehicles.

Also, some of what I describe below for spacecraft doesn't apply to launch vehicles as there isn't any time to send commands to a launch vehicle as the launch sequence is highly dynamic and relies heavily on pre-programmed event sequences. As far as I know, there are no commands sent to a launch vehicle during launch after liftoff, aside from an FTS destruct command.

There are standards at work for telemetry, e.g. CCSDS. There are also as many different implementations of the 'standard' as there are manufacturers of avionics.

In everything I work on, flight software does in fact format telemetry, packaging frames inside of headers. There are multiple layers of framing at work, and there can be many different types of telemetry frames coming down at various rates.

Some things happen on a deterministic schedule, some telemetry can be inserted out-of-order if the priority of the data is high enough, and some things may only come down if there is available bandwidth to send it without disrupting the flow of high-priority telemetry.

Depending on the operating modes of the avionics, there can be many different rates and maps. For example, if a spacecraft has entered safe mode (or if an emergency mode controller has taken over the operation of a spacecraft), the data rate and telemetry map can switch to fairly low bit rates (better for retrieving data on the ground if there is a pointing or power problem with the transmitters), and the maps can prioritize only critical engineering telemetry.

Many avionics systems can and do acquire data more rapidly than they send telemetry, in which case the telemetry maps define how often and what kinds of data to send. It is not unusual to have instrumentation for developmental hardware sending telemetry and higher than normal rates.

Likewise, any anomalous condition will often times be prioritized to send that status back as a high-priority telemetry frame.

On all the flight avionics that I work on, there are multiple boxes intercommunicating to flow data that ends up send down in telemetry. A high-energy event can (and indeed is very likely to) cause telemetry to stop being transmitted even if the avionics and flight software are continuing to acquire data.

Unlike launch vehicles, the rates and maps on most spacecraft can be commanded as well as switched autonomously by flight software.

Almost everything I work on is capable of sending unencrypted telemetry, however in flight use everything I deal with defaults to using encryption. Aside from encryption, there can be randomization also applied.

There are no analog telemetry signals on anything I work on, and as far as I know, modern launch vehicles to not transmit any analog telemetry- it's all digitally encoded, framed, and transmitted by flight software within the avionics.

The other Jim above mentioned that this type of event could spell the end of store-and-forward telemetry, and I have my own take on this. I don't believe that it is appropriate to ONLY log telemetry on a launch vehicle, and that engineering telemetry during the early phases of launch should be transmitted live.

I do see the value of a rugged data logger which is essentially digital storage packaged as robustly as it can be, with the intent of building a package that can survive a mishap and be recovered. This is a very difficult challenge on a launch vehicle as they operate right on the edge of having sufficient margin to even survive as the physics of launch are very challenging.

In the scenario above where a high-energy event kills the communication between avionics, a high-rate log might be able to provide some critical insight into sensor data in a mishap, were it to be recovered.

I to expect, however, with the proliferation of sensors generating engineering and operational telemetry that there will be some lower-priority data which will always be stored and made available for downlink if conditions permit.

It is always better to get all critical sensor data down in near real time than count on retrieval after a mishap. Picking what sensors are critical is a challenging activity. In the words of Mr. Musk, some failures can be counter intuitive and a sensor that you thought wasn't critical can, in retrospect, be the most critical sensor needed to decipher an event chain that leads to catastrophic failure of the vehicle.