12
$\begingroup$

That particular quote comes from an article about the Proton rocket (http://www.nasaspaceflight.com/2015/02/ils-proton-m-inmarsat-5-f-2-launch/), but I've seen this used elsewhere. The wiki page for SpaceX's Merlin engines also mentions something similar ('SpaceX uses a triple-redundant design in the Merlin flight computers.' - http://en.wikipedia.org/wiki/Merlin_%28rocket_engine_family%29)

Does it mean that for every relay, there's 3 sets of wires going to it, one from each computer? Does only one computer run at a time, and the others only pick up if there's a failure? How are failures detected?

$\endgroup$
15
$\begingroup$

Never go to sea with two chronometers; take one or three.

Dual redundancy presents a big problem. Suppose you bring two chronometers (or two computers, or two inertial measurement units). What do you do when the two devices disagree? Which do you trust? Triple redundancy solves that problem. Ignoring the possibility that two of the devices go whacko at the same time and in the same way, when one device goes whacko, the other two will still be in agreement.

Sans some very smart hardware/firmware/software, triple redundancy protects against but a single failure. Dual fault tolerance requires very smart computers or requires at least quad redundancy.


Update

In this update, I will focus on fault tolerance versus fault safety, and on common mode failures.

Fault tolerance (in NASA) means that the mission succeeds despite a failure. Dual fault tolerance means that the mission succeeds despite two independent failures. Fault safety means that a failure does not result in irreparable harm, but it does not necessarily mean mission success. There's a big difference between the two. A mission that succeeds despite a failure or two is fault tolerant. Fault safety might mean sending the astronauts/cosmonauts home. While the mission would be a failure, the astronauts/cosmonauts would still be alive after the abort.

For example, as far as NASA and Roscosmos are concerned, a vehicle bringing cargo to the International Space Station would be deemed fail-safe if a failure occurs on the vehicle and the vehicle intentionally splashes itself into the Pacific as a result. A few tens of millions of dollars of cargo might be lost, but the ISS will remain unharmed. Visiting vehicles to the ISS are required to be safe against two failures, meaning no harm to the ISS. Fault tolerance? NASA and Roscosmos don't care. Their primary concern is the continued safety of that trillion dollar asset.

The wikipedia article on redundancy is very wrong. It implies that triple redundancy is the be-all and end-all of redundancy, and that triple redundancy implies dual fault tolerance. Triple redundancy works against some single failures. Dealing with two failures is a much harder problem. A number of aircraft and space vehicles are quad redundant (e.g., the Space Shuttle, the F-16, the 747, and Orbital's Cygnus). SpaceX's Dragon uses massive redundancy (with non-rad hardened flight computers) to achieve two fail safety. Dual fault tolerance is an even tougher nut to crack. Sometimes, two failures means ending the mission and sending the crew home, or splashing an unmanned payload into the Pacific.

Finally, triple redundancy accomplishes nothing if every one of those triply-redundant systems exhibits the same common error. A good number of mishaps in space have been attributed to bad flight software or to bad commands issued to the flight software. It doesn't matter if there are one hundred flight computers if each and every one of them has the same bad code or is given the same bad command. Something bad will happen. Common mode failure is the thing that scares safety engineers the most.

Carbon-copy redundancy offers zero protection against common mode failures. The Space Shuttle used quad redundancy in its primary system to address the problem of two fault tolerance (to some failures). To combat the problem of common mode failures, the Space Shuttle had a fifth Backup Flight System. The Shuttle BFS software was built by a completely different contractor than that responsible for building the primary avionics software system. The job of the BFS (which was never used) was to bring the vehicle back to Earth. While the mission would have been a failure, the astronauts would still have been alive.

Other spacecraft have taken things even further. The Japanese HTV Abort Control Unit runs a separate set of software dedicated toward keeping the Space Station safe. The ACU uses distinct attitude sensors from those used by the primary system.

Yet others argue that the idea of a distinct backup system written by a different organization is a bad idea. Per this line of reasoning, that distinct backup removes pressure on the designers of the primary system to write their software correctly and is an extra expense that would be better spent on the primary software. Moreover, the mistakes aren't so much in the code as in the requirements and design. The designers of the backup are likely to make those very same mistakes.

$\endgroup$
  • $\begingroup$ Very nice answer, thank you. Can you expand on this point though? "SpaceX's Dragon uses massive redundancy ... to achieve two fail safety" Do they have a lot of computers on board? And if so, why are they only two-fault safe instead of more? $\endgroup$ – Nickolai Feb 3 '15 at 15:06
  • 3
    $\begingroup$ The SpaceX Dragon has 54 computers on board. The primary flight system comprises six computers, three sets of two computers each (triple redundancy X dual redundancy). This, along with massive verification and validation (V&V) was enough to satisfy NASA that the Dragon achieves the two fail safe requirement. SpaceX has an internal V&V team that tries to poke holes in the SpaceX flight software. SpaceX also hired a separate independent V&V contractor to poke holes in the SpaceX flight software. Finally, NASA has yet its own analysis team that tries to poke holes in the SpaceX flight software. $\endgroup$ – David Hammen Feb 3 '15 at 15:58
  • $\begingroup$ @DavidHammen Do you have sources on that? That must be an interesting read... $\endgroup$ – mike Nov 28 '17 at 20:31
  • $\begingroup$ @mike - aviationweek.com/blog/dragons-radiation-tolerant-design, for one. $\endgroup$ – David Hammen Nov 30 '17 at 1:18
6
$\begingroup$

Redundancy is implemented in various ways. Generally speaking, triple redundancy means that the system can withstand two failures without having the system fail.

In greatly simplified terms, redundant avionic systems have multiple computers running and comparing results -- voting out a system that gives a different answer. This voting is then effectively the way failures are detected. Sometimes, a completely separate system with completely different software is available as a final backup. The Shuttle, for example, had a backup flight software (BFS) system available to it for this purpose.

$\endgroup$
  • $\begingroup$ OK, so it can withstand 3 failures, but 3 failures in which systems/components? And what if one of the computers fails, and you only have 2 voting, how would split decisions get resolved? $\endgroup$ – Nickolai Feb 2 '15 at 23:07
  • 2
    $\begingroup$ Triple redundancy means that it is two-fault tolerant, not three. $\endgroup$ – Mark Adler Feb 2 '15 at 23:14
  • $\begingroup$ @MarkAdler you are absolutely correct. $\endgroup$ – Erik Feb 2 '15 at 23:50
  • $\begingroup$ @Nickolai triple-redundancy doesn't necessarily mean three computers. Also, a failure modes and effects analysis is typically what is used to determine what failures are covered. $\endgroup$ – Erik Feb 2 '15 at 23:52
  • 1
    $\begingroup$ @Nickolai you asked a very intelligent question about how to tell which is wrong when there are 2 strings left and they differ. The shuttle was at least triple redundant and much, much effort went into resolving this question at each level of redundancy. It's done by a 'selection filter' that operates differently at each level of redundancy. Short answer, with 3 values, the mid value is used, with two values they are averaged and with one..it is used. See page 490 in this document nasa.gov/centers/johnson/pdf/… $\endgroup$ – Organic Marble Feb 3 '15 at 2:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.