Is it common and good engineering for a pair of cables to be easily plugged into each other's connectors in modern spacecraft

Question

Space News's Human error blamed for Vega launch failure

Analysis of the telemetry from the mission, along with data from the production of the vehicle, led them to conclude that cables to two thrust vector control actuators were inverted. Commands intended to go to one actuator went instead to the other, triggering the loss of control.

“This was clearly a production and quality issue, a series of human errors, and not a design one,” Lagier said.

Well if someone says "clearly" then it must be right. The same person was also quoted in the BBC's Inverted cables doom European Vega rocket saying:

"This was of course a production and quality issue. It was a human error and not a design one," the chief technical officer told reporters.

this time using "of course", with the same implication.

There must be several different ways that a pair of similar-function cables can be matched to their destinations so that they can't each be successfully connected to the other's. A few I can think of are:

• incompatible lengths
• blocking pins in the connectors

These are in addition to things that are subject to human error but make it harder:

• color coding schemes
• big labels; e.g. "R" vs "L" or "D" vs "G" (French) or "D" vs "S" (Italian)

Question: Is it common and good engineering for a pair of cables to be easily plugged into each other's connectors in modern spacecraft, or are there other known, similar order of hundred million dollar spaceflight failures that could potentially have been prevented by an engineer adding a blocking pin or other simple measures to prevent otherwise identical cables from getting swapped so easily? This seems so foreseeable and potentially preventable that I'm guessing "no" but maybe I'm wrong.

Answer 1

Off the top of my head I can think of several failures caused by miscabling.

• On Apollo 6 the signal to shutdown a malfunctioning second stage engine was cabled to a different engine, resulting in two shutting down instead of one. See: Apollo 6, reason for premature engine shutdown of two engines of second stage of Saturn V?

• The system used to release a payload from the shuttle payload bay was miswired on shuttle mission STS-051. The miswiring in the pyro initiator circuits caused both the primary and backup separation cord to fire simultaneously, showering the payload bay with high-velocity debris. See: https://space.stackexchange.com/a/44771/6944

• There was a truly fascinating wiring error on the shuttle Auxiliary Power Units that went undiscovered for years. Short version - the shaft speed on the unit was measured and used in the speed control system by three magnetic sensors. A voting scheme was used to try and eliminate a bad sensor. However, the wiring error effectively caused one of the sensors to output a zero speed reading at all times. (The ground had no insight into individual sensors.) So on STS-079 when a different sensor failed and started outputting an erroneous zero speed reading shortly after MECO, the voting scheme chose the miswired sensor and the failed sensor as matches, output zero speed, and the unit shut down. The fun thing was that the ground testing unit was wired correctly so it was very difficult to diagnose! STS-79 Postflight Report

Answer 2

Not in flight but this 1996 NASA lesson learned document lists multiple instances of mis connected cables during ground assembly and test including on Galileo.

It in turn references JPL documents attempting to avoid re-occurrence, unclear if ESA has a similar document/process.

The concept here is keyed connectors, many mil spec connectors come in multiple keying variants specifically to avoid this problem, along with other design features like looming cables together into harnesses such that plugs cannot reach incorrect connectors.

It is not normal however to custom key like parts, such as two actuators that are otherwise common, since this means two custom parts exist where the only difference is the connector which will increase cost and complexity and risk having two of one part rather than one of each at some critical juncture.

In this case all that can be done is attempt to make the wiring harness as hard to misconnect as possible (different lengths etc), and make sure a test step exists to confirm the correct actuator moves when demanded not just 'AN' actuator moves when demanded. A related test step in some process can also be to deliberately disconnect a single connector and confirm the right errors are detected.

Edit - related lesson learned report originally in comments 1 2

Answer 3

Not flight related, but everyday: Flammable gas cylinders have left hand threads. Oxygen, and other oxidizing gasses have right hand threads.

The two examples I can think of involve propane and acetylene tanks. (I think that small propane cylinders now have a unique right handed thread for connecting to barbecues)

---

On a different tack: Firehose couplings in some (most? all?) jurisdictions are hermaphroditic. You can use either end This means you don't get to the far end and discover that you have the wrong end of extension hose.

This sort of approach can apply to other things. You set an identity on the subunit with dip switches. Now part of your power on sequence is an inventory check of subunits. If the query to the Port Yaw Thruster comes back, "I'm a starboard Yaw Thruster" you get a red light, and know that either the thruster is misconfigured, or the cable is wrong.

I suspect that a lot of this why control is tending to go digital. Then you have a data connection for telling a subunit what to do, as well as a power connection. In some cases (power over ethernet) for low power items, power can be combined with the same cable. Each device then has a small logic board, a tiny single board computer (See Raspberry Pi and Ardino for examples) that listens for commands, reports status back, and can do some types of diagnostics.

---

Good connector/wiring harness design quickly gets non-trivial.

• Use of blocking pins is one way to keep the wrong connection from being made. This is fairly straight forward, and can be implemented in the field. This would enable smaller inventory.

• Shape+colour can be used. Colour by itself is marginal, as a significant number of people are at least partially colour blind. Colour combined with stripe/dot patterns could be used.

• In cases like the OP's, where you have a common subunit that MUST be connected to the correct cable, design the unit so that you can clip the pins to match the blocking pins in the connector. This allows a single inventory item to be converted to a 'starboard yaw thruster' on the work bench by pulling X pins. At that same time, you apply a label to the unit: "Starboard Yaw Thruster" so that when some tech is weaseled into some awkward corner with two of these, he knows which one to try where. Also: The frame where the thruster attaches is labeled "Starboard Yaw Thruster" with a bolt pattern that makes in impossible to install more than the one way that keeps the control port in reach of the control cable.

A lot of equipment isn't designed for maintenance. The U.S. airforce learned this through WWII. My dad had a bad heart and so his war job was teaching mechanics how to service the B-29 super-fortress. He claimed that most field work could be done with a large phillips screwdriver and a monkey wrench, and that a good crew could change an engine in 45 minutes.

As we move into the era of Space-X and their like, more attention will be focused on refitting. Designers need to bear in mind that maintenance work will sometimes be performed at the end of a double shift, or with a serious hangover.

It's impossible to make a system fool-proof. Fools are too clever by half. But make it as fool-resistant as you can.

---

I am not an aerospace engineer. I'm a just a techy nerd with an interest in aviation and space, and with some experience with how things can go wrong.

Answer 4

This is literally where Murphy's Law, "Anything that can go wrong will go wrong", came from. If you design a parts so they can be fitted incorrectly, they will be fitted incorrectly.

"Murphy" is Edward Murphy Jr.. He was an aerospace engineer working on rocket sleds. In 1948 he suggested installing strain gauges to measure the force of rapid deceleration. Some were wired backwards giving anomalous zero readings.

While there is plenty of contention about whether Murphy's Law was coined by Murphy or about Murphy, the lesson remains: the best way to avoid a mistake is to make it impossible.

---

What you're referring to is defensive design and it permeates every industry which involves connecting two things together: hardware and software.

You can prevent mistakes by making sure all possible choices are the correct ones. This can be by making the bad choices impossible, or by making all choices correct.

On top all orientations are correct, and it will work with several types of plugs. On bottom there is only one possible type of plug and one possible orientation.

Another example is USB connectors. Prior to USB-C, USB connectors had one particular orientation and is (if the socket and plug are designed correctly) physically impossible to insert a USB plug in the wrong orientation. Though the awkward location and small size of many USB sockets lead to an interface problem of having to "flip it over three times" before it will fit. USB-C solved this by making all orientations correct.

In software this is done by type safety. Every piece of data is given a type like text, integer, or decimal and functions will only accept data of the correct type. More types can be defined and they can get very specific. For example, instead of putting 3.44 into a decimal one might use a newton_seconds type. It's still just 3.44, but it will only work with functions which take newton_seconds. Type safety like this would have saved the Mars Climate Orbiter which fed pound-force seconds to a system expecting newton-seconds.

---

There is, of course, a trade off between the expense of needing different connectors for everything, versus how likely an expensive failure is. This is addressed by Risk Management. Sometimes a much cheaper alternative is some paint.

For example, the ubiquitous "No Step" to prevent non-load bearing parts from being stepped on.

The eye catching "Remove Before Flight" to alert crew, at a glance and from some distance, that there's something they forgot to do.

Or by simply painting the plugs and sockets unique colors. RCA jacks use this to differentiate between video and left and right audio.

If a plug has a particular orientation, color both the plug and socket the same on just one side. When in alignment, the colors line up. When out of alignment, they do not. This helps to both guide the user to the correct orientation, and to alert anyone to an incorrect one.

This also helps screw in connectors. Paint lines or dots on the plug and socket. When they line up, you're firmly seated. When they don't line up the fitting is not correct. This guides correct fitting without the need for fancy tools like a torque wrench, and also visually alerts the user that a fitting is incorrect, or might have worked its way loose.

---

The multitude of techniques to avoid incorrect connections, or make them obvious, speaks just how costly the problem is to every industry that needs to connect two pieces of hardware.

Answer 5

First time answering here, so let me know if I need more detail. I worked as an engineer in the avionics testing lab for Space Shuttle for about 5 years, and I have personally witnessed incorrect cables connected multiple times, sometimes resulting in permanently damaged equipment. It's actually not that difficult to mix up your cabling.

With most of the Multiplexer/DeMultiplexers (MDMs) and other boxes, there were many cable connections, but really only a handful of unique connector shapes/formats--less than 6 types if I remember correctly--for hundreds of connection points. We would routinely disconnect cables to put in a break out box for checking voltages whenever there was an errant signal detected during simulation testing. There was always a technician and QA person on hand to do the work, with detailed instructions from the systems engineers. Even with all the training, detailed instructions, and multiple sets of eyes on the task, errors happened. A couple incidents come to mind:

--Once, while the lab was powered down, a technician disconnected a cable from a breakout box and connected it back to a flight rated MDM that we were burning in after some maintenance. QA checked the serial numbers on the cable afterward and found they were wrong. Turns out the tech took the cable off from a different break out box that was installed close by for unrelated troubleshooting (the boxes look the same, so easy mistake to make). Same connector on both cables though, so it physically fit and looked the same. He swapped the cables back, got the right one that time, and we all called it a day. Next day, we power up the lab, and the first test leads to multiple failures. Finally I was able to trace it down to the same MDM on the same connector. I remembered the wrong cable error and checked what was coming from that source. Sure enough, the pin on that cable that connected to the erroring signal was a 24V always on power source. We fried the MDM. Oops.

--Another incident called for a break out box to be installed on another MDM, and the tech had some trouble getting the connector to fit. It was the right size, but it seemed like there was a pin out of alignment that was getting it hung up. He fiddled with the pins a bit and tried again, and with a little extra effort got it to connect and lock. Testing begins, ends, and the box is removed. Next time we power up, we experience more errors. Turns out that while the connector was the same, the pin pattern was different, and he had grabbed the wrong cable from the shop. So a pin on the MDM was mashed down completely and couldn't be seated in the actual cable when it was reconnected. They tried to carefully bend the pin back out, but it snapped off, and since we were only a few flights away from the end of the program and it wasn't a critical signal that was broken, the decision was made to just live with it.

So in my 5 years, I personally saw it happen at least twice that I remember, and saw the incident reports of it happening quite a few more times, just in the one lab I worked at. As for why the same connectors are used, I can tell you that we had to have 4 sets of every type of cable in our lab for testing and troubleshooting, and I'm sure other labs had similar requirements. Spare cables multiply cost with every unique type. Plus those connectors each have to be flight rated, which is a time consuming and costly process. Better to do it just a handful of times rather than hundreds of times for unique connectors.

Answer 6

It is better to use keyed or polarized cable connectors that could not be inserted to a wrong socket or in a wrong rotation.

But what should be done with the second stage of the Saturn V used for Apollo 6? When a fuel line ruptured, two instead of one of the five engines were shutdown because some cables were switched between two engines.

So the connectors should be keyed not to be switched between several engines. But a test stand should be able to test all engines used for a stage. Of course switching between connectors of the same engine should be impossible too, in a test stand as well as a rocket stage.

So keying of connectors should prevent switching cables within the same engine on a test stand and between several engines of a stage but allow testing of all engines in a test stand without needing adapters for all engines of the same stage.

Testing a valve used at several positions within the same engine should be possible with the same test equipment but wrong connections within the same engine as well as several engines should be prevented.

Keying of connectors should not increase the number of adapters needed for test during manufacturing and pre launch testing.

Keying should be possible for up to 16 connectors in each of up to 16 engines, so we need at least 8 bit for keying. With some reserves there should be 10 to 12 bits. So careful design and documentation as well as recording of thousands of keying combinations is necessary.

Answer 7

In more conventional civillian design, otherwise identical parts are often made hot swappable and truly identical, it just means less things to keep track of. Cables and connectors are, in my experience with non-critical systems, highly likely failure points, just like anything else that moves.

When aerospace people talk about the certainty and reliability of simple, mechanical things, one can only assume they mean mechanical things of excellent quality. They're not like your average cheap USB. Those cables probably cost a LOT, and may have been just as hard to design as the electronics. They probably don't want to mess around too much there, because that could get expensive.

Different cable lengths certainly seem reasonable, but not when everything is the same distance.

Rather than physically different connectors, digital control signals can be used. However, this means that the actuators must be correctly programmed with the right ID for the location. Swap those, and the system won't be able to tell what's going on.

In fact, even keyed connectors could be accidentally swapped, if the entire device is installed in the wrong place. Then you need keyed mechanical attachment as well.

If a cheap 9DOF sensor is installed in every motor, and the code can be done safely(Perhaps with a fully isolated set of wires and controllers!), a lot can be done to verify that everything is in the right orientation, and you could quite possibly do this cheaper than adding more connector types. This is what I would do, although my engineering skills are not aerospace grade.

But even then, the code that verifies it could be backwards!!

QC is incredibly hard, and there are people (like me) who cannot reliably tell left from right without really thinking about it. Swaps like this can occur anywhere in the whole chain, and get expensive to stop in every possible place. There is currently no substitute for testing, having lots of different pairs of eyes, and being willing to speak up about things.

Airlines use Crew Resource Management to study how to prevent mistakes, it's a whole science.

So as far as I'm concerned, it should have been caught by manual check, AND by digital data verification, itself based on peer reviewed parameters. Most big disasters aren't one identifiable mistake, they're a whole series of them, and each one is more or less equally to blame.

Answer 8

Another solution: Instead of physically keying cables, include an extra wire in the cables that is used to communicate with a tiny bit of electronics that simply outputs an ID code. That lets the device on the other end query what it's connected to and report back if it's wrong.

Once confirmed during basic electrical systems checkout after assembly, there would be no further reason to continue reading, so no negative contribution to thing that can cause launch delays on the pad.

Answer 9

I think two-wire reversals are not uncommon, likely to be found from time to time with very minor consequences, e.g. two heaters are swapped with respect to their command codes.

However, as everyone loves a photo here is a bigger event: 2013 Proton failure

and a little explanation:

The statement appears to confirm earlier Russian press reports that investigators combing through the debris at the Baikonur Cosmodrome spaceport in Kazakhstan found evidence that several sensors were installed upside down. The Roscosmos statement said investigators found evidence that several of the sensors found at the crash site showed signs of a forced assembly, which would be consistent with their being installed at 180 degrees from their correct position.