# How has NASA's mission operations software architecture evolved?

As many of us know, the mission operations software of NASA has changed significantly over time. Needs have changed with the space program and mission operations software has adapted to keep up with these demands as well as the current state of the industry. However, mission operations software is not a monolith. It is composed of many different parts, and they need not all change at the same pace. If it ain't broke...well, you know the saying.

That said, what is the oldest software architecture in use in mission operations today? What is the newest (will accept deployed and high potential to-be-deployed)? Are there any trends in architectural changes over the last 20-30 years?

(If necessary to constrain the scope, answers should prioritize JSC and chiefly Mission Control.)

• As an anecdote, the majority of GUI programs for Flight Ops are now programmed using Python and interfacing with pub/sub telemetry streams from the main servers. – CourageousPotato Sep 4 at 4:25
• @CourageousPotato That doesn't surprise me. Thanks for sharing! – called2voyage Sep 4 at 14:53

The biggest single change in the Houston Mission Control Center (MCC) occurred in the late 1990s. This was the change from a mainframe based system architecture to a workstation based system with a client/server architecture.

The project was led by John Muratore and it's discussed at some length in his oral history. Several of the papers in the Control Center Technology Conference Proceedings also discuss the upgrade. (This document is the source of the illustration).

The switchover was gradual with the new system first being used in a following mode, then gradually assuming control function-by-function. The last mainframe computer was not removed until 2002. (Building 30 Historical Documentation page 17)

• Reading through part of the conference proceedings now. Very enlightening! – called2voyage Sep 3 at 23:36
• I remember CLIPS, it's one of the things that I referred to about not working well. (I was a console jockey not a MCC developer, so there may have been stuff going on behind the scenes/screens that I am completely ignorant of.) – Organic Marble Sep 4 at 0:49
• Regarding AI, that transition happened shortly before one of many AI winters. Mission ops personnel developed a good number of ART/CLIPS-based expert systems. Those all fell to the wayside when the AI winter hit. By its nature, AI is a multidisciplinary field. When budget cuts come (and they always do), it's the multidisciplinary efforts that take the brunt of the hit. That definitely happened with regard to the use of AI in Mission Control. – David Hammen Sep 4 at 8:00
• My project was one of the victims of that AI winter. At that time I was tasked with identifying opportunities for uses of advanced computer science concepts in Mission Control. Muticomputing (a bunch of computers sharing data) and expert systems (e.g. ART/CLIPS/GenSAA) were explicitly excluded; NASA was well aware of those techniques. I focused on signal processing and pattern recognition. I saw lots of opportunities for those techniques in identifying unexpected variations in device start-up / shut-down transients and in steady-state operations. – David Hammen Sep 4 at 8:24
• Despite having saved a Shuttle mission due to quickly recognizing a failure in the Shuttle's Regenerative CO2 Removal System, the project I championed was canceled when the AI winter hit. My software did not save a flight, but it did help save a mission. Saving a flight means people don't die. Saving a mission means the mission doesn't have to be cut short so as to enable the vehicle to return to Earth. – David Hammen Sep 4 at 8:30

I had the distinct honor of working for John Muratore on two distinct projects. One was the X-38. But before that, I worked on developing advanced software for his Real Time Data System project.

The RTDS project started in 1986 when Muratore saw the archaic structure of Mission Control. The primary goal of the project was to transition from a single mainframe computer that drove dumb terminal displays at Mission Control consoles to a distributed system comprising many minicomputers and data servers, each of which ran their own sets of software, that displayed data to users on non-dumb terminals, and that published data to / subscribed data from other applications. The project took off in part because it had the explicit backing of Eugene Kranz.

One challenge was the completely fractured nature of computing at that time. Organic Marble's answer has a graphic that claimed "RTDS will run on almost any UNIX workstation". That made me chuckle.

This was the age of the minicomputer. Nothing was standardized. UNIX had not yet been standardized. The UNIX library calls never were the same from one vendor to the next. The C programming language had not yet been standardized. Every vendor provided its own compiler that had its own interpretation of what C was, and had its own library of standard functions that were anything but standard. The manner in which computers stored and transmitted data had not been standardized. Consider the 32 bit hexadecimal number 0x12345678. Some computers would store this just like that (little endian). Others, would store that number as if it were 0x87654321 (big endian). Yet others, as if it were 0x78563412 (mixed endian). Back then one had to expect every random order when switching to a different computer.

The reason RTDS "ran on almost any UNIX workstation" was because John continuously wrangled with vendors to supply the project with "free" computers for short trial periods. We had to be agile well before the concept was known elsewhere. But this also made our code get bloated with #if defined computer_X do_this #elif defined computer_Y do_that ... constructs because every computer was different back then.

My contribution to the project was in the form of AI software than did not use expert systems. I had been charged with identifying opportunities for using AI in Mission Control, but they didn't need me to recommend expert systems. NASA already knew about that. I focused on signal processing and pattern recognition. One system identified what electrical devices had been turned on or off based on transient signatures. My software could easily detect when the cabin fan properly turned on or off, when someone had flushed the properly functioning toilet, when someone powered up the properly functioning vacuum cleaner, or turned on or off a properly functioning scientific experiment. But it had less success if the equipment was malfunctioning. So that was a failed idea.

But I did have one key success. Extended Duration Orbiter (EDO) missions used a Rube Goldberg device called the Regenerative CO2 Removal System to remove carbon dioxide from the cabin atmosphere. The device went through a ridiculously complex sequence of operations that over the course of time accumulated carbon dioxide from the cabin atmosphere and later vented that CO2 to vacuum. There was a chance of a very bad failure at several key points in this ridiculously complex sequence of operations where cabin atmosphere instead of accumulated CO2 could have been vented to vacuum.

The RCRS had built-in failure detection software that looked for these failure modes. In the case that anything suspect happened, the RCRS placed all of its valves in safe mode, and then erased all of its memory. If the machine did shut itself down, there was no way to query the machine regarding why it did so. With multiple failure points, diagnosing why the machine had shut itself down would have been challenging.

The machine did not send internal state information, but it did send sensor readings. I was tasked with writing a pattern recognition application that made sense of these limited data. My app ran silently for several EDO flights. Years later, in 1998 (STS-90), it pinged loudly, a few seconds before the machine shut itself down.

My application saved Mission Controllers eight hours of diagnosis time. My application most certainly did not suggest the brilliant solution they came up with. It simply told them "the device failed HERE". They quickly verified that that diagnosis was correct. And then they went to work coming up with a way around the problem.

The so-called "Shuttle Flight Data File" included, amongst other non-booklike things, multiple rolls of duct tape. The arrived-at solution used that duct tape.

• Lesson learned with regard to aluminum tape (aka duct tape) and plasticized gray tape (aka duck tape): Never leave home (or Earth) without them. And don't leave your home without them, either. – David Hammen Sep 6 at 11:43
• Great stuff. I was involved with writing the SMS model of the RCRS. Ridiculously complex it was. I worked in SE&I when Muratore ran that; my memories of him are not as positive. – Organic Marble Sep 6 at 12:02
• @OrganicMarble - Muratore was an <expletive deleted>, no doubt. Hard. Harsh. Sometimes downright nasty. When you stepped into the building you knew exactly where John was based on all of the yelling. But he was also very good in the sense that he shielded his team members from the wackiness of NASA upper management. – David Hammen Sep 6 at 12:07
• That IFM for the RCRS was a classic, too. When I took IFM training one of the classes (the best class) had you go through the procedure on the engineering model of the RCRS that was installed in one of the vacuum chambers. It bypassed a failed sensor IIRC. – Organic Marble Sep 6 at 12:40
• Thank you so much for writing this answer. This is the kind of detailed account I was hoping for! – called2voyage Sep 6 at 14:03