Which CI/CD system do/would space companies like SpaceX use for continuous integration in their devops practice? [closed]

Question

I wonder whether there are, or are recognizable in near future, industry specifics for CI/CD solutions in a space engineering company like SpaceX, compared to other companies where mass manufacturing plays a role, for example to car (supplier) companies?

Terms:

• CI/CD - continuous integration and delivery to support continuous improvement of software
• DevOps - body of methods, practices and also a mindset to leverage shorter delivery cycles of software; DevOps is agnostic with regards to project methodology, programming language and operation system¹.

Topicality:

While this is not directly about space exploration, this topic does have an impact; imagine following use cases:

• your custom metal 3D printer, or a robotic assembler gets stuck due to some software error. How long does it take to deliver the patch? What is acceptable outage time?
• you should secure accurate operation of your mission software by sufficient acceptance tests which include automated testing like test coverage². Do you need awareness through measurable facts here, while ECSS compliance requires 100% test coverage? How much is enough/a must for human spaceflight?

Research effort:

My research so far using these sources of information:

• the stackshare.io site (which does not need to be the source of current, correct and complete data, still somebody has published it once)
• LinkedIn search for peoples' titles and expertise around CI/CD and DevOps
• Job sites of major companies

What I have found so far:

• SpaceX does not list a CI/CD system. Ok they look currently for a build/release software engineer.
• BlueOrigin lists even less on StackShare; interestingly they use CloudFlare (DDoS protection). According to this position, they seem also to start thinking in this direction.

Following from the wording of the job descriptions and other sources, there is no (enteprise scale) CI/CD yet there, neither DevOps is a big topic yet. But they move in this direction, so there is no simple "no" answer I think.

There is also no simple strategy like "implement your own", or "take product X" because these systems being around since about 20 years are still in their development; for example, the issue and requirements board aka backlog of a major open source product counts about 30000 items! Possibly, many of them reflect common topics, but on this scale one can expect that some will be also industries specific or related.

For sure, in the beginning there must be a well-engineered concept, a CI/CD system is after all a smart helper³.

---

Footnotes

¹ For example, CI/CD is used to build and test Kubernetes "with a need to execute over 10,000 CI/CD jobs per day across over 100 repositories of code". Also, Linux community has decided to use CI/CD for the Linux kernel.

² Quote from the linked source about Margaret Hamilton: "without that software, not to mention Hamilton's well-documented insistence on rigorous testing, humans would've never reached the moon safely".

³ ".. Uber’s self-driving technology hadn’t been programmed to detect jaywalkers at all"

Answer 1

The question assumes there is a concept of "Continuous Delivery" in space flight software. There is not. The required product assurance processes do not allow it.

I'll consider ECSS standards here, as those are what I have experience with. Here you can find an introduction on the applicable software standards.

ECSS defines 4 criticality levels of space flight software:

(Table 4-1 from ECSS-Q-ST-30-02C)

Basically, any software that is classified as criticality level A, B, or C is subject to stringent product assurance standards. This means a lot of testing, including testing on the actual flight hardware.

It is this last point that make continuous delivery impossible: while you could do some CI/CD during the software development phase (between requirements baseline and qualification testing), flight hardware is insanely expensive and there's often only one actual spacecraft that you cannot jeopardise.

Therefore development cycle for flight software looks like this:

1. Derive software requirements
2. Review requirements
3. Design software (not develop)
4. Review design
5. Develop software according to requirements (you could do some CI/CD here), but you'd be delivering to yourself, or your QA department)
6. Create software version (e.g. v1.0)
7. Review v1.0
8. Qualification testing of v1.0 (in simulation, Processor-In-The-Loop, Hardware-In-The-Loop, etc.; see David Hammen's answer)
9. Review test results
10. Accept v1.0

From step 8 your software starts interacting with multi-million dollar/euro hardware, so there better not be any errors in it at this point.

There's no iteration in this process: every step is followed by a review and everything has to be traceable. All this to minimize the chance of a software failure (with associated consequences!) in flight.

In some cases, e.g. very long missions to the outer solar system, multiple versions of the software are delivered and uploaded to the spacecraft to speed up the launch process: i.e. at launch the spacecraft has a minimum software version to enable the launch and the full version is uploaded somewhere along the way. This is rare though, as it introduces a huge risk.

Now, as I mentioned, the above rigorous processes apply to level A, B, and C. Level D software still needs to be of good quality, but software at this level is considered non-critical in the sense that nobody gets hurts or millions get lost if it fails. Cubesat software is often classified at this level, because cubesat software only gets activated in orbit where it cannot do a lot of damage. This is why you can see open-source software like FreeRTOS on a cubesat.

At this level, you could use CI/CD, especially if you are mass-producing cubesats. Then you can just follow the regular development processes for embedded software. I don't see any particular requirements with respect to the CI/CD pipelines to be necessary, so I'd imagine they'd use whatever they see fit. Nonetheless, predictability and traceability is still key: imagine a group of students buying a cubesat and seeing it bricked because some developer pushed some change and broke the build. Therefore, I don't think the development cycles you see in "terrestrial" applications will ever become commonplace in space flight software development.

Answer 2

You are using web site development terms. While some of those concepts do indeed apply to the development flight software, some do not, and some are so far off as to make "completely off base" a minor criticism. In this post I'll describe some of things that can be done to test whether the software running on a spacecraft is doing what it is supposed to do.

Unit testing

Some chunks of flight software are nicely amenable to unit testing. Some however are not. How do you unit test a Kalman filter? A Kalman filter takes a large number of inputs, some of which may or may not be present at some point in time, and eventually over the course of time arise at a (one would hope) improved estimate of some aspects of the spacecraft's state. Kalman filters do not act as a unit. It is their behavior over the course of time that one needs to test.

Unit tests typically form the bulk of a CI/CD pipeline. With flight software, this is not even close to enough.

That said, having automated unit tests where appropriate, and having a CI/CD pipeline that automatically performs these tests with every push or with every merge request to master/develop is a very nice thing.

Non-realtime software in the loop testing

Here one tests how the flight software might behave as if it was running on a modern (fast) computer and as if it was not constrained by realtime concerns. This form of testing is very nice for (for example) testing whether the admittedly ad-hoc tuning parameters in a Kalman filter are behaving as expected.

This is also a nice environment for testing whether the spacecraft's fault detection, isolation, and recovery software works as expected. The tester can throw rocks galore at the flight software to see how it responds.

This kind of testing cannot be done in a realtime environment as failures are (or should be) far too infrequent. This kind of testing also is not what one wants to do in a CI/CD setting. Thousands of runs (and perhaps tens of thousands runs) need to be performed.

Realtime software in the loop testing, aka hardware out of the loop testing

Here the flight software runs as a separate process on a non-representative piece of hardware, but does so as a realtime process. A multiple day test is not conducive to CI/CD. Nonetheless this kind of testing is important because the next stage is even more intensive.

Realtime hardware in the loop testing

Here the flight software runs as a separate process on a representative piece of hardware, and does so as a realtime process. This is where the rubber meets the road. This testing is also ridiculously expensive as each representative piece of hardware costs multiple hundreds of thousands of dollars apiece.