3
$\begingroup$

I wonder whether there are, or are recognizable in near future, industry specifics for CI/CD solutions in a space engineering company like SpaceX, compared to other companies where mass manufacturing plays a role, for example to car (supplier) companies?

Terms:

  • CI/CD - continuous integration and delivery to support continuous improvement of software
  • DevOps - body of methods, practices and also a mindset to leverage shorter delivery cycles of software; DevOps is agnostic with regards to project methodology, programming language and operation system1.

Topicality:

While this is not directly about space exploration, this topic does have an impact; imagine following use cases:

  • your custom metal 3D printer, or a robotic assembler gets stuck due to some software error. How long does it take to deliver the patch? What is acceptable outage time?
  • you should secure accurate operation of your mission software by sufficient acceptance tests which include automated testing like test coverage2. Do you need awareness through measurable facts here, while ECSS compliance requires 100% test coverage? How much is enough/a must for human spaceflight?

Research effort:

My research so far using these sources of information:

  • the stackshare.io site (which does not need to be the source of current, correct and complete data, still somebody has published it once)
  • LinkedIn search for peoples' titles and expertise around CI/CD and DevOps
  • Job sites of major companies

What I have found so far:

  • SpaceX does not list a CI/CD system. Ok they look currently for a build/release software engineer.
  • BlueOrigin lists even less on StackShare; interestingly they use CloudFlare (DDoS protection). According to this position, they seem also to start thinking in this direction.

Following from the wording of the job descriptions and other sources, there is no (enteprise scale) CI/CD yet there, neither DevOps is a big topic yet. But they move in this direction, so there is no simple "no" answer I think.

There is also no simple strategy like "implement your own", or "take product X" because these systems being around since about 20 years are still in their development; for example, the issue and requirements board aka backlog of a major open source product counts about 30000 items! Possibly, many of them reflect common topics, but on this scale one can expect that some will be also industries specific or related.

For sure, in the beginning there must be a well-engineered concept, a CI/CD system is after all a smart helper3.


Footnotes

1 For example, CI/CD is used to build and test Kubernetes "with a need to execute over 10,000 CI/CD jobs per day across over 100 repositories of code". Also, Linux community has decided to use CI/CD for the Linux kernel.

2 Quote from the linked source about Margaret Hamilton: "without that software, not to mention Hamilton's well-documented insistence on rigorous testing, humans would've never reached the moon safely".

3 ".. Uber’s self-driving technology hadn’t been programmed to detect jaywalkers at all"

$\endgroup$
  • $\begingroup$ Could you define some of your terms? I don't know what "CI/CD" or "DevOps" mean in this context $\endgroup$ – Dragongeek Nov 22 '19 at 11:18
  • 1
    $\begingroup$ They almost certainly use the exact same tools that everyone else uses for these tasks. The tools are flexible, and their requirements are so specialist that they'd need to go through all the hassle of rolling their own solution. $\endgroup$ – Starfish Prime Nov 22 '19 at 11:21
  • 1
    $\begingroup$ @Dragongeek its probably a sign that the question might be a bit off-topic here. They are software development jargon, and come larded down with a lot of context and idiom to the point where they can't really be described briefly enough to fit in the question. $\endgroup$ – Starfish Prime Nov 22 '19 at 11:22
  • 1
    $\begingroup$ @StarfishPrime there are about 30 questions on this site using the keyword compliance; so I think it's time to introduce it. By that, the question is not directly on software exploration, but about compliance regarding its software part. $\endgroup$ – J. Doe Nov 22 '19 at 12:01
  • 7
    $\begingroup$ I'm voting to close this question as off-topic because it is about software development and not space exploration. $\endgroup$ – GdD Nov 22 '19 at 12:59
4
$\begingroup$

The question assumes there is a concept of "Continuous Delivery" in space flight software. There is not. The required product assurance processes do not allow it.

I'll consider ECSS standards here, as those are what I have experience with. Here you can find an introduction on the applicable software standards.

ECSS defines 4 criticality levels of space flight software:

Table 4-1 from ECSS-Q-ST-30-02C (Table 4-1 from ECSS-Q-ST-30-02C)

Basically, any software that is classified as criticality level A, B, or C is subject to stringent product assurance standards. This means a lot of testing, including testing on the actual flight hardware.

It is this last point that make continuous delivery impossible: while you could do some CI/CD during the software development phase (between requirements baseline and qualification testing), flight hardware is insanely expensive and there's often only one actual spacecraft that you cannot jeopardise.

Therefore development cycle for flight software looks like this:

  1. Derive software requirements
  2. Review requirements
  3. Design software (not develop)
  4. Review design
  5. Develop software according to requirements (you could do some CI/CD here), but you'd be delivering to yourself, or your QA department)
  6. Create software version (e.g. v1.0)
  7. Review v1.0
  8. Qualification testing of v1.0 (in simulation, Processor-In-The-Loop, Hardware-In-The-Loop, etc.; see David Hammen's answer)
  9. Review test results
  10. Accept v1.0

From step 8 your software starts interacting with multi-million dollar/euro hardware, so there better not be any errors in it at this point.

There's no iteration in this process: every step is followed by a review and everything has to be traceable. All this to minimize the chance of a software failure (with associated consequences!) in flight.

In some cases, e.g. very long missions to the outer solar system, multiple versions of the software are delivered and uploaded to the spacecraft to speed up the launch process: i.e. at launch the spacecraft has a minimum software version to enable the launch and the full version is uploaded somewhere along the way. This is rare though, as it introduces a huge risk.

Now, as I mentioned, the above rigorous processes apply to level A, B, and C. Level D software still needs to be of good quality, but software at this level is considered non-critical in the sense that nobody gets hurts or millions get lost if it fails. Cubesat software is often classified at this level, because cubesat software only gets activated in orbit where it cannot do a lot of damage. This is why you can see open-source software like FreeRTOS on a cubesat.

At this level, you could use CI/CD, especially if you are mass-producing cubesats. Then you can just follow the regular development processes for embedded software. I don't see any particular requirements with respect to the CI/CD pipelines to be necessary, so I'd imagine they'd use whatever they see fit. Nonetheless, predictability and traceability is still key: imagine a group of students buying a cubesat and seeing it bricked because some developer pushed some change and broke the build. Therefore, I don't think the development cycles you see in "terrestrial" applications will ever become commonplace in space flight software development.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Well, I heard the Indian moon lander used continuous delivery, that there was a software upload a few hours before the landing attempt $\endgroup$ – JCRM Nov 22 '19 at 20:54
  • 1
    $\begingroup$ @JCRM Uploading one or two new versions after launch is not "continuous delivery" in the sense implied by the OP. $\endgroup$ – Ludo Nov 22 '19 at 20:56
  • $\begingroup$ @Ludo actualy I've been thinking about it but thought people might want to crucify me with downvotes for such suggestions :-) CD means also delivering to operation environment. I like your answer anyway! Question: why is the simulation in step 8 not a separate step, i.e. simulation in a simulated physics environment to catch - to a possible/feasible extent - "most stupid" errors or unforeseen side effects? this would close the CI loop before doing hardware acceptance tests if anyhow possible before launch. So also the activities allowed but the risky updates you've mentioned might be simulated $\endgroup$ – J. Doe Nov 22 '19 at 22:17
  • $\begingroup$ @J.Doe The process is simplified, with multiple test phases with associated reviews and possibly rework. The point I wanted to get across is that there's no cycling back, so CD doesn't really make sense in my opinion (at least, I associate CD with increasing functionality at each delivery, in a agile development setting). $\endgroup$ – Ludo Nov 22 '19 at 22:41
  • $\begingroup$ funny, I associate continuous delivery, in an agile development setting, with stuff breaking and that breaking being an accepted risk. $\endgroup$ – JCRM Nov 25 '19 at 9:48
3
$\begingroup$

You are using web site development terms. While some of those concepts do indeed apply to the development flight software, some do not, and some are so far off as to make "completely off base" a minor criticism. In this post I'll describe some of things that can be done to test whether the software running on a spacecraft is doing what it is supposed to do.

Unit testing

Some chunks of flight software are nicely amenable to unit testing. Some however are not. How do you unit test a Kalman filter? A Kalman filter takes a large number of inputs, some of which may or may not be present at some point in time, and eventually over the course of time arise at a (one would hope) improved estimate of some aspects of the spacecraft's state. Kalman filters do not act as a unit. It is their behavior over the course of time that one needs to test.

Unit tests typically form the bulk of a CI/CD pipeline. With flight software, this is not even close to enough.

That said, having automated unit tests where appropriate, and having a CI/CD pipeline that automatically performs these tests with every push or with every merge request to master/develop is a very nice thing.

Non-realtime software in the loop testing

Here one tests how the flight software might behave as if it was running on a modern (fast) computer and as if it was not constrained by realtime concerns. This form of testing is very nice for (for example) testing whether the admittedly ad-hoc tuning parameters in a Kalman filter are behaving as expected.

This is also a nice environment for testing whether the spacecraft's fault detection, isolation, and recovery software works as expected. The tester can throw rocks galore at the flight software to see how it responds.

This kind of testing cannot be done in a realtime environment as failures are (or should be) far too infrequent. This kind of testing also is not what one wants to do in a CI/CD setting. Thousands of runs (and perhaps tens of thousands runs) need to be performed.

Realtime software in the loop testing, aka hardware out of the loop testing

Here the flight software runs as a separate process on a non-representative piece of hardware, but does so as a realtime process. A multiple day test is not conducive to CI/CD. Nonetheless this kind of testing is important because the next stage is even more intensive.

Realtime hardware in the loop testing

Here the flight software runs as a separate process on a representative piece of hardware, and does so as a realtime process. This is where the rubber meets the road. This testing is also ridiculously expensive as each representative piece of hardware costs multiple hundreds of thousands of dollars apiece.

| improve this answer | |
$\endgroup$
  • $\begingroup$ thank you for this elaboration! As I understand the summary is "CI/CD is mostly not appliable i.e. not enough in space systems engineering context". Not enough means though to me "there are also other methods beyond it" which is beyond the question scope. Still there is ECSS and by the way, the Kubernetes' CI/CD indeed runs those 10K jobs daily. Also Linux kernel community has decided to introduce CI/CD. So I do not understand your reference to the limited scope of web site development. $\endgroup$ – J. Doe Nov 22 '19 at 13:15
  • $\begingroup$ Terminology is irrelevant -- every new "method" creates its own lexicon. CI, Kaizen, 5S, Agile, you name it. What it all comes down to is: do the people involved interact sufficiently and respect each other's ability sufficiently? $\endgroup$ – Carl Witthoft Nov 22 '19 at 13:26
  • $\begingroup$ @J.Doe - I would rate CI/CD as a "good start" (maybe) for a hard realtime system where one slip of the software jeopardizes hundreds of millions to multiple billions of dollars, and perhaps even human lives. Suppose a programmer mistakenly screws up a web site and erroneously causes a 404 error. The cost of that mistake is zero lives, and perhaps a few hundreds of thousands of dollars. Now suppose a programmer mistakenly screws up the controls on a spacecraft (or the controls on a medical device). The cost of that mistake might be in the billions of dollars and multiple human lives. $\endgroup$ – David Hammen Nov 22 '19 at 13:27
  • $\begingroup$ so while the question about industry specific requirements to a CI/CD which after all helps to automate testing and reproduce errors, you mean it has to be quite high end? indeed, one might want to run a pipeline in a supercomputing environment, which requires support in the CI of such environments. $\endgroup$ – J. Doe Nov 22 '19 at 13:31
  • 1
    $\begingroup$ @J.Doe - You're chasing the wrong problem, at least for how. For now, the problem of space debris is a much, much lesser concern than is the problem of a spacecraft's flight software going. The problem of collisions with space debris remains rare and is is very predictable.The problems that result from flight software and/or flight hardware going batsh*t crazy by far remain the dominant causes of spacecraft failures. $\endgroup$ – David Hammen Nov 22 '19 at 14:29

Not the answer you're looking for? Browse other questions tagged or ask your own question.