8
$\begingroup$

Due to the increasing availability of large quantities of data and computational power, machine learning (ML) is providing increasingly better results in various fields. Much research has been done on robotic perception and control with various ML algorithms, including navigation, and control in stochastic and partially observed environments. Model-free control.

Of particular interest to me, Deep Reinforcement Learning (DRL) has been applied to various control task in simulations. Here is one example to achieve the optimal control for a rocket landing with model-free DRL. As far as I am aware however, there are not many examples of ML being used for actual rockets or spacecraft navigation and controls.

  • Are classical approach and more sophisticated optimal or robust control still better than current ML algorithms when it comes to implementation for space vehicles?

  • Did ML algorithms used in combination with classical control methods or to generate guidance trajectories for any space mission?

$\endgroup$
6
$\begingroup$

This is an attempt at the first part of your question: "are classical control algorithms better than machine learning approaches". Since I am personally more interested in launch vehicles, the answer is mostly about potential launch guidance applications with fuel-efficiency and safety in mind. I think it's okay because launch vehicles are the biggest areas to apply control theory (I'm convinced that deep space probes do not have very complex GN&C modules, their trajectories are calculated and optimized on the ground - if this is wrong, please let me know).

Regarding the second part ("was ML used to guide any real mission") I have no knowledge of it. I have done a good bit of reading on guidance algorithms when implementing UPFG and didn't encounter a single application of ML. Also, there is not much publicly available information on the most modern algorithms (classified), so even more difficult to say. I would welcome any supplementary information on this.

Now, onto the wall of text (sorry, it just turned out larger than I initially planned).

Introduction: machine learning vs control theory

I don't assume everyone here is knowledgeable in ML and CT, so just a few words to get everyone on the same page.

Machine learning (ML) is a subfield of computer science and statistics, with the key idea of having an algorithm learn how to perform some task from data samples, instead of explicitly programming it. Typically, in a supervised setup, we have a set of example inputs and corresponding correct outputs (which we know the algorithm should give). The ML system then learns how to transform those inputs into the right outputs. This is very useful if the task at hand is extremely difficult to solve explicitly, but there exists a lot of data we could learn from (e.g. computer vision, language translation).

Reinforcement learning (RL) is a more specific kind of ML, in which the algorithm, instead of receiving input-output pairs, interacts with some enviroment in order to learn to perform a given task as good as possible. In this setup we don't have data samples per se, but instead there exists some active environment that reacts to the agent's actions dynamically (examples include computer games, robotics, control systems).

Control theory (CT) is a very broad branch of mathematics related to, well, controlling dynamical, usually continuous systems. There is a given object that we can observe and interact with, and a controller which, given the observed signal and a desired state, generates control commands that are fed back into the object. The goal is to make the object reach this desired state, but we could also want to reach it as fast as possible, or be robust to some uncertainties etc. CT relies on finding a model of the object (transfer function) that describes how does it respond to some input, and then constructing a controller specifically for this particular object.

CT-based spacecraft guidance

All spacecraft guidance systems known to me use guidance basing on some variant of control theory (although I am just a hobbyist in this aspect, so a quote from some actual expert could be insightful). The rocket vehicle is the object; we can control it using engine gimbals and sometimes also throttle, and we can measure things like position, velocity, angular momentum etc. The real problem lies in the mathematics of vehicle's behavior and interactions between the vehicle and the environment. The gravity changes with altitude and so does performance of the engines, the vehicle's mass changes with time (and throttle), position and velocity depend on Newton's laws of dynamics. It's difficult because this system is dynamic - the entire previous trajectory influences the future state. This means that what you get doesn't only depend on what you do, but also on what you have done previously, forcing you to plan ahead.

The CT approach requires one to formalize all those interactions in a set of mathematical equations. One then next has to sort of invert these equations in order to find such a form that outputs a control signal given the desired and known state of the vehicle. Without getting deep into the intricacies of how it's done: bottom line is that you describe your system with mathematics which you then manipulate to find an answer to the problem (optimal guidance). If you want to read more, I have written a somewhat elaborate explanation of the Space Shuttle's guidance algorithm.

General problems with ML

A machine learning system can only get as good as the data you give it. There is no other source of knowledge - so if the algorithm hasn't encountered some situation during learning, you can be fairly sure that it will be unable to deal with similar cases later. This leads to another problem: how do we know that the data we have is a perfect representation of the actual phenomenon?

As previously outlined, a RL system learns from interacting with environment. It starts pretty much with trial and error, trying out various strategies, observing how does the environment respond to its actions, until it (hopefully) finds the most efficient way to do the task. Since learning is an iterative process that takes hundreds of thousands of attempts, it is obviously impractical to let an algorithm learn in actual flight conditions. The only way to make this work would be through simulated environments.

This brings us to the crucial problem. The toy example with a 2D rocket with 7 sensors and 3 control signals is an obviously inadequate simulation of the real world - a ML algorithm trained in such a "world" would not perform any well in the real-world scenario. It would have learned a different problem than what we'd actually ask it to do. So the question is: can we make a simulation with enough fidelity that an algorithm trained on it would successfully transfer the knowledge onto the real-world case?

Explainability

The data you train your algorithm on has to be as close as possible to the environment that it is going to interact with in practice. The more it deviates, the more unpredictable the behavior of your system is going to be. Before you put your ML system in charge of a real launch vehicle, you will be asked: "okay, it did well in simulation, but will it crash the $200M worth of vehicle & payload"? And if you were honest, you would have to answer: "well, I don't know, because so far it has only flown virtually, and a real flight is more or less unknown grounds for it".

This shows the greatest, and here IMHO the most important, difference between ML and CT. With a control-theoretic guidance, what you get is a set of equations. You can run all sorts of analysis on them (e.g. check its response to distortions, stability etc.), you can predict the possible outcomes, and most importantly: should anything go wrong, you can just backtrack through the math and see on which level has the mistake been made, and correct it.
When a machine learning system gives you an answer, you essentially can either take it or leave it. And they sometimes do give rubbish answers - for an extreme example, check out adversarial examples, samples designed to "cheat" neural networks. Modern ML systems consist of tons and tons of linear algebra that does not directly encode the behavior - that is somewhere in the thousands of anonymous parameters which change during training. Their structure is just too huge and too general to be effectively analyzed, just like you cannot practically cut someone's brain open and look at the individual neurons to see what is that person thinking. Currently, machine learning is still lacking universal tools for model analysis (think of something like MRI ;) ).

This is a disadvantage of ML systems, which store the knowledge implicitly, as a specific (and fragile) combination of internal parameters, versus CT systems, which are transparent because the knowledge is explicitly built into the equations themselves.

Performance

Your question starts with the assumption of "increasing availability of large quantities of data and computational power". But is it actually true?

Is the flight data actually so readily available in this case? Do researchers have access to enough sets of mission telemetry data, particularly involving deviations or failures, to build a realistic enough simulation environment to train RL models? Please keep in mind, the algorithm can only become as good as the data it was trained on.
For reference, although computer vision is arguably more difficult than spacecraft guidance, the first powerful image recognition system based on deep learning methods needed about 1.3 million images to train.

Second, can we really talk about increase of computational power in the context of launch vehicles/spacecraft? Modern ML systems process hundreds of thousands of parameters, performing millions of floating point operations to generate a single answer. We are used to using high-performance GPUs to run those systems. While there exist some more energy-efficient hardware solutions, can we be so confident with the assumption that there is enough computing power on the vehicle that the ML guidance is feasible?
By contrast, classical CT guidance gives you a bunch of equations that can be evaluated really fast even on a low-power guidance computer.

Economy

To recap, a machine learning based approach to guidance and control for a spacecraft would comprise at least the following tasks:

  • acquisition of data and construction of a high-fidelity simulation environment for a reinforcement learning system,
  • design of a simple and robust enough RL system,
  • comprehensive evaluation of the system, with failure analysis, stability etc.,
  • hardware application for the purpose of a launch vehicle/satellite.

This is millions of dollars worth of investment, and at least one of those is (at the time of writing the answer) an open research problem in machine learning (system explainability). All that expense has to be justified: how will this system be better than the CT-based guidance? I can't find a reference at the moment, but I have read that trajectories generated by the Apollo-era GN&C were so close to truly optimal, that the amount of wasted fuel during ascent was on the order of single feet per second of delta-v. Cost of the fuel for any rocket launch is minuscule - in the case of SpaceX, it is less than 1% of the entire launch operation. Saving 0.1% of that is just not worth it - a ML system would have to offer something else to rationalize its cost. That "something else" would have to outweigh all the problems listed above, particularly the safety concerns.

So "are classical (...) [algorithms] better than current ML algorithms", depends on what does it mean "better". No serious industry changes something that's tested, proven and cost millions to develop, unless the replacement offers some solid advantages. If we're talking about getting to orbit faster and cheaper, the answer would be: CT is good enough.

$\endgroup$
  • $\begingroup$ "I'm convinced that deep space probes do not have very complex GN&C modules (...) " ==> this is correct, for now at least. Trajectories are optimized on the ground and maneuvers are uploaded to spacecraft. For day to day ops, several companies including Advanced Space, AIKO Space and SCISYS are working toward on-board autonomy of some systems like position estimation, data to downlink to ground, and rover paths on Mars. $\endgroup$ – ChrisR Jul 8 '18 at 17:26
  • 1
    $\begingroup$ One type of ML, at a very rudimentary level, which you didn't mention is Kalman Filtering. It's a sequential estimator, meaning that it is "seeded" with a model and is sequentially provided with measurements. These measurements are then used to compute the estimate of the state of an object (e.g. the position and velocity of the spacecraft/rocket). They do not require large amounts of data like most ML, simply an approximate mathematical model of the dynamics (gravity, affect of wind, etc.) and some instantaneous measurements. $\endgroup$ – ChrisR Jul 8 '18 at 17:32
  • 2
    $\begingroup$ Explainability is a huge hurdle, because we strive for systems that don't 'just work' but ones that advance our understanding of how these things work. Instead of an algorithm we understand and can expand upon we get a black box that works in most cases (and fails spectacularly in some ) and we're no closer to understanding how the problem was solved than before. No new science to build upon! Until we can reverse-engineer working AI reliably, to a human-understandable state, it will be frowned upon in scientific applications. $\endgroup$ – SF. Jul 8 '18 at 18:31
  • $\begingroup$ @ChrisR Your comment about Kalman filter could as well be an answer. To me it seems closer to control theory rather than machine learning, so I didn't talk about it. $\endgroup$ – Przemek D Jul 8 '18 at 19:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.