how and why is image processing used on spacecrafts?

Question

Electronics engineer here trying to dive into aviation and aerospace engineering. I was wondering which (advanced) image processing techniques are being used in satellites nowadays. And which purposes do they have?

For example in some industrial processes computer vision is used to monitor the growth of vegetables. A more complex case related to automobile security is where you develop a system for dead angles. So this involves specific hardware with software which uses person detection to detect whether a biker (not an object), jogger, pedestrian or whatsoever is in a range which is considered as too close to the vehicle. Another security related case can use person detection and facial recognition. A last example: using computervision to detect whether an older person fell and is unconscious. (in other words not sleeping nor falling and standing up again). If the systems detects that the person is unconscious it will automatically call a doctor or whatsoever. Those were some easy and very difficult examples. They combine many image processing algorithms (e.g. edge detection, smoothing et cetera). Now I would like to know in which cases image processing can be used in an aerospace (satellites or aviation) context. I will probably look at it and try to implement it myself as a side project for autodidactical purposes :)

Answer 1

The Earth Obervation satellites do use some image processing, especially image compression to reduce the size of the downlinked data. A typical compression pipeline looks like this:

detector -> offset correction -> gain correction -> bad pixel correction -> compression

Because of the volume of data, this is typically done on dedicated hardware, shuch as this:http://www.space-airbusds.com/media/document/ens_4_coreci_2014_bd.pdf

Computer vision can also be used for docking guidance. Generally, high-level algorithms that do not need to run in real-time are executed on the ground, where the hardware constraints are not a problem. Onboard spacecrafts, mass and power are limited, and if you use radiation-hardened hardware it is much less capable than off-the-shelf processors.

Answer 2

The Mars2020 Rover mission will use image processing for navigation on the surface in a similar way to the current Curiosity Rover, as its design is rooted on Curiosity's highly successful architecture.

above: illustration of Mars2020 Rover design from here.

However, image processing will also be used in several new ways during the descent and landing phase.

From Entry, Descent, and Landing Technologies:

RANGE TRIGGER - IT'S ALL ABOUT TIMING

The key to the new precision landing technique is choosing the right moment to pull the "trigger" that releases the spacecraft's parachute. "Range Trigger" is the name of the technique that Mars 2020 uses to time the parachute's deployment. Earlier missions deployed their parachutes as early as possible after the spacecraft reached a desired velocity. Instead of deploying as early as possible, Mars 2020's Range Trigger deploys the parachute based on the spacecraft's position relative to the desired landing target.

above: Illustration of the Range Trigger concept - using the historical /Mars Science Laboratory(MSL)/Curiosity landing site as an example.

Also, during the descent stage, images will be compared in real time to stored terrain maps to improve the accuracy of the landing location:

HOW TERRAIN-RELATIVE NAVIGATION IMPROVES ENTRY, DESCENT, & LANDING

Terrain-Relative Navigation significantly improves estimates of the rover's position relative to the ground. Improvements in accuracy have a lot to do with when the estimates are made.

In prior missions, the spacecraft carrying the rover estimated its location relative to the ground before entering the Martian atmosphere, as well as during entry, based on an initial guess from radiometric data provided through the Deep Space Network. That technique had an estimation error prior to EDL of about 0.6 - 1.2 miles (about 1-2 kilometers), which grows to about (2 - 3 kilometers) during entry.

Using Terrain-Relative Navigation, the Mars 2020 rover will estimates its location while descending through the Martian atmosphere on its parachute. That allows the rover to determine its position relative to the ground with an accuracy of about 200 feet (60 meters) or less.

It takes two things to reduce the risks of entry, descent, and landing: accurately knowing where the rover is headed and an ability to divert to a safer place when headed toward tricky terrain.

above: Illustration of Terrain-Relative Navigation. "Terrain-Relative Navigation helps us land safely on Mars - especially when the land below is full of hazards like steep slopes and large rocks! From here.

In December 2014 the vision system was tested in the Mojave Desert. :

above: "A prototype of the Lander Vision System for NASA's Mars 2020 mission was tested in this Dec. 9, 2014, flight of a Masten Space Systems "Xombie" vehicle at Mojave Air and Space Port in California. Credit: NASA Photo/Tom Tschida" From here.

NASA tested new "eyes" for its next Mars rover mission on a rocket built by Masten Space Systems in Mojave, California, thanks in part to NASA's Flight Opportunities Program, or FOP.

The agency's Jet Propulsion Laboratory in Pasadena, California, is leading development of the Mars 2020 rover and its Lander Vision System, or LVS. In 2014, the prototype vision system launched 1,066 feet (325 meters) into the air aboard Masten's rocket-powered "Xombie" test platform and helped guide the rocket to a precise landing at a predesignated target. LVS flew as part of a larger system of experimental landing technologies called the Autonomous Descent and Ascent Powered-flight Testbed, or ADAPT.

LVS, a camera-based navigation system, photographs the terrain beneath a descending spacecraft and matches it with onboard maps allowing the craft to detect its location relative to landing hazards, such as boulders and outcroppings.

above: cropped image of a prototype of the Astrobotics vision system under testing, from: http://www.nasa.gov/centers/Armstrong/Features/XombieTestsAstroboticAutolandingSystem.html