6
$\begingroup$

The first Starliner landing was in the very early morning. The live coverage was only able to show infrared pictures of the descent, however NASA's documentation of the event includes a colour video of the landing:

Something weird about this video is that there seems to be a lot of stuff floating around in the air around the capsule and parachutes, and in particular moved around by their motion.

gamma-boosted snippet from the NASA video

It looks a bit like dust particles, or like video static – however it can't really be static noise, because that would not be affected by the spacecraft, and uncorrelated with the actual image information.

What's going on there?

I have a guess:

It is actually static, created by the image sensor because it was still to dark to shoot colour video. In fact the raw video contained a lot more severe noise, but they tried to reduce it in post-processing. Specifically, they used a (probably CNN-based) machine-learned denoiser. Such denoisers work essentially by pattern recognition: they have been trained on all kinds of video, including noisy and noiseless ones, and try to pick out parts that could be real instead of noise. In this case, the neural net “hallucinated” seeing floating dust particles in the static (maybe learned from underwater footage of jellyfish).
Again: this is just a guess, I don't know how they really processed the video.

$\endgroup$
  • $\begingroup$ Can you somehow show exactly what you are referring to? Annotate a still and give the exact time? I don't see anything odd in the video. $\endgroup$ – Organic Marble Jan 16 at 21:15
  • $\begingroup$ It's most visible at 16:41, right when the capsule hits the ground and the chutes start moving sideways. $\endgroup$ – leftaroundabout Jan 16 at 21:37
  • 1
    $\begingroup$ Is it possible to use one (or more) of these screen shots from around the 16:41 timeframe and either add an arrow or describe in detail which area you are asking about? Thanks! 1, 2, 3, 4, 5 $\endgroup$ – uhoh Jan 16 at 21:51
  • 1
    $\begingroup$ @uhoh no, you don't notice it in a still frame, only the movement gives it away. I added a short animation snippet where it's clearly visible. $\endgroup$ – leftaroundabout Jan 16 at 23:16
  • $\begingroup$ @leftaroundabout oh I see what you mean! I want to use the word optical flow in an intelligent-sounding sentence in this context but can't, so I'll just add some random links visionhelp.wordpress.com/optic-flow-undulating and youtube.com/watch?v=_Rpi7WS-HTw and youtu.be/5VyLAH8BhF8?t=317 I think you can ask a related question about denoising and flow in Signal Processing SE In this case there may be compression noise involved along with the shot noise from the sensor $\endgroup$ – uhoh Jan 17 at 0:00
5
+25
$\begingroup$

I sincerely doubt this is "dust" of any kind. It looks and behaves like the classic sensor noise that is visible any time an image sensor is used at high gain. Noise in this case is typically described as random fluctuations in the voltage levels recorded by the image sensing system caused by factors such as local heat and variations between sensor pixels. In "normal" use, these things don't matter: if we record that something is 99% black for 99.1% black, you probably won't care... usually. But if there's very little light and so almost all the pixels are within 1% of 99% black, we can multiply the values supplied by the sensing system.

The odd bit out is that the noise appears to sort of flow with the parachute lines, but that is easily explained by the fact that many video compression algorithms work by recording the changes between frames instead of the the full content of a given frame. This is often done in comparison to "key frames", which is when there's a sudden, large-scale change in the image content; recording the delta between the previous frame and the current frame will end up taking more space than simply a compressed full-size frame.

After doing some research for this answer, I discovered there's a name for this type of compression artifact: motion compensation artifacts. In a general sense, the Wikipedia page for motion compensation and how it relates to compression is extensive, but the very small text on motion compensation block boundary artifacts in the article on compression artifacts gives a perfect description of what we see in this video:

Block boundary discontinuities can occur at edges of motion compensation prediction blocks. In motion compensated video compression, the current picture is predicted by shifting blocks (macroblocks, partitions, or prediction units) of pixels from previously decoded frames. If two neighboring blocks use different motion vectors, there will be a discontinuity at the edge between the blocks.

If you watch the gif at high magnification and watch the boundary area between the parachute lines and the clear area to the left of the parachute, you'll see that the "drift" occurs in visibly discrete blocks that look kind of like the picture below. Note that this is almost invisible in a static picture, even though it's pretty clear in motion. illustration of block layout

The end result is that the noise (as recorded by the compression algorithm) gets tied to things around it that have more detail/contrast (in this case, the parachute) such that movement of the parachute and its lines is effectively transferred to the low-contrast areas around it.

leftroundabout raises a good point in their comment that you'd not expect to many compression artifacts on an earthbound stream, but the difference between raw video and high-quality video compressed with a modern algorithm can easily be two to three orders of magnitude. Video compression can be significantly more efficient in many cases because of things like motion compensation, so storing completely uncompressed video is extremely rare except if someone is actively editing or working with said video.

Here are two examples:

  • An uncompressed screenshot of my desktop right now takes ~24 megabytes, while the same thing as as JPEG consumes merely 1MB of storage.

  • Wikipedia says that uncompressed HD video at 24 FPS takes 1.4Gbit/s - that's 175 megabytes per second of video, which works out to 420GB for a single 5-minute video. By comparison, HD video on DVD can take no more than 29.4Mbit/s or 3.675 megabytes per second. Even Blu-Ray discs only bother to encode at 40Mbit/s.

Pure speculation on my part: I suspect that the video in question was recorded, compressed lightly, and that the already compressed video was bumped up a couple stops to make it easier to see the darker regions, then compressed again. That first compression step would not have prioritized the noise in the dark regions because it would be nearly invisible, but it became much more visible after the re-exposure / enhancement.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ That's what I think. Any references/examples? $\endgroup$ – JCRM Jan 29 at 22:27
  • $\begingroup$ Yeah, this is just the same conjecture that's already in the comments or even the original question. And while the compression hypothesis is certainly plausible, I just don't see why they would have so heavy compression on this video that the motion-leech would be easily visible. This was an earthbound camera in a big-budget mission, they wouldn't care if the video of the short landing sequence takes many gigabytes to save. $\endgroup$ – leftaroundabout Jan 30 at 0:22
  • $\begingroup$ I was tempted to stick "You could argue this is a laymen's version of Carl's answer, but editing his to add details about how video encoding works didn't seem right". I could be not up to date on these things, but I'm unaware of widely used codecs that use a NN. $\endgroup$ – William Jan 30 at 5:53
  • $\begingroup$ @leftaroundabout - I added a portion to address your concerns. $\endgroup$ – William Jan 30 at 16:24
  • $\begingroup$ Ok. Yes the slightly-compressed-but-up-a-couple-stops explanation makes a lot of sense. And I even seem to see those blocks you're talking about too... though I still wonder if that's just another case of pareidolia. Also I'm not sure I agree that the motion happens only in those blocks – to me, it seems to be moving in the lower right corner of frame (in the gif). At any rate, we can conclude that it's really hard to tell the difference between random noise and actually significant structures... — I don't think this can be conclusively settled, without the original unprocessed file. $\endgroup$ – leftaroundabout Jan 30 at 23:27
0
$\begingroup$

I agree with uhoh's comment -- this is photon shot noise and/or detector readout shot noise raised to a visible level by way of high gain applied to dark image areas. You can see similar effects most dramatically in footage (now there's an obsolete term!) from night-vision cameras, where the front-end gain makes detected photon events quite visible to the viewer.

| improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ So how do you explain why that noise is “pushed around” by the parachutes? $\endgroup$ – leftaroundabout Jan 17 at 13:54
  • $\begingroup$ @leftaroundabout I suspect that's an artifact of readout sequence, if not simply an optical illusion caused by the motion of a bright object in a dark field. $\endgroup$ – Carl Witthoft Jan 17 at 14:01
  • $\begingroup$ @leftaroundabout It moves in the same direction as the parachutes. Stating that it is 'pushed around' is confusing causation with correlation (if any). $\endgroup$ – user10509 Jan 17 at 15:26
  • 4
    $\begingroup$ I suspect the "pushed around" appearance of the noise is more an artifact of the video compression algorithm than anything else. $\endgroup$ – Tristan Jan 17 at 15:53
  • 1
    $\begingroup$ noise is random - different from frame to frame. $\endgroup$ – JCRM Jan 18 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.