# How to transform from the spacecraft coordinate system to image coordinates?

Assume e.g. that in the spacecraft coordinate system, I have a target vector

$$\vec{p_\text{t}} = \begin{pmatrix} -11.980\\ -68.051\\ 55.031 \end{pmatrix}\text{km}$$,

which is related to the Earth Mean Equator (EME) J2000 reference frame. I also have the spacecraft offset vector relative to EMEJ2000

$$\vec{p_\text{sc}} = \begin{pmatrix} -262041633.729\\ 115960324.885\\ 89603436.304 \end{pmatrix}\text{km}$$,

and its rotation quaternion (in the form $$[\text{cos}(a/2), n_x \text{sin}(a/2)$$, $$n_y \text{sin}(a/2), n_z \text{sin}(a/2)]$$)

$$q_\text{sc} = \left( 0.86476171, -0.43595577, -0.05902956, 0.24216783 \right)$$,

as well as the offset vector of the spacecraft origin relative to the camera

$$\vec{p_\text{cam}} = \begin{pmatrix} -0.001234\\ -0.000157\\ 0.001390 \end{pmatrix}\text{km}$$,

and its rotation quaternion (also in the form $$[\text{cos}(a/2), n_x \text{sin}(a/2)$$, $$n_y \text{sin}(a/2), n_z \text{sin}(a/2)]$$)

$$q_\text{cam} = \left( 0.99999815, 0.00009373, 0.00027332, 0.00190238 \right)$$.

How can I use this information to calculate the position of the target in image (pixel) coordinates?

The camera has a FOV of $$2.208° \times 2.208°$$, $$2048 \times 2048$$ pixels, a resolution of $$1.882 \cdot 10^{-5}$$rad, a focal length of $$0.7173\text{m}$$, and a pixel size of $$13.5 \cdot 10^{-6}\text{m}$$.

• You need to know the pixel pitch of your camera: how many arc-seconds per pixel? Also, check your units. p_sc is 300 million km, which is the diameter of earth's orbit. at that distance, the entire earth subtends less than 5 arc-seconds. even if it were supposed to be 300 thousand km, that's still most of the way to the moon; what satellite is this? the camera lever arm seems reasonable, but what does the target vector mean? the normalization is weird (88.334 km), and it's either nearly perpendicular to p_sc (if we draw it from the tip of p_sc), or nearly indistinguishable from zero. Jan 6, 2022 at 20:16
• Hey, thanks a lot for you feedback! This is data from the Rosetta spacecraft. The target vector points from Rosetta to the center of mass of comet 67P. I would like to know where it is located in an image. Of course you're right about the cameda data. I update the post.
– mapf
Jan 6, 2022 at 21:13

I wrote a paper on a similar problem, long time ago. I'll try to provide the gist of it.

You need a model for the camera, so that you can map a point in 3D space to a point in the image. A simple model that is useful for this is the pinhole camera model. This model assumes an image plane at the focal depth of the camera.

In the following I use $$p_\text{name}^{\Psi}$$ to denote vector $$p_\text{name}$$ expressed in coordinate frame $$\Psi$$ and $$H_{\Psi_1}^{\Psi_2}$$ to denote a homogeneous coordinate transformation from $$\Psi_1$$ to $$\Psi_2$$.

Start with the 3D coordinates of the target and apply a coordinate transformation to a coordinate frame that has its origin in the camera's aperture:

$$\begin{bmatrix} p_\text{target}^\text{camera} \\ 1 \end{bmatrix} = H_\text{EME}^\text{camera} \cdot \begin{bmatrix} p_\text{target}^\text{EME} \\ 1 \end{bmatrix} .$$

The transformation matrix $$H_\text{EME}^\text{camera}$$ can be computes as

$$H_\text{EME}^\text{camera} = H_\text{sc}^\text{camera} \cdot H_\text{EME}^\text{sc},$$ with $$H_\text{EME}^\text{sc} = \begin{bmatrix} R_\text{EME}^\text{sc} & p_\text{sc}^\text{EME} \\ 0 & 1 \end{bmatrix}$$ and $$H_\text{sc}^\text{camera} = \begin{bmatrix} R_\text{sc}^\text{camera} & p_\text{camera}^\text{sc} \\ 0 & 1 \end{bmatrix}.$$

$$R_\text{EME}^{sc}$$ and $$R_{sc}^{camera}$$ are rotation matrices that follow from your spacecraft and camera quaternions. See the Wikipedia page on quaternions to see how to convert from quaternions to rotation matrix.

With the target point converted to the appropriate camera coordinate frame, you can project $$p_\text{target}^\text{camera} = (x, y, z)^\text{target}_\text{camera}$$ onto the image plane as follows:

\begin{aligned} a &= \frac{f}{z} x, \\ b &= \frac{f}{z} y, \end{aligned}

where $$x,y,z$$ are the coordinates of the target in the camera coordinate frame, $$f$$ is the focal length of the camera, and $$a,b$$ are the 2D coordinates of the target in the image plane.

From this, use the pixel size (pitch) to convert to image pixel coordinates.

The focal depth of the camera is typically known. This model does not take into account distortion by the lens, but I assume that the images you are provided with have been corrected already for visual distortions. If you have raw imagery that has not been corrected yet, you need to apply some correction model, which goes beyond this answer and my expertise.

• Hi thanks a lot for the elaborate answer! One question though: I assume in the last two equations $f$ is the focal depth, but what is $z$? And are $x_\text{camera}$ and $y_\text{camera}$ the $x$ and $y$ coordinates of $p^\text{target}_\text{camera}$?
– mapf
Jan 10, 2022 at 19:29
• Also, if you don't mind, it would be great if you share a link to your paper anyways.
– mapf
Jan 10, 2022 at 19:31
• @mapf I made some edits to clarify. Is this better? Also, the paper is available from my university for free nowadays, so I added the link in. It's in robotics, so much of it is not really related; section III-A is the relevant paragraph.
– Ludo
Jan 10, 2022 at 20:36
• @mapf sorry, my bad, that's some notational sloppiness from my side. I fixed it: you need to make $p$ a 4x1 by adding a 1 below it.
– Ludo
Jan 11, 2022 at 12:55
• @mapf Getting the rotations and vectors correct is always tricky. It's easy to mess up a minus-sign somewhere. I recommend making a 3D visualization of each step of the coordinate transformation to see if it matches your expectations. Start with all coordinate frames aligned (no rotation) and points on the principle axes of each coordinate frame (i.e. $[1,0,0], [0,1,0], [0,0,1]$) and get the translations right. Then start rotating each frame along one axis at the time to get the rotations right. Also recall that the inverse of a rotation is $R^T$, but that the inverse of $H$ is not $H^T$.
– Ludo
Jan 11, 2022 at 14:49