Bore sight is the direction along which the star tracker camera-lens system is pointing.
From this pdf
A single star tracker gives two good attitude estimates (x- and y-axes) and one poor attitude estimate (the twist around the bore sight) ... Typically for narrow-angle star trackers, the attitude estimates on the twist around the bore sight are about 20 times worse than those of the other axes.
I have drawn a diagram based on what I have understood.
- Red axis is the bore sight.
- Sensor is represented using a grey rectangle (right side of the image).
- Field of view is marked using a grey circle (left side of the image).
- The apparent path of a star (when viewed from the sensor, and the space craft is rotating about one of the three axes) are marked using three arcs (superimposed on the grey circle).
- The rotation about green and blue axes are related to cross bore sight accuracy (they form a cross).
- The rotation about the bore sight is marked in red.
It looks as if the cross accuracy can be increased by increasing the focal distance of the lens system. The about-bore-sight accuracy can be increased by increasing the sensor size. About-bore-sight accuracy also seems to depend on the position of the star image on the sensor.
If the star image falls exactly at the centre of the sensor, even a 360 degree rotation about the red axis will not move the image to another pixel on the sensor. The second method being costlier, it is not surprising that accuracy about that axis is traded off.
The below two images attempt to illustrate the angular accuracy for the cross-axis and about-bore-sight directions.
The top image shows a side view of the star tracker. The circles on the right side represent pixels on the sensor. The accuracy would depend on the angle that sensor would need to rotate for a star's light to move from one pixel to its neighbor. This can apparently be increased by increasing the focal distance (marked l).
The bottom image shows the "front" view of a star tracker. The pixels on the sensor are shown as circles. Imagining that the star light falls on the outer most pixel of the sensor, we can see the angular accuracy about-bore-sight. It seems to depend on the size of the sensor (marked r). If the star light was focused exactly on the centre pixel (coloured green), then rotation about bore sight would not shift the image to any neighboring pixel making the accuracy wrose.