Schaeperkoetter is quoting a footnote from page 437 (4th ed) of David Vallado's Fundamentals of Astrodynamics and Applications. The thesis's notation and derivation are straight out of that textbook. A different way of looking at it and writing it is given in Richard Battin's An Introduction to the Mathematics and Methods of Astrodynamics on pages 138-140, which might be helpful, but as usual he doesn't explain much.
The determinants in Vallado's notation are equal to vector triple products ($(A \times B) \cdot C$) in Battin's notation, which are zero when all three vectors are coplanar. The three vectors in this case are the middle observation, and its first and second derivatives. Because those derivatives are estimated from the other two observations, if all three observation directions are coplanar, then the velocity and acceleration measured from the first to the last also lie in that plane, so the triple product is zero. In any other case, at least one of those three is not in the same plane, so the determinant (the volume of the parallelepiped with those vectors as edges) is nonzero.
Orbital motion is always confined to a plane. This problem only arises when all the observation vectors lie in the same plane. Since the end points (considering the observations as position vectors, not just angles) all necessarily lie in that plane, the zero determinant condition occurs only when the starting points of the three vectors (the observer locations) are also all in that plane.
That said, while Laplace's method is important historically and pedagogically, it is too numerically unstable for practical use. If you want to actually determine orbits from angles-only observations, use Gooding's method:
- Gooding, R.H., "A New Procedure for Orbit Determination Based on Three Lines of Sight", Technical Report 93004, Defence Research Agency, Farnborough, Hampshire, 1993
- Vallado, D.A., "Evaluating Gooding Angles-only Orbit Determination of Space Based Space Surveillance Measurements", Analytical Graphics, Colorado Springs, 2010
- Henderson, T.A., Mortari, D., and Davis, J., "Modifications to the Gooding Algorithm for Angles-Only Initial Orbit Determination", Advances in the Astronautical Sciences 136.238, 2010.
The last paper is by Schaeperkoetter's thesis advisor, Daniele Mortari, and two of his other students. It is particularly important because it addresses the question of how best to use Gooding's IOD when you have more than three measurements.