Question: Given a set of XYZ (3D) coordinates in the Earth-fixed frame, after converting them to the inertial frame, how do I perform orbit fitting in the least-squares sense (meaning minimising all least squares residuals below)? My objective is to come up with an orbit ellipse (6 elements) that best-fits those points such that the L2-norm of the residuals are minimised, but I'm not sure where I should start... I know there are programs and software out there to do it, but I'd like to try to make my own :)
What you are looking for is called orbit determination, and in particular batch least squares orbit determination.
To learn about it I can recommend Statistical Orbit Determination by Bob Schutz, Byron Tapley and George H. Born.
I understand that you want to try to do it yourself. Nevertheless, if at some point you are looking to do it with software, it is not very complicated either. Here is an example with Orekit: https://nbviewer.jupyter.org/github/GorgiAstro/orbit-determination-examples/blob/a5b592656cadb411f7edafdf15e16d8b3ca3eeaa/01-keplerian-od-with-iod.ipynb [disclaimer: I am the author of this example]. It demonstrates:
- Initial Orbit Determination (IOD) using Gibbs method using three position vectors, which is necessary if you don't have any clue about the orbit, because the Batch Least Squares estimation requires a first guess
- Batch Least Squares (BLS) estimation