TLDR: For certain orbits, a 3-axis magnetometer can be matched with a model of the Earth's field to provide reasonably good attitude information. In other cases, including high or equatorial orbits or a need for higher precision, a little more input information is needed.
In general, you need two measured orientation vectors to determine attitude in 3D space. Your cell phone, for example, can sense local-down to reasonable accuracy with an accelerometer, but needs a separate way to sense rotation about that. A magnetometer in compass mode, along with a very simple model of Earth's field and a bit of information about local position, can provide that.
(As an aside: The "bit of information about local position" is needed to determine the local variation, which is where a compass points relative to geographic North. A 3D magnetometer, as opposed to a compass mode one, can provide part of that information via measuring the dip angle, but that's rarely enough by itself. And magnetometer information on Earth's surface is always a bit suspect due to local influence)
Except for geostationary satellites, satellites move through Earth's magnetic field. They can take measurements at different times hence different places, sampling different field orientations. In the limit of perfect 3D field measurement, a perfect model of the possible modes of satellite rotation (which might change over time), a perfect model of Earth's field at each point in the orbit, and perfect knowledge of the orbit, it's clear there's enough information in 3D field measures at different times to calculate the satellites orientation.
As the information degrades, due to i.e. practical sensor limitations and imperfect field/orbit/satellite information, this math gets harder and harder to do.
Note that some orbits make that easier: Inclined ones (the Earth's field varies more north and south, much less along the equator), eccentric ones (which sense more angular variation in the field) and low ones (stronger field and faster orbital variation).
Naively doing this involves fitting thousands of 3D field measurements taken over time in a local coordinate system fixed to the (rotating) spacecraft, to pre-calculated field-at-point-in-space values for the magnetic field in the Earth's coordinate system, along with a time-varying transformation between those systems that represents the attitude of the satellite as a function of time. That time-varying transformation is in turn made up of a model of the satellite's moments of inertia, along with a (perhaps time varying) model of external torques, and some initial rotations. The result of the fit is an attitude as a function of time value over the rest past up to Right Now.
The naive process is optimal, but computationally impossible, so this computation is normally done via a sequential approximation process that leads to better and better values for "now" without worrying about improving past values. This is typically done via Kalman filtering processes; one paper that describes that data reduction process in detail is "Three-Axis Attitude Determination via Kalman Filtering of Magnetometer Data" F. Martel, P.K. Pal, M.S. Piasaki
The objective of this work has been to develop a low-cost system for
estimation of 3-axis spacecraft attitude information based solely on
3-axis magnetometer measurements from one satellite orbit. Such a
system will be useful for missions that operate in an inclined,
low-Earth orbit and require only coarse attitude information.It can
also serve as the sensor part of a low-cost 3-axis closed-loop
attitude control system, or as a back-up attitude estimator.
A single 3-axis magnetometer measurement can give only 2-axes worth of
attitude information and no attitude rate or disturbance torque
information. Therefore, this attitude determination system must use a
sequence of magnetometer measurements.
The Kalman filter discussed
in this work is applicable to nadir pointing Earth satellites
operating at low altitudes in inclined orbits. The inclination and low
altitude of the orbit are necessary to the proper functioning of the
filter. The orbit must stay close enough to the Earth, within about 4
Earth radii, so that a spherical harmonic approximation of the Earth's
magnetic field gives a reliable attitude reference. Some inclination
of the orbit is necessary to make the attitude of all three axes
sufficiently observable. Pitch information in a l-orbit magnetometer
time history gets poor for low inclinations, although theoretically.
there is still some pitch information even in equatorial orbits; the
Earth's magnetic poles do not coincide with its rotational poles.
The Kalman filter, as an algorithm, is structured like as an ongoing update cycle:
For details of the math involved, please see the paper with has pages and pages of things like this:
way too many and too complex to even summarize in MathJax.
Bottom line, this can converge pretty rapidly, in the cases where it works well even over one orbit:
The absolute accuracy depends on the accuracy of inputs. The paper takes as typical 2% errors on moments of inertia, magnetometer imperfections, etc and shows a degree-scale error budget for the final position:
For some applications, that's great! But for others, it's not good enough, and you need to add additional information (or create a more perfect satellite, which might not be a choice). There's particularly a need to improve the magnetometer-only results for small, imperfect cubesats without any internal stabilization. An interesting paper on minimal additions to get maximal performance is “CubeSat Attitude Determination via Kalman Filtering of Magnetometer and Solar Cell Data” E.P. Babcock and T. Bretl:
This report documents the design and implementation of an extended
Kalman filter (EKF) for attitude estimation using three-axis
magnetometer and two-axis solar cell measurements. The motivation for
such a system is to utilize sensors already present on most CubeSats,
namely three-axis magnetometers for active magnetic detumbling and
four faces of solar cell arrays for power generation. The system is
developed and simulation-tested on a 1-U CubeSat in a 600 km dawn-dusk
orbit.
It walks through the mathematics to combine the time-varying magnetometer readings with coarse angular information from how much current comes from each of several solar power cells that are on different faces, pointing in different directions.
