Specifically a location in the celestial sphere that is fixed, most likely a star, which creates a set of still coordinates with respect to the Sun. There are many ways to do this, but I am interested in frames used for interplanetary missions, specifically for New Horizons and Cassini, and how the frames are used for telemetry and navigation.
That would be the International Celestial Reference Frame. The International Earth Rotation and Reference Systems Service (IERS) is the group responsible for maintaining the reference frames used by astronomers and others, as well as maintaining time. (They're the ones who decide when the next leap second will occur.)
While the predecessors to the ICRF were based on stars (e.g., the frame widely known as J2000 was based on the Fifth Fundamental Catalogue (FK5)), the ICRF is not based on stars. It is instead based on quasars. There are two advantages to using quasars instead of stars:
- Quasars are very, very, very far away, drastically reducing the problem of proper motion.
- Astronomers detect those quasars with radio telescopes. This means they can use Very Long Baseline Interferometry to get an extremely precise reading of where those quasars are in the sky, much more precise than visual frequency telescopes can obtain.
The IERS released revision 2.0 of the ICRF in 2009. I don't know if JPL has upgraded to this newer standard in the more recent releases of its Development Ephemeris.
Update: How the frames are used for telemetry and navigation.
I'll get to telemetry and navigation, but before that, they are first used for mission planning. You need to know the orbits of the Earth, the target planet, and any intermediate planets used for gravity assist to find the launch opportunities for a mission. You need a set of planetary ephemerides to do this. Supporting those interplanetary missions is the primary reason JPL originally developed its Development Ephemeris back in the 1960s, and has continued to refine them ever since.
The ephemeris plays an important role not only in mission planning but also in mission operations. The vehicle is never exactly on the trajectory on which it is supposed to be. The results from orbit determination tell the mission controllers what trajectory the vehicle is on (to within some error), and that estimated trajectory coupled with the ephemeris in turn informs the controllers about midcourse corrections the vehicle needs so that it will reach the target.
The reference frame comes into play in multiple places. The ephemeris is necessarily expressed in some specific frame of reference. Which one? The scientists involved in these operations use a plethora of reference frames, more so in the past than now. There are different fundamental planes (e.g., Earth equator versus the ecliptic). Since the Earth precesses and nutates, which equatorial plane? Since the orbits are not quite Keplerian, which ecliptic? This non-constancy of the equatorial and ecliptic planes leads to even more choices with regard to frames of reference. The developers of an ephemeris need to be very aware of this plethora of frames so as to correctly interpret all the observations that lead to the ephemeris. The users of the ephemeris need to know the frame in which that ephemeris is expressed.
The developers of an ephemeris explicitly assume that the frame they are using is an inertial frame. What if it isn't? What if that frame is rotating with respect to some true inertial frame? That the frame is rotating results in a mismatch in the equations of motion the planets actually follow in that frame versus the equations of motion used in the development of the ephemeris. Having a very good estimate of what constitutes an inertial (non-rotating) frame of reference helps a lot in improving the accuracy of those ephemerides. The same applies to a lesser extent to the problem of vehicle orbit determination.
This frame rotation problem motivated JPL to be one of the first organizations to switch from the FK4-based Mean of 1950 frame to the FK5-based J2000 frame, and from that frame to the ICRF. That old FK4 frame wasn't bad; that's what was used to plan and operate the Pioneer and Voyager missions. It wasn't great, either. A better frame means better ephemerides and better orbit determination. Less fuel needs to be allocated for midcourse corrections with a better estimate of what constitutes an inertial frame, and reduced fuel mass means more mass can be added as scientific payload.
Orbit determination is one key part of navigation. Another part is vehicle pointing. The vehicle's flight software may not need to know where the vehicle is in space, but it certainly does need to know its attitude. The star trackers and other attitude determination equipment are expressed in terms of some frame of reference, and the flight software needs to be aware of this. It also of course needs a frame of reference in which to express that computed attitude.
Finally, there's telemetry. The Earth-based communications equipment need to point toward the vehicle and the vehicle needs to point it's communications equipment toward the Earth in order for communications to occur. The vehicle side of this problem can be made simple by telling the vehicle where to point as a function of time. The vehicle doesn't need to know where it is or where the Earth is. It just needs to where to point its antennae. Earth-based side of the problem is much trickier. This requires knowledge of where the vehicle is in inertial space (orbit determination) and also requires knowledge of the Earth's rotation. This leads to the incredibly complex Earth rotation/nutation/precession model. The current best estimate involves the IAU 2000/2006 precession nutation model, which has thousands of terms, plus Earth Orientation Parameters supplied by the IERS.