Efficiency in communications affects all aspects of the communication system design.
It's useful to look at this in terms of a model like the OSI model. Now that's aimed at computer networks, but the general principle is similar:
- at the lowest level, you have the hardware: transmitter, receiver, antennas. Anything you do here influences the total system throughput: a bigger antenna or more powerful transmitter = more data can be sent.
- at higher levels, you have communications protocols like the turbo coding used by New Horizons. This makes sure you can send your message in as few bits as possible.
- the top level is the application: the actual science data and diagnostics/housekeeping information you're interested in.
At the application level, a lot of research as been done into data compression. Typically, you want to send all data from the instruments to Earth (no compression), or you want to send enough that the original data can be reconstructed (lossless compression).
In recent missions, we've seen lossy compression being used: the spacecraft decides which parts of its data are irrelevant and can be thrown away. New Horizons has done this in two ways:
- by cropping images. For instance, this image of a region of Charon is a rectangular cropped region. The remainder of the image (which would have shown empty space so it wasn't very interesting) was not sent at all.
- by compressing the image data using a lossy algorithm (JPEG, in this case). JPEG throws away data based on what the human eye can see: tiny differences are rounded off because they'd all look the same anyway. There is a risk in this: by throwing away data you reduce the scientific usefulness of the image. So New Horizons sends us compressed images now so we have at least something to look at, and will send the full, uncompressed data set over the next year and a half.
Data compression will become more common. Some of the new space telescopes generate way more data than they can ever send to Earth, so the spacecraft has to contain powerful algorithms that help decide which images are interesting enough to send to Earth.
Note: this answer focuses mostly on the science data as that will be bulk of the data volume. I haven't found good information on how housekeeping data is handled.