Another interesting paper is The Electra Proximity Link Payload for Mars Relay Telecommunications and Navigation (C.D. Edwards, et al., 2003), which is reference #21 in the DESCANSO paper @BrendanLuke15 linked. It describes a software-defined radio based on a 32-bit SPARC and a million-gate FPGA, saying
The modulation chain within the FPGA includes a V.38 scrambler,
differential encoder, Reed-Solomon encoder and interleaver,
convolutional encoder and then selection of Manchester or NRZ-L coding
into the final digital modulator block.
The SPARC runs at 24 MHz, but the selectable output data rate runs from 1 ksps to 2 Msps in powers of 2. The basic modulation is BPSK (QPSK and 8PSK are mentioned in passing as possible future hardware modifications), but that's then k=7 rate=1/2 convolutional encoded (3-bit Viterbi soft decode) and interleaved with RS (255,239) or (204,188).
This is probably incomprehensible to most people, but it at least gives you a set of search terms to start chasing. A reasonable place to start is probably with Reed-Solomon, which is used in all commercial CDs and DVDs. If you want even more detail, then try Real-Time Navigation for Mars Missions Using the Mars Network (E.G. Lightsey et al., 2008) next.
On the other hand, it looks to me like all of the fancy stuff is fixed. When the orbiter commands a data rate change, it looks like all the rover does is double or halve the symbol rate, not change the encoding or interleaving. This means the bandwidth varies in the same simple way, as they try to crank up $B$ to as close to $C$ as their varying SNR permits.