Start with the Basics of Space Flight, in particular the chapters on telecom, navigation, and the Deep Space Network.
Since you are specifically asking about reliable data transmission, I'm guessing that you thinking about all the retransmission protocols we use in Earth communications. The direct-to-Earth communications to date have generally not used automatic retransmission protocols, due to the long light-time delays and the value of bits on the link. Usually humans are in the loop to decide whether or not to request a retransmission of something that didn't quite make it. Sometimes they do, and sometimes the information has lost value due to the passage of time, and there are better uses for the scarce downlink bits.
The reliability in deep space communications comes from the use of advanced error-correcting codes, high power, big antennas, and very careful link calculations.
We do use automatic retransmission protocols on short relay links in space, as well as automatic data rate adjustments, in particular between Mars rovers and landers and Mars orbiters.