I've paraphrased your question a bit somewhat, into: "What frequency bands and choices of antennas are used for satellites to send data through deep space? For example, if we place a cubesat in an orbit about Io (the moon, Jupiter I), which antenna is best used? What method of transmission is used?".
To answer the bulk of your question on frequency: deep space communications tend to be relatively lower frequency, because on top of free-space losses, it also contends with atmospheric attenuation (H2O and O2). See the chart below.
(Credits to RFCafe for sharing their attenuation chart)
Thus, a good compromise between high data rates and minimising additional attenuation due to atmospheric losses, is somewhere X-band or below ( < 10GHz). An example: the MarCO Cube Satellites relay to the InSight Lander during EDL at UHF (300MHz - 3GHz), but the data is relayed back via an X-band high-gain antenna.
Adding this commented note by user uhoh: You want to go to higher frequency because (using those link budget conventions) for the same size antenna the gain goes up because the size relative to the wavelength increases and the radiation pattern becomes narrower and therefore more intense at the peak.
(Credits to NASA JPL for sharing their artwork on MarCO A and B)
This leads to the second part of your question - what kind of antenna? This is a tricky one. In deep space, we hope to use parts with the least number of moving objects and be easily stowed away for launch. In most cases a high-gain antenna could be a phased array, with the gain depending on the number of elements (a 30 dB gain array needs about 1000 elements and a 20 dB gain array needs about 100). It is simple in physical packaging, but of course expensive in the number of RX/TX modules per element. In the case of MarCO, they used flat-panel reflectarray antenna (a feed horn that emits into reflecting elements for high gain).
You can find out more about their mission here!
See answers to these for further reading: