tl;dr: Considering Voyager-like conditions, going from 3.66 and 70 meter dishes to 0.5 and 5 meter telescopes, and from 3.6 cm to 1.55 micron wavelengths, we get an increase in received power of 10,000 times and an increase in data rate of 1,000 times!
Reference Systems
Reference systems for a spacecraft downlink to Earth will be loosely based on Voyager for X-band and NASA's DSOC (Deep Space Optical Communications) for optical.
Type Power(W) f(GHz) λ(cm) TX diam(m) RX diam(m)
------ -------- --------- -------- ---------- ---------
X-band 22 8.4 3.6 3.66 70
Optical 4 193,500. 0.000155 0.5 5
Using the longer optical wavelength of 1550 nm instead of 850nm lets you have a nice optical fiber communications single-mode laser diode efficiently coupled to a single mode fiber, then use EDFAs (erbium-doped fiber amplifiers) to optically amplify the signal to several Watts while keeping it within a single mode fiber. This is necessary to take advantage of the diffraction limited optics of the telescope to produce a narrow transmit beam.
I used 0.5 meters for the spacecraft's optical "dish" because that's the diameter of an actual telescope mirror that is on each of the voyagers now.
Link Budget
From this answer:
$$ P_{RX} = P_{TX} + G_{TX} - L_{FS} + G_{RX} $$
- $P_{RX}$: received power on Earth
- $P_{TX}$: transmitted power by Voyager
- $G_{TX}$: Gain of Voyagers transmitting antenna (compared to isotropic)
- $L_{FS}$: Free space Loss, what we usually call $1/r^2$
- $G_{RX}$: Gain of Earth's receiving antenna (compared to isotropic)
$$G \sim 20 \times \log_{10}\left( \frac{\pi d}{\lambda} \right)$$
$$L_{FS} = 20 \times \log_{10}\left( 4 \pi \frac{R}{\lambda} \right).$$
Currently Voyager 1 is about 2.1E+13 meters (yes, 21 billion kilometers!) away.
Type P_TX (dBW) G_TX(dBi) L_FS(dB) G_RX(dBi) P_RX(dBW) photon/sec
------ ---------- --------- -------- --------- --------- ----------
X-band 13.4 50.0 317.3 75.7 -178.2 272,000
Optical 6.0 120.1 404.6 140.1 -138.4 113,000
That's an increase in received power of 10,000 times!
So right off the bat we see that by shrinking the wavelength by 20,000 times more than offsets smaller diameters of the "dishes".
A really surprising thing to me is that the number of photons ($E = h \nu$) is almost the same! At a handful of GHz we usually don't talk about photon rate because they are very difficult to count and even at liquid helium temperature the background photon rate is quite high.
But at optical frequencies we can certainly count individual photons! So instead of comparing the received power 1.5E-18 W to $k_B T$ (about 1.4E-22W at 10K) we can just go directly to photon counting statistics. Even at room temperature, the rate of thermally produced optical photons is very low. We are no longer in the Rayleigh-Jeans regime, discussed further here.
I will leave further discussion of photon counting to a future question and answer session. Instead of photomultiplier tubes which work well for visible and just barely infrared (say 800 nm) what is in vogue now is superconducting nanowire position-sensitive photon detectors for the downlink receivers at least. See the images below for example (demonstrated by LADEE's Lunar Laser Communication Demonstration.
According to Spaceflight 101's Lunar Laser Communication Demonstration and ESA's LADEE efficiencies are in the range of 1 bit per detected photon. It relies on precision timing of the photons and a bit more math than I'd like to learn today to show this.
So instead, I'll just quote @MarkAddler:
No, you don't need "at least some photons per data bit". 13 bits per photon has been demonstrated with laser communications.
You should read the full answer for context and to view the sources cited.
That's a (potential) increase in received data rate of 1,000 times!
Screenshots from Overview and Status of the Lunar Laser Communications Demonstration:
REFERENCES: