So we are looking to specify an
X-band high-gain antenna for the communication on a Entry-Descent-Landing vehicle
at circa 10 AU, and want to
determine the diameter of the antenna, power usage, transmission rate, and how to determine the strength of the signal being sent back to Earth.
This is what is called a link budget calculation.
As @A.Rumlin points out it's a big task, but you can make some estimates fairly easily by seeing this answer to How hard is it to receive direct signals from vehicles on the surface of Mars, and has anyone other than the DSN done so? But I'll add an even shorter answer here based on your application.
From here:
Link Budget
From this answer which is from this answer:
$$ P_{RX} = P_{TX} + G_{TX} - L_{FS} + G_{RX} $$
- $P_{RX}$: received power by spacecraft
- $P_{TX}$: transmitted power by wristwatch
- $G_{TX}$: Gain of wristwatch's transmitting antenna (compared to isotropic)
- $L_{FS}$: Free space Loss, what we usually call $1/r^2$
- $G_{RX}$: Gain of spacecraft's receiving antenna (compared to isotropic)
$$G \sim 20 \times \log_{10}\left( \frac{\pi d}{\lambda} \right)$$
$$L_{FS} = 20 \times \log_{10}\left( 4 \pi \frac{R}{\lambda} \right).$$
I'll work an example, then you can put in your own numbers.
Let's assume you are using X-band at 7.5 GHz and your spacecraft as a 1.5 meter dish. The wavelength $\lambda = c/f$ is 0.04 meters so your spacecraft's dish has a gain (relative to isotropic) of about 41 dBi.
Let's say your receive antenna is one of the three big 70 meter Deep Space Network dishes, then $G_{RX}$ is 75 dBi
The "free space path loss" $L_{FS}$ (basically $1/r^2$) at 10 AU is about 293 dB. Yikes!
If your spacecraft's transmitted power $P_{TX}$ is 5 Watts or about 7 dBW, then we have
$$ P_{RX} = 7 + 41 - 293 + 75 = \text{-170 dBW}$$
That's $10^{-17}$ watts.
If your receiver is cooled to "say 20 Kelvin, the noise equivalent power will be about $k_B T \times \Delta f$ where $k_B$ is the Boltzmann constant. For 10 kHz that's -189 dBW or about $1.3 \times 10^{-19}$ watts, much lower than the received signal strength. For more on that see this answer for example.
This means that you could trim down the size of the transmit or receive antenna, or increase the bandwidth.
You also have to perform the same calculation for the uplink, but this one is usually easier because all the terms are the same except the transmit power and the receiver noise temperature.