This is long, but it's still barely scratching the surface. I'll try to take it nice and slow, so bear with me.
Generally, when dealing with radio links, it's a good idea to start with a link budget.
In a link budget, you start with your transmitter's output power, end with the signal strength actually received (most often, this is the minimum required signal strength at the receiver), and in between, list all factors that contribute to either loss or gain of signal strength. This can be everything from antenna gain to attenuation because of absorption in the atmosphere.
Very broadly speaking, there are three ways you can increase signal strength:
- either by pumping more power into the transmit-side antenna,
- or by having a bigger antenna at the transmit side,
- or by having a bigger antenna at the receive side,
or of course some combination of those.
"Bigger antennas" is a bit of a misnomer, but it's close enough to get a basic understanding of what's involved physically. What you really use in such a case is an antenna with a higher antenna gain.
In radio, one often refers to an "isotropic" antenna. This is a purely theoretical construct, which cannot exist in the real world, but which is said to radiate equally in every direction, and to radiate exactly 100% of the power it receives in the form of radio signals. In physics terms, it's a RF point source. Real-world antennas can then be compared to such a theoretical antenna.
When an antenna focuses the signal in some particular direction, it accepts the same amount of power but transmits (or receives) that power in a narrower beam than the isotropic antenna does. We call that "antenna gain", but it's important to keep in mind that there is no actual gain involved; except for losses (which are typically resistive or inductive), the same amount of power is always involved. An antenna can even legitimately have negative gain compared to an isotropic antenna, but such cases are not relevant here. Look at a satellite dish, or a rooftop TV antenna; those are two different types of the general class of directional antennas, so called because they concentrate the power into a beam that is much narrower than that of an isotropic antenna. For spacecraft that travel far from Earth, bigger parabolic antennas (like that satellite TV dish) is used; for a real-world example of this, consider the ground stations for NASA's Deep Space Network.
There is, of course, one more option. Instead of simply pumping more power into the antenna, you can design your signal encoding in such a way that it can be decoded at a lower received power. Very often, this involves reducing the transmission bit rate. Basically, when each bit ("symbol") is allowed more time, the receiver has more time to determine its value, and it can use this time to metaphorically dig the signal out of the noise. The ultimate limit to this is described by the Shannon–Hartley theorem.
Given all this, we can see what can be done given your constraints.
One side of the link (the satellite) is in low Earth orbit, and the other side of the link is on the ground. Thus there is little that can be done about the distance that needs to be covered by the signal.
The frequency needs to be relatively high, so that antennas are physically smaller, because things that are sent into space need to fit inside the spacecraft that are used to launch them. Unfortunately, this means that the signal losses over a given physical path are larger, because free-space path loss scales with distance in terms of wavelengths, and wavelength is the reciprocal of frequency. Thankfully, physical antenna sizes go down with frequency, so we can compensate for this.
The transmitted power level is low. This means that we need to be able to pick out and decode a signal that may be close to, or possibly even below, the ambient noise level at the frequency in use. This benefits from long signal integration times, which reduces our attainable symbol rate. Thankfully, this is a good thing for the power budget of the satellite; you can get away with less on-board power generation capability, and need less cooling. (Believe it or not, but getting rid of waste heat is one of the hardest things in spacecraft design. Reducing excess weight is another.)
Assuming that a low bit rate is acceptable, we can choose the signal encoding such that the required received power level is attainable given the other constraints. This is good, because we really don't have much left to play with!
All this means that you would want to pick a frequency that is as high as reasonably possible (while avoiding the atmospheric absorption bands, for example that of water around 2.3 to 2.4 GHz or so) to get more antenna gain with a same-physical-size antenna, and put a relatively large antenna on the satellite in order to listen in the direction of the Earth without listening to too much else. The physical size of the antenna and the frequency would probably be selected together, such that the area seen by the satellite's antenna is no larger than the Earth itself in its intended orbit, and might be smaller. You would choose a signal encoding that ensures that the signal can be decoded at the received power you would be seeing, with a good deal of margin for loss of signal strength. Hopefully you don't need to transmit a lot of data in a short period of time; if you do, you might have to go back to the drawing board.
All of those factors (frequency, antenna type and size, signal encoding, bit rate) affect each other, so you would be choosing them together. If one of them is fixed (for example, the physically largest dimension of the satellite will be restricted by, if nothing else, the payload fairing of the chosen launcher), then something else has to give.