How can the optimal number of satellites be determined for a plane of satellites? Specifically, I would like to see how to do this in general, and a specific example worked out, such as the following:
The question is badly underspecified, so let's try a more precise, but probably simplistic scenario.
You want to place a sequence of satellites in a circular orbit. All satellites follow the same orbit. The orbit is in the Earth's equatorial plane. You want to know, how many satellites you will need in total, if you want that an observer, located on a specific point on the Equator, can always "see" at least one satellite, where "seeing" means "having a direct line of sight which is at least 5 degrees above the horizon". The Earth is assumed to be a perfect ellipsoid, and we don't take into account refraction of light and radio waves in the upper atmosphere.
In that case, let $R$ be the Earth's radius (6378 km), and $h$ the altitude from its surface (780 km in your example). Let $a$ be the minimum angle of the line of sight above the horizon. Seen from a polar point of view, with the Earth centre as the origin, we can define the angle $b$ such that the piece of orbit that the observer can "see" has angle $2b$. In that case, we get the following equation:
$(R+h) \tan a \sin b = (R+h) \cos b - R$
and we want to solve that for $b$. The left size of the equation grows with $b$ and the right side decreases with $b$ ($b$ is in the $[0;\pi/2]$ range) so there will be one intersection point. It is rather tedious to make an actual expression, but numeric resolution is easy with some dichotomy. For the parameters above, this yields $b = 22.42^\circ$ (approximately), for an orbit chunk of size about 5602 km. If you want to always "see" at least one satellite, then they will have to be spaced by no much than that amount. With an orbit radius of $R+h = 7158 \, \mathrm{km}$, the orbit length is about 44975 km, and a bit more than 8 satellites are needed (so, in practice, 9 satellites).
Note that the conditions are restrictive. If the observer is not "right under" the orbit (geometrically, on a point which is not at the intersection of the orbital plane and the Earth surface) then the orbit chunk that the observer sees is smaller. If the orbit is not in the equatorial plane, then we can no longer ignore the Earth rotation and the satellite speed. For GPS and similar systems, a complex set of orbits is used (instead of a string of satellites following a single orbit), to get enough coverage with at least three satellites(*) visible from any point at any time, and preferably more, if only to be able to cope with satellite failures.
(*) Four satellites are needed to get a 3D position, but with three satellites you can compute a line of possible positions, and you can infer your position as the intersection of the line and the surface on which you are. This works well for boats, which tend to be at sea level.
What Thomas Pornin said. But I'll draw a picture:
When you plug in $R = 6378$, $h = 780$ and $a = 5^\circ$ you get $\cos(b) - 0.0875 \sin(b) = 0.8920$.
From there numeric methods can be used to find $b \approx 22.42^\circ$.
Ok this is an old question, but I think I can contribute a bit if anyone is interested!
For continuous coverage (always at least one satellite in view) or better (multiple satellites in view) the optimal configuration is usually a Walker constellation (GPS, Galileo) or a "Streets of Coverage" constellation (the wikipedia article calls this a "Walker Star" constellation - I hadn't heard it called that before). Iridium is an example of the latter. This wikipedia article gives a basic introduction, a google search on those terms will take you as deep as you like.
In terms of which Walker constellation is optimal (ie. choosing the inclination, number of planes, plane phasing), this is usually a numerical exercise of evaluating the various candidates, though some can be trivially eliminated by inspection. The allowable inclination bounds will be set by the angle b referenced above (sometimes called the Earth Central Angle) and the latitude range you intend to cover. For circular orbits, the coverage will be symmetric about the equator and uniform in longitude over a sufficient time span.
For intermittent coverage constellations, my experience has been that for average revisit times < the period of the orbit a Walker constellation will often still be optimal. Beyond that and you may be able to find something better.
Without defining an objective function (i.e. what you are trying to optimize) the question is ill-formed. What defines optimal? If I say 6 satellites and someone else says 5, without an objective function you have no objective means of determining who is correct.
Two trivial examples:
If the objective is the cheapest option and there are no other technical requirements being met, then one satellite is the answer.
If the objective is to maximize revisit time over a certain target then infinity satellites is the answer.
Both examples are degenerate and serve to illustrate that in practice objective functions tend to be weighted considerations; for example, a balance of performance and cost.
