RP-1/LOX (also known as kerolox) is run at a mixture ratio of about ~2.5-2.6, while LH2/LOX (also known as hydrolox), has optimal mixture ratios ranging from 4.13 at sea level to 4.83 in vacuum. The STS ran with a ratio of about 6.
At 4.12, the bulk density of hydrolox is 0.29g/cm³, at 4.83 its 0.32g/cm³. Bulk density for kerolox is about 0.81–1.02g/cm³, depending on mixture ratio.
This means a first stage using hydrolox would have to be about 2.5 times to 3.5 times as big as a stage using kerolox.
Furthermore, while hydrolox has a higher Isp then kerolox, building high-thrust kerolox engines is easier then building high-thrust hydrolox engines. The the RS-68A, which is the most powerful hydrolox engine ever constructed (this was not until the 90's), only produces about 3.5 million newtons of thrust, compared to about 7.7 million for the F-1 and almost 9 million for the F-1A.
For the first stage, the higher energy density of kerolox means the stage can be kept at a reasonable size and producing engines with high enough trust is feasible. A bigger first stage would not have been feasible.
For the same reasons, hydrolox was selected for the upper stages. The increased volume was not problematic, but the increased performance was sorely needed.
For the SPS (the SM's main engine), there were other concerns. Cryogenic propellants such as LH2 need to be kept cool, otherwise the fuel would just boil off. The added weight for a cryoplant and energy requirements were prohibitive. Simply ignoring boiloff over such long periods of time was also infeasible, as too much of the fuel would evaporate.
The SPS engine thus needed a fuel that is more stable, and is easy to store and handle, and is reliable. The SPS engine needs to fire multiple times, up to ten or more, with absolute reliability. Lighting a rocket engine is surprisingly difficult to do. Using a hypergolic propellant for the SPS engine means that the propellant ignites when fuel and oxidizer mix, making it easy to handle. Furthermore, the propellant needed to be storable in zero-gravity. The SPS was pressure-fed, so using a propellant that does not mix with the pressure gas was necessary. More information about the SPS subsystem can be found in the APOLLO EXPERIENCE REPORT -
SERVICE PROPULSION SUBSYSTEM
by Cecil R. Gibson and James A. Wood.
So, in summary:
- The S-IC used kerolox because of the higher energy density and thrust then hydrolox,
- the S-II (and S-IVB) used hydrolox for the better Isp and thus delta-v,
- the SPS used Aerozine50/N2O4 for long-term storability and ease of ignition and
- the LM used the same Aerozine 50 / N2O4 combination for both RCS (attitude control) as well as for the APS and DPS engine for the same reasons
Each stage used the fuel that was most suitable for the task at hand, resulting in different fuels being "optimal" for each stage. Raw performance, e.g. thrust or Isp, is not the sole deciding factor. Energy density and storability as well as ease of use for the given task also play major roles in selecting the propellant. Another great overview over the different fuels and their pros and cons can be found on Rocket propellants by Robert A. Braeunig.