SpaceX chose to use RP-1 because the development path for a good RP1 engine was a full decade shorter! Plus many other factors favoring KeroLox for launch from Earth surface, for example the very high energy density of RP-1 allows smaller tankage, which is a great advantage when flying through the atmosphere. The Excellent thrust density of RP-1 also allows simpler engines to deliver good thrust to mass ratio, which is the most important criteria for launch engines.
What is this "RP-1's lower energy density" you mention? It beats the pants off both Methane and Hydrogen for energy density. It actually beats any other fuel, short of a few really strange exotics like Fluorine-based rockets.
SpaceX chose Methane for the Starship's Raptor engines because it is a lot easier to work with than Hydrogen, provides easier tankage, and makes for a much cheaper engine and plumbing. It is not as good in performance for vacuum operations, but is a lot better than RP-1. Crucial for SpaceX, there is a relatively easy pathway to making both Methane and Oxygen using materials on Mars, which is their ultimate goal. The same is (enormously) not true for Hydrogen. Longterm storage of Hydrogen requires much fancier, larger and extensive infrastructure than for Methane.
Methane is not the ideal fuel for operation from earth surface, but it is ... ok. More efficient than RP-1, reasonably good thrust density, and much superior performance once out of the atmosphere. It is, as you noted, a good compromise, and "the best of both worlds"
Hydrogen provides one thing, and only that one thing, better than any other fuel. It provides more thrust per mass, thus more efficiency, than anything else. This is a real advantage out in the vacuum of space, where you need to get the most speed out of a given mass of ship possible.
It require ludicrously large fuel tanks, because the fuel's natural density is so low.
It requires somewhat fancy tankage, because the storage temperature is so incredibly cold.
It also requires pretty fancy tankage, plumbing and handling, because it leaks through cracks that other gases would never notice. It reacts to contaminants in quite amazing ways. And most metals, when cooled to liquid hydrogen's temperatures, become brittle due to both temperature and because the hydrogen seeps into most metals.
And forget about using any organic/plastic plumbing. It just falls apart at those temperatures. This makes hydrogen engines expensive and difficult to design, as one is more severely limited in materials, lubricants, etc than with other fuels.
BO chose Hydrogen because it is fashionable and "green". It is, frankly, a ludicrous fuel choice for their New Shepard suborbital hopper. That hopper flies in exactly the regime where KeroLox > MethaLox > Hydrolox.