The concept of hybrid rocketry is very interesting, and seems like it has many advantages over its solid and liquid forebears. But having only a superficial idea of the challenges preventing it from mainstream use (namely combustion instability and adverse grain regression rates) and not wanting to fall for a case of "what you see is all there is", I must ask what else there is that is preventing hybrid rocketry from being used for more large scale applications, instead of just suborbital joyrides and small scale amateur endeavours.
According to this recent article by one of the Space Propulsion Group partners, paraffin-based fuels give hybrid rockets more oomph, because the fuel that is exposed to combustion melts, atomizes, and becomes entrained in the flowing oxidizer [inset]. This enlarges the surface area over which the fuel can vaporize and react.
More fuel in the mix, faster burn, higher thrust. This does seem to address the problem with slow grain regression seen with other hybrid fuels. Since this makes possible single channel combustion, the problematic multi-channel design can be avoided (thus more fuel in the cylinder & no chunks of fuel flying loose during burn). Working with NASA Ames, they've hot fired hybrids with LOX and nitrous oxide. Recently they have been testing a 56-cm motor capable of 100,000 newtons of thrust.
However, they are still subject to low-frequency combustion instabilities, along with high-frequency instabilities common to all kinds of rockets. It really flickers like a candle, as you can see in this video:
The SPG website is here.
Just a quick response here, but from doing some hybrid engine research as part of a student organization I think there is also the issue of layers beneath the actual boundary layer of wax melting and causing some instabilities with the grain. I believe additives such as carbon black help this issue by lowering radiation being transmitted through to lower layers of the wax, but I'd imagine on larger rockets it also presents itself with just the sheer size of the fuel grain. I'm not entirely sure on this, however, and am still a student.
IMO it's just pretty new. Liquid fuel rockets have the advantage of the wealth of knowledge generated during the space race. Liquid fueled rocket engines are a pretty much solved problem with respect to science. The difficult/expensive part is engineering and manufacturing. If you already know liquid fueled engines then hybrids don't bring much to the table. However, the real advantage to hybrids is their accessibility to amateurs and startups.