Were there specific breakthroughs which allowed SpaceX to get such an incredible thrust to weight ratio? The next closest American engine was almost twice as heavy (199 vs 82), and the next closest overall was still quite far behind (199 vs 159).
As I understand the two major items are continuous refinement of the design, and 3-d manufacture of key components of the engine. The Merlin is the most recently designed engine used on a large scale, and thus can better take advantage of modern design of such components.
Looking at some of the other top contenders, it seems that most of the very efficient engines were developed in the last decade or so. Likely computer aided design, complex simulations, and other such technology has allowed more careful design then in the past.
It also seems like the Densified Liquids was a major contributor. That was the major difference between the Merlin D and the Full Thrust, which gained about a 30% ratio.