I am curious if a rotating detonation engine were made stable, how much more efficient it would be than a standard rocket engine. With a methane or hydrogen and LOX mixture, what is the theoretical maximum ISP and how would this be determined?
A rotating detonation engine is at its heart a gas-turbine engine - i.e. it's meant for fuel to be mixed with air that you don't have to carry with you. Aerojet Rocketdyne has been testing them, but only mentions their application as a more efficient way to burn fuel--basically, the rotating shockwave helps to get complete combustion. But a rocket combustion chamber already achieves an excellent environment for complete combustion, so even if LOX were used, a better ISP is unlikely, as the maximum ISP depends on the fuel itself more than the engine.
Maximum ISP comes from burning the mixture of fuel and air, and then finding the work that could be done it you let it expand fully - whether to Earth's atmosphere or to the vacuum of space. This answer runs through those calculations for hydrolox, arriving at a absolute maximum of 528 seconds--in vacuum, with an infinitely long engine bell. The current best vacuum hydrolox engine gets 470 seconds, or 89% of maximum. The main losses for that 11% are from the nozzle not extending to infinity. If the losses were from incomplete combustion, an improvement of 25% is most commonly stated (one source), which would lead to about a 2-3% increase--maximum.
I was able to find an "ISP" for a hydrogen-air rotary detonation engine (AerospaceAmerica) stated as up to 3600 seconds at Mach 4-8, but this ISP only takes the fuel weight into account, not the oxidizer.
In summary, probably no increase over current technology, with a maximum possible increase of 2-3%.
I have done additional research on this and I have found a fairly strait forward presentation by NASA on the topic that was very informative:
One of the most useful graphs:
From this graph, you can see that the specific impulse has the possibility to be much higher. That being said, the impression I get from this presentation, is that the RDE is mainly a way to get a specific impulse that is on par with a much more complex rocket engine by getting the same specific impulse with a much lower chamber pressure. As a previous answer said, I don't know that the actual theoretical max ISP is greater with detonation, but I think it is much easier to approach that. Strangely, this graph does not seem to support this and seems to imply a different maximum.