That "Inside the LEO Doghouse" article you found is a very good one. It's written by a guy I kind of work with and if you didn't read the whole thing, it's worth taking the time to.
Open expander cycles can still be very efficient. You can get Isp with LOX/LH2 up in the vicinity of 450 s, which is only a few s below RS-25 & RL-10.
You have to understand that the biggest performance driver for an expander is the turbopumps. The biggest constraint then is what's backpressuring your turbines. Since in a closed expander, you must dump the turbine discharge into the MCC, your turbine discharge pressure has to be higher than the MCC pressure. That means that the pressure ratio across your turbines will be very limited, on the order of 1.5-2.5. The higher the PR, the more efficient your turbine will run. The only way to increase that PR in a closed expander is to run the MCC at a lower pressure (like RL-10) which limits overall thrust/performance or give your turbine higher pressure fluid going in, but that makes for more work the pump side has to perform which has to be powered by the turbine. That work, of course, has to come from somewhere, which leads me to the next big limiter on expander cycles:
The secondary limiter on expander cycles is the amount of enthalpy you can pick up from cooling your combustion devices. It's easy to make a chamber longer to increase surface area for heat pickup, but ultimately, you're temperature limited by the turbine materials. You're also infringing on your engine envelop that way. Another way is to increase the turbine mass flowrate, but again, that's more work that the pumps have to do. As you scale up in thrust class, these things like heat pickup and pump/turbine work don't scale 1 for 1. It's all very circular and iterative with closed expanders and there's tradeoffs everywhere. Unless we get massive breakthroughs in materials science, there's only so much we can get out of that cycle.
Interestingly, SSME has a chamber pressure in the vicinity of 3,000psi, so you can imagine what the pressure levels in the rest of the engine are like. Before it gets to the chamber, it has to go through all of the coolant lines, the preburners, and finally the injector. Some of that also gets tapped off to power the LPFP and then gets used for tank repress. Coming straight out of the HPFP, the LH2 is at like 7,000 psi! It's insane.
ORRRRRRRR, you can open the turbine discharge up to atmosphere as in an open expander and your turbine PR instantly goes from <2 to >10. With that, many of these issues aren't as bad and you have a lot more room to scale up. I got the cycle to close at around 45klbf for an open expander that I work on, and that is based on real hardware that's actually been tested. There's a lot more room to go even higher than that too if we had different pumps.
Yes, an open expander is less efficient, but not exactly why you think. You're right that the loss in efficiency comes from the amount of propellant that remains unburnt but directing it to the tanks for repress isn't going to do much--it takes so little to repress the tanks so you end up venting the rest. The smarter thing to do is eject that fluid out of a much smaller nozzle so you're at least getting something out of it (like Merlin, even though that's a GG cycle) that often get's used for roll control on a single engine vehicle or attitude control on a multi-engine vehicle. The losses come from the pumps having to put work into that fluid but as you pointed out, that fluid is being ejected unburnt, so you're getting way less work out of it than you put in. Regardless of if you use the propellant for repress, your pumps are still putting work into the fluid that they're not getting back out in combustion. Draw a control volume around your engine and you'll see what I mean.
Another problem is the specific heat of different propellants make some better than others for different cycles. For the expanders that we're talking about, LH2 is going to get you the most enthalpy from cooling the MCC, plus it's effortless to light, but LH2 makes a terrible core stage fuel. And with expander cycles being so sensitive to heat pickup, LH2 having a nice low critical pressure (around 200 psia if I recall) makes it better to work with since you don't have to deal with 2-phase flow in your coolant channels. Methane, on the other hand has a much higher critical pressure (almost 700 psia if I recall) so you can easily run into 2-phase flow in your coolant channels unless your pressures are high enough. 2-phase flow creates unpredictable heat transfer and can lead to hot spots and burn-throughs of the chamber hot wall if it gets bad enough. Methane is also a PITA to light.
You are correct about your additional potential advantages #2 & #3. But as long as we're comparing to SC or GG cycles, there's one more added benefit. Because everything in an expander cycle is so intrinsically coupled together, they're safe in an anomaly. Say your fuel pump begins failing for whatever reason--it's not putting out pressurized fluid, so the turbines don't get what they need and the engine slowly powers down. Say your Ox Pump or Main Ox Valve fail, your mixture ratio will start going down, your coolant won't get enough heat to power the turbines, and the engine shuts down. Say you even have a failure of one of the fuel valves--your mixture ratio is going to go way up and burn through your chamber wall. Once that happens, your coolant leaks out into the chamber instead of going to power the pumps, and you guessed it, the engine shuts down. It's very unlikely that you get a vehicle destroying explosion when an expander has a failure.
The LOX/LH2 open expander I work on bleeds about 2% by mass of the propellant overboard.
Anton brought up some good points that I missed. However, there are some things I take issue with.
What do you mean with all the ullage talk? The boiloff is the ullage. Another place you definitely don't want 2-phase flow is going into your pumps, so I'm not sure what you mean by running the boiloff through your pumps.
Also, a lot of engines start with the pumps already spinning. That's just called tank-head start like the staged combustion SSME does and that doesn't really have much to do with how easy an engine is to light. The propellant combo has a way larger effect on ease of ignition. And "virtually unlimited" starts is not really a thing that exists. Starting a rocket engine takes a real toll on all of the hardware. Pumps are running off-design, so seals rub, and if they're hydrostatic bearings, those are rubbing too. The chamber takes a beating from the mixture ratio excursions and heterogeneous flow distribution, especially if it has to pass through stoichiometric, and will eventually have a burn-through. That's why # of starts is a very high priority design requirement when we're setting out to develop any new engine.
You're right that RL-10 is an extraordinarily reliable engine though. But it's not the go-to for upper stage engines just because it's an expander. It's because it has excellent performance and reliability. The reliability is key when you're talking about upper stage engines, where you often need multiple starts. Part of that reliability is from it being an expander though--the fewer combustion devices you have, the better. GG's are often used as upper stage engines too though--SpaceX M1vac and J-2X. I also believe that SpaceX's Starship upper stage will also be powered by staged combustion Raptors.
I was told by a company insider once that the RL-10 basically subsidizes everything else at Aerojet until RS-25 restart came along. The cost with additive has made things cheaper though. The entire development program I have been working on for an open expander has been under 18Mil. But simply for hardware and labor, we're only talking about 2-3Mil per copy. It is almost entirely additive. The reason the SSME/RS-25 is about $40M per copy and a 7 year lead time is mostly the nozzle. The nozzle has 1,080 individual coolant tubes that have to be braised together. Much of that work gets done by hand. Until recently, that would take 6 YEARS per nozzle. I think we've got it down to 3 or 4 years now, but not because of additive, just process changes. NASA is incorporating additive into RS-25 bit by bit now to reduce cost, but it's progressing very slowly.
You mention nozzle size at SL too. There are technologies to mitigate that now. Aside from Aerospike, which has been around for a long time, we now have Thrust Augmented Nozzles and Dual-Bell nozzles, but both of those are at least 10 years away from flight readiness.