It seems to me you've got two questions: 1) Why can open-expander-cycle engines be larger than closed-? and 2) Why aren't there more expander-cycle sea level engines? Your question[s] shows you are already aware of the scalability problems that haunt expander engine designers. Before we can look at where such problems originate, let's talk a bit more about how expander-cycle engines operate.
An expander-cycle engine is a pumped engine, but rather than using a gas-generator (which is basically a small rocket engine, but burned with an O/F ratio far off of the stoichiometric ratio to keep exhaust temperatures low) to power its turbopump, an expander cycle uses the phase change of its liquid propellant to provide the pressure needed to run the turbopump. This phase change is induced by regeneratively cooling the combustion chamber, making expander engines extra efficient as it utilizes what would otherwise be waste heat to do the work required to run the turbopump.
The takeaway here is that expander-cycle engines require waste heat to do the work of turning the pump. This is where we start to run into scaling issues. If you double the dimensions of your combustion chamber, you require 8 times the volume of propellant to fill it, but you only have 4 times the surface area from which to obtain waste heat. Eventually as you scale it up, you will run into heat flux issues where you simply cannot obtain enough work from the limited cooling area to pump the volume of propellant required to run the engine.
But you can postpone that issue if you can keep your combustion chamber smaller. This has the delightful advantage of increasing your combustion pressure and consequently your performance. The trick here is preventing backflow problems. If you're piping your turbopump exhaust into your combustion pressure, you need to ensure that the pressure of the exhaust exceeds your chamber pressure or things will begin to flow the wrong way. This limits your chamber pressure, as your turbopump exhaust pressure will never be very high, due to it all coming from the gasification of your propellants.
In fact, it even limits how efficient your turbopumps can be. If you design a pump with a very high pressure-ratio (a ratio of the pressures of the flow driving the pump vs. the flow being driven by the pump), then your pump feeding propellant into the chamber could cause your chamber pressure to exceed your pump exhaust pressure even before combustion begins. And that's a shame, because more efficient turbopumps require less work to drive the same volume of propellant.
All these backpressure problems go away if you disconnect your pump exhaust from the chamber entirely. Now you've got the open expander cycle. Now you can make a higher-pressure chamber and a more efficient pump and you can crank your thrust up. The open expander cycle has another bonus, which is you can provide ullage to your tanks just by venting the boiloff through the pump system and out the nozzle. Another bonus is you also start your pumps spinning, so it's incredibly easy to re-light your engine and you get virtually unlimited ignitions. There's a reason the RL-10 is so successful.
Now onto question two...
Here the speculation begins on my part, simply because well, sea level expanders do exist. Or they're about to. At least, JAXA is building one and I feel pretty confident that they'll succeed. So I'll answer instead "why aren't there more"?
I think the answer is threefold. Firstly, many of the advantages of an open-expander-cycle engine aren't required at sea level. You don't need free ullage, you only need one ignition, and super-high TWRs aren't that helpful. As a result, people have been building gas generator boosters & sustainers for a very long time, and a substantial technology gap has emerged. Why use poorly-known technology when you can use very well-understood technology? Maybe with Mitsubishi's efforts, this technology gap will start to close.
Secondly, expander engines have historically been INCREDIBLY expensive to produce. Since you get more performance by more efficiently cooling your combustion chamber, the regeneratively cooled chambers (and nozzles) are full of very very dense holes. That's very hard to manufacture traditionally This is the same reason the SSME was so costly (among others), and it's why it's often cited that the RL-10s are the most expensive part on any rocket they ride up on. This goes hand-in-hand with the first reason. If you need the advantages of an expander-cycle, the costs may be justified, but they're not for a sea-level booster motor. I believe the cost will begin to go down substantially as additive manufacturing matures and we might start to see more sea-level expanders.
Finally (and my shakiest answer), I think it's also because you can't have as large a nozzle at sea level. If your expansion ratio is much too high, you're going to underexpand your thrust too much and at best loose thrust and at worst destroy your nozzle. I've been saying that expander cycles get their turbopump work out of cooling the combustion chamber, but they also get it out of cooling the nozzle. Not to mention that nozzles are also subject to the square-cube law, and it turns out that big sea-level optimized nozzles just won't give you the work you need to run your larger pumps. Your mileage may vary.