As far as I know liquid fueled rockets suffer from lowered efficiency when using less than full throttle since the pressure of combustion is lowered and so thermodynamic efficiency. Could a variable throat diameter nozzle fix this by restricting the outflow of the gas and so raising back up pressure in the combustion chamber? Or would the divergent part of the nozzle also have to vary in shape? Are there any succesfull designs that maintain engine efficiency at lower throttle settings?
According to Sutton's History of Liquid Propellant Rocket Engines and Rocket Propulsion Elements, the sustainer engine in the Lance missile could throttle from 22 kN down to 62 N, a better than 350:1 ratio (i.e. a minimum throttle of about 0.3%) using a moving pintle injector with a 15% efficiency loss at the low end. The pintle injector is used in two other notable throttleable engines: the Apollo LM descent engine and the SpaceX Merlin.
Normally, extremely low throttle settings lead to flow separation in the nozzle: the exhaust separates from one side of the nozzle wall and adheres to the other, unbalancing the thrust (an ordinary kitchen faucet displays similar behavior when opened very slightly). The Lance engine has an extremely short nozzle with only a 4:1 expansion ratio in order to avoid this, with consequently poor specific impulse of 227 seconds at full thrust; normally you expect sea level Isp in the 270s from the IRFNA/UDMH propellant combination.
I'm skeptical that dynamic control of the throat diameter could be practical. Without it, to do very deep throttling in-atmosphere, you need either a stubby nozzle like the Lance's, or one of the other standard altitude compensation techniques to control exhaust flow separation: aerospikes, expansion-deflection, stepped nozzles, etc. Not a lot of research has been done here because there's not much demand for deep throttling on first stage engines.
For vacuum engines, things are easier. The LM descent engine was designed to throttle down to 10%; in practice the minimum used was around 30%. At 30% thrust, specific impulse dropped from ~305 to ~298, less than 3% reduction. There was a throttle range from 65%-92% that was avoided because it would cause excessive erosion of the nozzle, but no significant problems at the low end that I know of. There was no need to go as low as 5% on the LM's flight plan—high-thrust suicide burns are more fuel-efficient than long gradual descent—but if it had been necessary, it was probably doable.