For maximum thrust, the pressure of the exhaust at the "exit plane" (the end of the nozzle) should be equal to the ambient air pressure it's exhausting into.
The larger the area of the nozzle's cross section, the lower the pressure of the exhaust stream at that point. Intuitively, if the exhaust is at higher pressure than the air when it leaves the nozzle, you would be able extract a little more work from it by containing it in a slightly bigger nozzle.
So, when exhausting into vacuum -- near-zero pressure -- you should be able to expand the exhaust stream nearly infinitely and still get additional thrust. However, you very rapidly reach the point where the thrust gains are canceled out by the additional mass of the nozzle, and the nozzle has to be limited in diameter to the size of the rocket in any case, so vacuum engines remain finite in size.
The ratio between the area of the nozzle throat and the end of the nozzle determines the pressure drop from the combustion chamber pressure to the exit pressure. Typical first-stage engines which have to exhaust into sea-level air pressure commonly have an area ratio of 10:1-20:1. Engines designed for vacuum operation commonly have ratios above 50:1; the version of the RL10 used on the Delta IV second stage has a ratio of 280:1!