I've heard that the size of Saturn V was pretty much as big as could've been built in the 1960s. And the Soviet N1 did basically fail because it was too large for their infrastructure and engine size at the time. Generally, there's an economic advantage with larger scale, given enough demand for the-bigger-the-better one piece payloads, which I want us to fictionally assume here. What are the technical challenges that limit how large a chemical launcher, from the surface of Earth to orbit, could be built?
SpaceX is proposing launch vehicle, as a first stage, and a second stage that would transit to Mars. Interplanetary Space Transport (ITS).
The final size is not really confirmed, and they should actually build it before we compare it, but using chemical rockets, it is quite a bit larger than a Saturn V.
The Saturn V used 5 F-1 engines with around 1.5 million lbs of thrust. The ITS first stage is planned to use 42 (!!!!) Raptor engines for a total of 29 million lbs of thrust.
Clearly larger than a Saturn V is in the realm of possible.
Large engines are hard, as the F-1 engine development showed, and as the Soviet experience that lead to 30 NK-15/NK-33 engines for the N-1 showed.
SpaceX has taken a middle of the road approach, but closer to the N-1 approach with fairly large numbers of medium sized engines.
Moving the booster around is quite tricky. NASA solved it by barging the large components. US Rail does not support such large objects. Soviet rail seems like it was able to handle larger as most of their components were moved by rail.
SpaceX plans to solve it by flying the vehicle around. (Or at least manufacturing near the launch site, and after that, land and relaunch mostly in place).
It's generally thought that the Sea Dragon design was technically feasible; this would be 18000 tons (6 times the mass of Saturn V) at liftoff and deliver 500 tons to LEO, comparable to the SpaceX ITS in expendable mode. Mass fraction and specific impulse would be worse than Saturn V but economy of scale would make it more cost effective per payload ton. It was never produced because the demand for such large payloads never arose, of course.
I don't know of any specific hard technical limitation to going bigger than Sea Dragon, though mass fraction probably gets poorer and logistic issues get hairier as you embiggen.
Engines haven't changed much, so most improvements to size would likely come from the use of lightweight composite materials. The performance of these isn't as much better than aluminum alloys as is sometimes imagined though, so I doubt that would improve the situation a whole lot.
Nuclear thermal launch would improve payload fractions. The ability to use purely hydrogen propellant alone would dramatically improve exhaust velocity and thus the payload fraction of the rocket. This is essentially the only benefit, as in all such cases the temperatures and pressures are limited by the properties of chamber materials. It's quite a benefit though. The logarithmic term in the rocket equation makes high exhaust velocity extremely desirable. The key capability tradeoffs of course come from the high cost of nuclear systems and safety issues. It is worth noting though that no practical research has been in this area for several decades, so it's not clear what thrust to weight ratios could be achieved today.
The major technical challenges of up-sizing rocket boosters, as I see it, are in two major categories: weight, and risk mitigation.
The most profitable business is weight reduction. There is money to be made on developing future materials (metals, and fuels) and structural design. These areas are relevant because the rate at which the size of the spacecraft increases, currently, will match and overtake the power efficiency very fast. Which is why we see the SpaceX Interplanetary Transportation System as three smaller craft taped together for added boosters.
Therefore the best thing for weight, multiple layered systems, is also the best thing for mitigation of risk for these "glorified explosions". Realize that not only are you controlling an explosion to propel these things into space, you are also preventing a fuel supply from igniting mid-trajectory. This is a very expensive cargo to be taking from point A to point B, and having a lesser amount of fuel mid-flight and capital loss to control is always prevalent.