Apollo already had some hardware for rendezvous on lunar orbit, so why not do the same on earth orbit and use smaller rocket? I suppose designing smaller rocket is simpler and also building double amount of rockets might be even cheaper than bigger rocket that needs humongous machinery. Economy of scale kind of. Was the diameter of the payload important?
Assembling or fueling the Apollo spacecraft in Earth orbit from multiple smaller launches was considered; it's referred to as the Earth Orbit Rendezvous mission mode.
The original EOR studies were based on the Direct Ascent mission plan, where the Apollo Command/Service Module would land on the moon with the help of an additional descent stage, then return, without a dedicated Lunar Module. For direct ascent, the trans-lunar stage would have massed nearly 200 tons while in Earth orbit, requiring the very large Nova 8L or Saturn C-8 booster (essentially a scaled up Saturn V with 8 engines on the first stage); splitting that into two launches in the Earth Orbit Rendezvous plan would have required two boosters of the Saturn C-4 type -- more or less a Saturn V scaled down to 4 engines. 4-5 launches of a still smaller Saturn C-3 were also considered, but that would have required orbital assembly rather than just fueling, which I don't think would have been practical.
However, the Lunar Orbit Rendezvous plan with its tiny, separate Lunar Module brought the required mass in Earth orbit down to about 125 tons, requiring only a single launcher. This was originally going to be a Saturn C-4, but as the design weight of the Apollo spacecraft grew, the C-4 got upgraded into what eventually became the Saturn V. Lunar Orbit Rendezvous was chosen because it was the fastest and cheapest solution to get to the moon before 1970.
It would have been technically possible to combine Lunar Orbit Rendezvous with Earth Orbit Rendezvous, launching the spacecraft and translunar stage on one booster, and fuel on another, refueling the translunar S-IVB stage in orbit, then going on with the Apollo mission as actually flown. This could be done with one C-4 booster (~99 tons payload) for the spacecraft and one C-3 (~45 tons payload) for the tanker.
However, it would have complicated the mission plan considerably. While waiting for the Earth orbit rendezvous, the hydrogen fuel used by the Saturn upper stages would be beginning to boil off; the sooner you get on your way, the better. Small spacecraft had been docked before -- Gemini with a small Agena target vehicle -- but it would be significantly more difficult to dock a 40-ton tanker with an 80-ton translunar spacecraft. Additional procedures for docking and fuel transfer would mean more training, more technical issues to solve, and more opportunities for things to go wrong. Two independent launches also increases the chances of a failure. There are twice as many guidance systems, staging systems, telemetry systems, and so on, the failure of any of which makes it impossible to continue the mission.
Launching just the Lunar Module on one booster and docking to it in Earth orbit wouldn't have been attractive. The LM massed only 16 tons in its final configuration, so even a C-4 wouldn't be big enough for the rest of the translunar spacecraft, and the docking maneuver would have to be flown by the entire 110-ton S-IVB/CSM stack, not just the CSM -- the actual Apollo TDE maneuver was done after the spacecraft was in translunar coast.
There would have been some economies of scale associated with building twice as many boosters, but the rockets would still have been quite large. The C-3 and C-4 were originally intended to be 8 meters in diameter. The stages would have been individually easier to transport compared to the Saturn V's 10-meter stages, but not enough easier to compensate for the complications in the mission. The payload size wasn't a constraint here; maximum payload diameter was around 6 meters, at the Lunar Module's "hips", and the translunar S-IVB stage (which would have been used in the C-4 as well as the Saturn IB and Saturn V) was 6.6 m in diameter.