As mentioned here, the Apollo CSM was not originally conceived solely for the lunar landing mission, but rather as a general purpose spacecraft for NASA's long term plans which included crewed space stations in Earth orbit as well as lunar missions.
According to Chariots For Apollo, a three-crew lunar-capable spacecraft was being discussed as early as summer of 1959 as a follow-up to Project Mercury. By late 1960, the project had its name, and requests for proposals for a 3-crew ship were going out to the aerospace industry.
Through 1961 the concept was to use this general-purpose spacecraft along with an additional descent stage to land directly on the moon without an LM; this was considered the simplest way to use the Apollo CSM in a lunar landing.
The lunar orbit rendezvous concept was driven by the goal of minimizing the weight going to the lunar surface. LOR was being seriously considered in late 1961 with both one-crew and two-crew landers, but the capabilities of the one-crew lander were really minimal:
There were illustrations and data on a "shoestring" vehicle, one man for 2 to 4 hours on the moon; an "economy" model, two men and a 24-hour stay time; and a "plush" module, two men for a 7-day visit. ... Arthur Vogeley pictured the shoestring version as a solo astronaut perched atop an open rocket platform with landing legs. To expect Gilruth's designers to accept such a "Buck Rogers space scooter" would seem somewhat optimistic.
The single-person landing craft would have had an unpressurized cockpit, and would not have been suitable for the multi-day stays of the later Apollo mission. The only accomplishments of the mission would be a brief "flags and footprints" EVA and collection of a much smaller payload of lunar surface material, as compared to the relatively broad range of experiments performed by the Apollos. Crew transfer to and from the landing craft would be EVA.
I don't know if a pressurized-cabin, multi-day-stay, single-crew LM was ever considered. The second crew member certainly helped with the workload during the final part of the descent and landing (though that was probably not obvious in 1961), and provided tremendous insurance against the possibility of accidents during lunar surface work.
By the end of 1961 the LOR plan was firm on the 3-crew CSM + 2-crew LM, and while there was still ongoing debate over the choice of mission mode, by mid-1962 NASA committed to LOR.
There was some thought given to do an LOR mission using a Gemini-based 2-crew CSM and a "shoestring", open-cockpit, 1-crew lander, but it lost out to the 3/2 Apollo configuration. While it may have been possible to launch such a mission on the proposed smaller Saturn C-3 booster, it would have been severely compromised in several ways compared to the Apollo configuration, and I think this gets to the heart of your question:
My concern here is to understand why, in the context of a race for the first man on the moon, where each additional kilogram sent to its surface involves tons of extra fuel, the choice of putting two men on the moon's surface on the first try may have prevailed over only one.
The Gemini spacecraft was extremely cramped for two people; they would have been confined to their couches for the duration of the mission. An Apollo-length duration was certainly possible, as proven by Lovell and Borman's 2-week Gemini 7 flight, but it would have been decidedly unpleasant.
Gemini was designed to operate in Low Earth Orbit, less than an hour from splashdown at any given point in the mission, so it had nowhere near the amount of system redundancy that Apollo did. The safety rules in place for Apollo simply would have been impossible for a Gemini-based architecture, and adding the necessary backup systems would have increased weight to the point where a C-3 couldn't do the job. In particular, an Apollo-13-style accident would certainly have been a loss of crew event; the Gemini landing craft wouldn't have provided the resources needed to recover that mission.
You can point to many features of the Saturn/Apollo design where safety, backups, extensibility features, and even creature comforts were chosen over minimal weight: 2 independent guidance computers on the LM, 3 fuel cells on the CSM, internal docking tunnel between the CSM and LM, CM's lower equipment bay, CSM's SIM bay (actually ballasted in the early flights!), retention of the oversized (but well proven) SPS engine, and so forth. When these features started adding up to more weight than the booster could handle, the solution was to add a fifth engine to the booster's first stage rather than to compromise features.
A 2-crew orbiter + 1-crew lander LOR architecture was chosen by the USSR, with many of the same drawbacks; the N-1 booster had TLI capability somewhere between the Saturn C-3 and C-4 proposals. The orbiter (Soyuz-based) would have been significantly more comfortable and capable than Gemini, but the lander wasn't capable of long stays and would have to be boarded by EVA.
