The primary function of the command module was to keep three crewmen alive and allow them to survive re-entry. Those requirements weren't negotiable.
The Soyuz spacecraft to some extent divides those functions into two modules, and so its re-entry module is more compact than the Apollo CM (Command Module), but this comes with an overall complexity penalty.
The point of the LOR (Lunar Orbit Rendezvous) strategy is that it allowed the lunar landing delta-v cost to be paid by a smaller craft. The LM (Lunar Module) cabin supported about 8 man-days of operation in very cramped quarters; the CM supported 36 man-days in relative luxury. Landing the CM would require a much larger descent-ascent system than the LM — greatly exceeding the mass of the "redundant" LM cabin.
In essence, you'd be calling for a small optimization of the direct ascent strategy — leaving the TEI (Trans-Earth Injection) fuel and possibly the heat shield in lunar orbit for the landing, at the cost of requiring a rendezvous (like LOR).
The fueled mass of the Apollo LM (15 tons) is more than 7.5 times the mass of the cabin (ascent stage, dry, is 2.1 tons, and consists of the cabin, ascent engine, tankage, and various additional equipment).
The Apollo CM, less the heat shield, is about 4.7 tons. Even stripped down, say to 4 tons, you'd need 30 tons for a complete lunar descent and ascent at the same mass ratio as the LM. So to save the mass of the 2-ton LM cabin you have to incur a 15-ton hit in the LM.
If you define a two-man mission instead, the CM could be scaled down to perhaps 3 tons, which leads to something like a 5 ton penalty. Remember, though, the Apollo CSM (Command/Service Module) was conceived as a general purpose 3-man spacecraft to ferry crews to space stations and so forth, not solely for moon landings. (After the Apollo lunar program, the spacecraft was used for Skylab missions and the Apollo-Soyuz Test Project).
A single cabin solution also removes the lifeboat capability that saved the lives of the Apollo 13 crew, though that wasn't a primary driver of the LOR architecture.