Are most trips to the Moon actually around 3 or 4 days, and not the 5.0 days of a half-ellipse?
The 5 spacecraft of the US Lunar Orbiter program took 3.8-3.9 days to reach lunar orbit.
The Ranger and Surveyor missions were on faster trajectories -- around 2.7 days from launch to lunar impact or landing.
The early Soviet Luna series missions that I've looked at took 3.3-3.6 days to reach the moon -- some were orbiters, some were (intentional or unintentional) direct impactors, some were direct soft-landers. The later sample-return missions/attempts (e.g. Luna 20) took 4.5 day flights.
I'm not sure what motivated these different trajectories. Ranger and Surveyor both had solar panels, so shouldn't have been "racing the clock" to complete their missions before batteries ran down. There may have been some drive to make the flights short simply because these early complex spacecraft were not very reliable; you'd want to get to the moon before things broke down.
I don't know if the early lunar projects started from a desired flight time, factored with the available launcher, to determine the maximum spacecraft mass; or if the final design of the spacecraft factored with the launcher to determine the minimum flight time, or some iterated back-and-forth combination thereof.
The Apollo landing flights generally took about 3.3 days from launch to lunar orbit. Apollo 8 was faster, not being burdened with a 15-ton LM, and reached lunar orbit in about 2.9 days.
Some more recent missions have taken much more leisurely approaches, possibly because progressive orbit-raising can be done with a smaller, lighter motor and the hardware is reliable enough to allow the longer flight time.
ISRO's Chandrayaan-1 took two weeks plus another week to reach its final orbit -- obviously not a Hohmann-esque single translunar insertion burn!
Likewise, JAXA's SELENE/Kaguya took more than two weeks to reach lunar orbit.
Or is there something about the mass of the Moon that makes the trip naturally quicker?
At some point in a translunar trajectory, the Moon's gravity does start accelerating the spacecraft toward the destination; for the Apollo missions it would be about 2.5 days into the flight when the moon's gravity began to dominate over the Earth's. So it's not surprising that the flight times would be significantly less than 5 days, but the exact math for the "Hohmann to massive destination" calculation is beyond me.