Apollo missions were on a free-return trajectory which limits your initial Lunar orbital insertion inclination close to the Earth-Moon plane. Any orbital inclination change is at that stage rather prohibitive in terms of required delta-v for the Lunar Module both on descent as well as later ascent phase to match Command Module's orbit:

Sketch of a circumlunar free return trajectory (not to scale). Source: Wikipedia on Free return trajectory
Farthest landing from the Lunar equator was during the Apollo 15 mission with its landing site at the Hadley Rille/Apennine Mountains, a bit more than 26° Northern latitude, with Apollo 17 closely following, landing at roughly 20° North in the Taurus-Littrow region. Other Apollo mission all landed a lot closer to the Lunar equator, some at slightly Southern, some at Northern latitudes:

Apollo landing sites. Image credit: NASA, Source: Smithsonian National Air & Space Museum
There would be other problems that you mention with landing at the Lunar poles, including surface / contact temperature delta that's a lot more challenging to handle due to surface convection than gaining or losing heat due to radiation alone in vacuum, but the missions were simply not designed for that. One thing that you gain with near-equator landing sites is ability to reuse slight surface rotation in your favor both on landing and liftoff, but more importantly, you also don't have to crank your orbital module's orbit to near polar inclination, which is both time and delta-v consuming.