....were rectangular lenses used?
I don't think so. Currently I haven't found absolute proof, but from what I found in the paper below:
From Lunar Reconnaissance Orbiter Camera (LROC) Instrument Overview update: also available here (not paywalled, thanks to @organicmarble's comment) and discussed further below:
- The visible lens has a 6.0 mm focal length and a focal ratio of 5.5 It has six fused silica elements and is optimized for the wavelength range from 395 nm to 690 nm. It provides a 90◦ field of view over the full 1008 pixel width of the detector.
- The UV lens has a 4.7 mm focal length and a focal ratio of 5.1. It has five elements and is optimized for the wavelength range from 290 nm to 370 nm. It provides a 60◦ field of view over a 512 pixel width on the detector.
- The WAC electronics are designed around the Kodak KAI-1001 Charge-Coupled Device
(CCD) detector. This detector has 1024 × 1024 9-μm pixels
- Per Table 3 though, image format monochrome: 1024 samples × 14 lines
Here also, while the sensors were square, according to the ASU web page LROC Specs
- Image frame width (km): 105 km (visible monochrome)
- Image format 1024 x 14 pixels monochrome (push frame)
So the visible and UV WACs (wide area cameras) are optically quite small, with focal lengths of 5 or 6 mm only, and a ~10 x 10 mm sensor. In both cases the entrance pupil is only about 1 mm, but lenses may be as large as several millimeters since they have 5 or 6 elements.
Basically, these cameras look like tiny point-and-shoot cameras optically, or big cell-phone cameras. They are tiny, and the large size of the large rectangular baffles as nothing to do with the size of the cameras or optical system.
It appears that the visible and UV cameras have a square very narrow, rectangular fields of view. They do not use "pushbroom" imaging like the narrow field cameras.
So the giant, rectangular solar baffles make more sense now than they did when I first wrote this answer.
However how they allign to the Sun is still a mystery to me! The relative orientation of the LRO's lunar orbit and the Sun seems complicated, there doesn't seem to be an easy explanation in the article linked below, or in IAC-07-C1.7.06 Mission Design and Operations Considerations for NASA's Lunar Reconnaissance Orbiter Space Sci Rev (2007) 129: 391–419 DOI 10.1007/s11214-007-9153-y See Figures 8, 9, 15.
From Lunar Reconnaissance Orbiter Camera (LROC) Instrument Overview Space Sci Rev (2010) 150: 81–124, DOI 10.1007/s11214-010-9634-2 (paywalled) update: and also available here (not paywalled, thanks to @organicmarble's comment):
3.2.1 WAC Optics
The WAC optics consist of four optical elements: the visible lens,the UV lens, the prism and the color filter array (CFA). The visible lens has a 6.0 mm focal length and a focal ratio of 5.5. It has six fused silica elements and is optimized for the wavelength range from 395 nm to 690 nm. It provides a 90◦ field of view over the full 1008 pixel width of the detector. The lens has a design MTF of greater than 60% at 56 line pairs per mm in all bands. The UV lens has a 4.7 mm focal length and a focal ratio of 5.1. It has five elements and is optimized for the wavelength range from 290 nm to 370 nm. It provides a 60◦ field of view over a 512 pixel width on the detector. Because signal is low in the UV bands, the UV system data is acquired by summing pixels 4 × 4. The optical design provides a 4 by 4-pixel ensquared energy of greater than 80% in both UV bands. The physical diameter of each of these lenses is greater than the format of the CCD detector. A prism is used to allow both lenses to image on the same CCD. This prism provides a straight-through path from the visible lens to the detector, and a periscope-type optical path for the UV lens. The latter offsets the point at the UV image is formed laterally, placing it closer to the visible image (and therefore on the photoactive area of the detector). Each lens and the prism are integrated and aligned into a signal assembly with the WAC detector and electronics. The color filter assembly (CFA) is a 9.5 mm by 9.3 mm by 0.6 mm fused silica substrate with seven vacuum deposited interference filters laid down is the geometry appropriate for push-frame imaging. It is bonded to the surface of the CCD detector with optical cement. The final focusing of the system is done by lapping a spacer between the CCD package and the back of the prism housing. The WAC lens/prism assembly was built by LightWorks Optics and the CFA was manufactured by Barr Associates of Westford, MA.
3.2.2 WAC Electronics
The WAC electronics are designed around the Kodak KAI-1001 Charge-Coupled Device
(CCD) detector. This detector has 1024 × 1024 9-μm pixels (1018 × 1008 photoactive, others masked for background/bias signal determination) and uses interline transfer to implement electronic shuttering.

Cropped screenshot from Lunar Reconnaissance Orbiter Camera (LROC) Instrument Overview Space Sci Rev (2010) 150: 81–124, DOI 10.1007/s11214-010-9634-2 and also available here (not paywalled, thanks to @organicmarble's comment)
Fig. 4 LROC Wide Angle Camera. The width of the visible optic baffle is 15.95 m (Image credit: Mike Malin, Malin Space Science Systems, Inc.)
Thanks to @Uwe's comment here is an image found in the Planetary Society's Bruce Murray Space Image Library on the page Lunar Reconnaissance Orbiter Wide-Angle Camera (LROC WAC)
