The Navigation Museum of The Institute of Navigation has an Apollo Sextant on display:
(Apollo Sextant (source))
The item description mentions it is fabricated in 1960, just after the Mars Study. In fact, the study is referenced:
In 1957 The Lab made a proposal to the U.S. Air Force, for a 150 kg., unmanned probe, which would fly by Mars and take high-resolution photographs at close range. The system was primarily inertial, but to correct for drift, it would carry "...a space sextant to make periodic navigation angle measurements between pairs of celestial objects: the sun, the near planets, and selected stars."
Hence, I think it is safe to assume that the instruments and processes described in this document are very heavily influenced, if not exactly the same, as in the Mars Probe study.
While I have not found a relevant report on navigation for the Mars Probe study, I did find report R-500 "SPACE NAVIGATION GUIDANCE AND CONTROL", published in 1965, by Dr. C.S. Draper and, among others, Dr. R.H. Battin, who was also involved in the Mars Probe study.
The sextant is described in section 11-3 "GUIDANCE, NAVIGATION, AND CONTROL INSTRUMENTATION IN APOLLO", in particular on page 11-53 and further. It is accompanied by the following schematic:
(Schematic diagram of the sextant in Apollo Block I (source))
In section 11-4 "OPERATION MODES OF GUIDANCE, NAVIGATION, AND CONTROL APOLLO COMMAND MODULE BLOCK I", automatic star tracking is described on page 11-76 and further. In particular:
The automatic star tracker on the sextant provides the capability of automatic IMU alignment as illustrated in two steps in the figures on the opposite page. Without astro- naut help, however, the star tracker cannot acquire a known alignment star unless the IMU is already roughly aligned to provide a coarse direction reference. The automatic IMU alignment capability described here, then, is most useful to re-correct the IMU drift after a long period of IMU operation,
So, as long as it had an roughly known orientation, the on-board computer could fine-tune the alignment using the automatic sextant by closed-loop control on the error signal (difference between expected star location and observed star location) using a fairly simple photometric sensor. Presumably, the initial lock could be achieved with a sufficiently accurate launch.
Volume 2 of the report provides a ton of details on the actual procedures, computations and details on the sensors.