I was wondering how the Apollo guidance computer handled the radiation encountered in space
TL;DR: It was so busy getting stuff done, it didn't care.
Being old, slow, massive and inefficient (by any modern standards, not by those in 1965) is a huge benefit when it comes to radiation hardness.
Let's start with the memory: Changing a bit in current S(D)RAM cells is trivial - introduce a bit of charge in the wrong place and the bit is lost. This couldn't happen in the AGC: The program was stored in rope core memory - losing a bit means physically burning a wire, which can't happen due to radiation alone. The RAM was similar, the bits were stored in the magnetization of a macroscopic piece of ferrite. Changing this requires a proper signal shape and more energy typical radiation can deliver.
The main part of the processor was made out of discrete logic gates - the size of each of those was on the order of 100µm - compare that to the current size of about 30 nm. That means the area was larger by a factor of 9 million! This is also about the difference in charge needed to change the state of a transistor and particles that carry this amount of energy are exceedingly rare. On top of that, the voltage margins were huge, the AGC ran at 14V, ten times more than modern circuits.
To get around all remaining problems, the programming and physical structure of the CPU had additional safety checks and parity bits installed - but these were not so much because of possible radiation issues, but for errors introduced through noise and possible failures of components. After all, the time estimated until a real hardware fault occurred was estimated at around 1000 operating hours in the mid-60s. Soft-faults due to bits transmitted incorrectly inside the computer were even estimated to be an order of magnitude more likely. There are some numbers about this in the documents on this page.
The Apollo Guidance computer (AGC) controlled the jet propulsion and thus maintained the spacecraft's altitude and navigation. As you pointed out, any defects or errors in the AGC could result in a crash or the crew becoming stranded in space. Once the spacecraft left Earth's atmosphere, the risks increased exponentially due to large amounts of radiation that could cause changes to the computer's memory. Two solutions were initially proposed but neither was actually used:
- A triple redundant system was proposed so that all calculations would be carried out by three identical systems and a voting system would choose the correct information. However, a major con was that the weight of the three computers had to be carried on both the lunar lander and ascent stage and since each AGC weighed 32kg, it would add up quickly.
- The second proposed option was to build in a error code checker that would have a fault detection system.
- In the end, neither solution was used and instead a sturdy enclosure around the AGCs was used to protect it from radiation. They also had a built in manual override to ease the astronauts onboard.