I don't understand why a computer was needed at all, either on the ground or inside the space craft.
As Ben (PearsonArtPhoto) pointed out, computers have always been a part of launching rockets. By no means an optional one. Computers are needed to avoid collisions with the debris around earth, to auto-pilot spacecrafts and to monitor mission data (sensors, the live-support systems etc) that can be learned from to enhance future missions.
The real challenge seems [...] entirely unrelated to math, or at least "real-time" math.
Despite there being many other challenges, the "real challenge" during the mission is mostly the computational one. Everything else has to be figured out before liftoff. If not, a single error may be fatal. Many exceptional events may happen while a mission, especially in the first few minutes - it is impossible for us humans to predict them in real-time.
I don't understand why a much more powerful computer would make any difference.
The memory-cycle time for the Apollo Guidance Computer was 11.7 microseconds. A single-precision addition in the assembler language took two memory cycles. Other basic instructions needed 1, 2 or 3 memory cycles. One memory cycle took 24 cycles of the 2.048 MHz clock. (by Uwe)
Despite being pretty slow compared to the technology of today, no human could possibly do calculations at that rate. That was enough to go to the moon. But the faster, the further you go, and as a ships complexity increases, it ceases to be enough. My first phone ran at up to 1.2GHz. Phones (especially Android) in reality cannot do computations as fast as their CPU technically could though, as they mostly run virtual machines (the JVM) and are busy with computing many UI-related tasks.
Concluding
Even though phones aren't as impressive as the computer that took us to the moon, it is actually insane, that we are carrying around little computers in our pockets that are way more advanced than those that took us to the moon! That's what we call technological progress... and I believe that it is fascinating ^^