First of all, the ground team could have, and in fact did, do most of the orbital navigation remotely. TheThis report mentions the fact that the on board computer was secondary for Apollo 8, with primary being systems from the ground. The spacecraft did have to do a few things, including making some realtime adjustments during the landing based on the actual topography, but the course corrections and burns and such were all managed from Houston. There was a desire to have a computer powerful enough to calculate the numbers on board just is case something happened that limited communication with Earth.
Computers have always been a part of launching rockets. In many instances, these were on the ground, helping to guide the rocket along its desired path. Knowing how much to steer in what direction allows one to overcome different winds, slightly offset engines, and other small problems that can be pretty much impossible to detect from the ground.
But you are absolutely right, you don't need a particularly powerful computer to do these calculations. As evidence, submit the Apollo guidance computer, which really wasn't that powerful at all. I think the common saying is just to recognize where technology has come, and if such a low end computer could do so much in the 1960s, just imagine what we can do today.
One thing you may be interested in, and it was the source for much of this, is this article talking about the power of the Apollo Guidance Computer.