There is a detailed list here of everything that needed to be done for a scrubbed Saturn V launch, which gives an idea of the scale of the problem. Lots and lots of individual tasks to ensure the booster is still safe and stable.
- Get the crew out and safe everything
- Drain the propellants from the booster (they usually can't be left fuelled for a prolonged time)
- Reinstall the servicing structures
- Safe and if necessary replace any pyrotechnics
- Potentially remove/recharge batteries
- Temporarily remove anything that can't be safely left sitting around (the RTG, in this case)
- Inspect the booster insulation for any new damage
- Drain (or replenish?) the propellants from the spacecraft
Some of these naturally mean a substantial period of time - it would take several hours to drain and refuel, for example. Inspection can be similarly time-consuming depending on how detailed a level of inspection is needed. And then you have to resume the whole launch process again.
Multiple scrubs or prolonged holds would lead to more detailed maintenance checks or requirements to replace parts with limited lifespans (eg pyrotechnics, seals), which of course would then eat up more time.
Edit: a graphical summary of the Saturn V two-day turnaround from OrganicMarble:

You can see that (very approximately) the first 24 hours are "make the spacecraft safe and check it's still all OK to launch", and then the next 24 are putting it back together and resuming the normal launch sequence. Some tasks were substantially longer than others - draining and purging the Saturn took about 2h30, but servicing the CSM cryo tanks or the LM helium tanks took twelve or sixteen hours to set up and take down again.
Thinking about it more generally, there are engineering tradeoffs here. You could design consumables and short-lifespan components to be able to cope with a few days on the pad without needing servicing - eg bigger batteries, more capacious helium tanks, and so on. And to an extent, they did that (eg the booster propellant was designed to be continually topped up while on the pad to cope with boil-off).
But the more of that you do, the heavier and more complex it gets. Having it designed to be just right to launch when freshly prepared allows you to save weight and complexity, at the cost of having to cope with long delays if you do scrub. Swings and roundabouts.