The initial plan was to visit all of the outer planets:
The Planetary Grand Tour was to send several pairs of probes to fly by all the outer planets (and Pluto) along various trajectories, including Jupiter-Saturn-Pluto and Jupiter-Uranus-Neptune. Limited funding ended the Grand Tour program, but elements were incorporated into the Voyager Program, which fulfilled many of the flyby objectives of the Grand Tour except a visit to Pluto.
For Voyager 1, scientists had to make a choice: either visit Saturn's interesting moon Titan, or visit Pluto. A visit to Titan would bend Voyager's path and make a visit to Pluto impossible.
Astronomers decided that in order to optimize their science at Saturn, they’d need an orbit that brought Voyager 1 up close with Titan. But that flyby also would put Pluto out of reach after the spacecraft lifted out of our solar system’s ecliptic plane.
“It was a pretty straightforward decision for them because they thought there was going to be a third Voyager mission that could come along and go to Pluto,” Stern says.
(see also the Voyager FAQ)
Back then, they chose Titan, and then NASA cancelled the third Voyager mission.
There's an alternate point of view in this Slashdot comment:
I had the opportunity to ask Ed Stone, the JPL Director & Voyager scientist, this question. His rather glib answer was, "well, Titan was 3 hours away, and Pluto was 3 years away - and I had to make payroll."
I haven't found a reason why Pioneer 11 didn't visit Uranus or Neptune, or if those visits were even seriously considered.
Pioneer 11 was launched out of the ecliptic plane by its Saturn encounter, that made it impossible to visit Uranus or Neptune. But it's possible a different trajectory would have given the opportunity to visit Uranus or Neptune.
Budget pressures may have played a role in not scheduling more flybys. Voyager 3 was cancelled in 1975. At that time, NASA needed money to fund development of the Space Shuttle.
