The [initial plan was][1] to visit all of the outer planets: 

> The Planetary Grand Tour was to send several pairs of probes to fly by all the outer planets (and Pluto) along various trajectories, including Jupiter-Saturn-Pluto and Jupiter-Uranus-Neptune. Limited funding ended the Grand Tour program, but elements were incorporated into the Voyager Program, which fulfilled many of the flyby objectives of the Grand Tour except a visit to Pluto.

For Voyager 1, scientists had to [make a choice][2]: either visit Saturn's interesting moon Titan, or visit Pluto. A visit to Titan would bend Voyager's path and make a visit to Pluto impossible.  

> Astronomers decided that in order to optimize their science at Saturn, they’d need an orbit that brought Voyager 1 up close with Titan. But that flyby also would put Pluto out of reach after the spacecraft lifted out of our solar system’s ecliptic plane.

> “It was a pretty straightforward decision for them because they thought there was going to be a third Voyager mission that could come along and go to Pluto,” Stern says.

(see also the [Voyager FAQ][3])

Back then, they chose Titan, and then NASA cancelled the third Voyager mission.

There's an alternate point of view in [this Slashdot comment][4]: 

> I had the opportunity to ask Ed Stone, the JPL Director & Voyager scientist, this question. His rather glib answer was, "well, Titan was 3 hours away, and Pluto was 3 years away - and I had to make payroll."

I haven't found a reason why Pioneer 11 didn't visit Uranus or Neptune, or if those visits were even seriously considered. 
Pioneer 11 was launched [out of the ecliptic plane][5] by its Saturn encounter, that made it impossible to visit Uranus or Neptune. But it's possible a different trajectory would have given the opportunity to visit Uranus or Neptune.  
Budget pressures may have played a role in not scheduling more flybys. [Voyager 3 was cancelled in 1975][6]. At that time, NASA needed money to fund development of the Space Shuttle.  


  [1]: https://en.wikipedia.org/wiki/Voyager_program
  [2]: http://astronomy.com/year-of-pluto/2015/06/why-didnt-voyager-visit-pluto
  [3]: http://voyager.jpl.nasa.gov/faq.html
  [4]: http://science.slashdot.org/comments.pl?sid=7613123&cid=50003607
  [5]: http://space.stackexchange.com/questions/3609/is-there-any-way-to-communicate-with-pioneer-11-through-voyager-1-or-2
  [6]: https://books.google.nl/books?id=RoJMadct4TQC&pg=PA289&lpg=PA289&dq=%22voyager%203%22%20nasa&source=bl&ots=ZaRNE_-Y4B&sig=1NynnV7_ccwaK6REpb3k1mqpqwo&hl=en&sa=X&redir_esc=y#v=onepage&q=%22voyager%203%22%20nasa&f=false