There are two ways to increase transmission speed. Both rely on increasing the power of the radio transmission.
1. Diameter: antenna gain increases with the square of the antenna diameter.
2. Transmission power: RF power increases with the log of the amplifier input power.
When you increase the antenna diameter, you run into limitations of the launcher: the fairing must be large enough to accommodate the antenna. Alternatively, you can build a folding antenna: the Russian radio telescope satellite Spektr-R has an antenna diameter of 10 m. But this hasn't been done a lot. It makes the spacecraft a lot more complicated and adds failure points.
Increasing transmitter power also has issues. New Horizons and Voyager are powered by RTGs, which are very expensive. A 2005 report estimated the cost of a New Horizons-class RTG at $60-90 million. The main driver is the cost of producing Plutonium-238.
So mission budget plays a large role. Antenna diameter feeds into the launch weight and fairing size, RTG power goes directly to cost. New Horizons used the biggest rocket available at the time apart from the Shuttle, Voyager benefited from multiple gravity assists. Increasing launch weight would have resulted in a slower launch and longer travel times, which also add to the cost.
Let's compare our 2 data points:
Voyager:
- 470 W of power available at launch,
- launch weight 721 kg,
- 3.7 m main antenna.
New Horizons:
- 213 W of power available at launch,
- launch weight 478 kg,
- 2.1 m main antenna.
Let's say you could increase transmission speed by a factor of 50 (go from a 2 m antenna to 10 m, and double the transmitter power). From Pluto, that's still only 25 kbit/s. This would shorten the download phase from 1.5 years to 2 weeks, but doesn't fundamentally change the equation: the encounter data still starts to arrive long after the encounter.
Also your spacecraft now weighs 4 tons (WAG based on the weight of Spektr-R) and needs approximately a Saturn V to launch at sufficient speed. That money would easily pay for a year of DSN time.