This is a very difficult question to answer - as the answer depends much more on the missions and other design parameters than the actual design.
First I wish to address the heat problem:
Actually solar cells create a lot of heat. For this reason for observation satellites typically the solar cells are far away from the camera. But so do nuclear power cells. Also for this question I consider only power cells which use natural nuclear decay (or Radioisotope Thermoelectric Generator, RTG), so not a nuclear fission reactor.
The nice thing about space is that any power that isn't used, becomes heat in some way. There is no losses through drag or anything. So this all has to be dissipated as heat.
Now waste heat comes from several different efficiencies:
- Using the equipment (create torque/photographing/data transmission ..) - $\eta_{equip}$
- The electical equipment (batteries, wiring) $\eta_{e}$
- Efficiency of the power source $\eta_p$
Now if we want to provide the same net power $W_{net}$ we can just compare the efficiencies.
Equipment efficiency
Barring slight variations in geometrical constraints the efficiency for both nuclear and solar energy sourced satellites is near equal - so it can be ignored.
Electrical equipment efficiency
Where nuclear power will always work in a steady way. (Or at least almost, considering the half-life time of the plutonium/uranium is quite long). Solar energy is really not that easily generated, the solar radiation reduces with the inverse squared distance from the sun. (For example the Rosetta probe was put into hibernation for 31 months)
And for celestial orbiting satellites, the time when it is in the shadow of the planet has to be taken into account. - On top of that solar panels are always quite large, requiring a lot of wiring to connect each separate cell.
$\eta_{e,s} < \eta_{e,n}$
Though given modern day advancements in electrical power storage, this is quite small $\eta_{e} \approx 0.93-0.96$ - So only a percentage or so could be gained from using a nuclear power cell, but this depends entirely on the mission profile.
Power efficiency
This is probably the biggest loss in spacecraft and will be the deciding factor:
In RTG engines typically the natural decay of the atoms heats another solid substance. This matter than uses electrostatic conversion to create power. Looking at the efficiency of the Curiosity rover $\eta_p = 125/2000 = 0.125$
There is however a ready to implement proposal to use a Stirling engine instead of electrostatic conversion. The heated plates would then heat a gas which then goes through the Stirling engine (called ASRG). The NASA predicted efficiency goes up to about 0.25.
Solar panels also lose a lot of power. This has mostly to do with the band-gap of the semi-conductor used. (Germanium, silicon). For simplicity consider that no radiation is reflected.
Basically when a photon hits a solar cell 3 things may happen:
The photon has less energy than required for the electrons to jump the gap -> all energy of the photon is lost and converted to heat.
The photon has exactly the right amount of energy -> the whole energy of the photon is converted into electrical power
The photon has more energy than required to jump the gap -> the part of energy that is required to jump is used, the rest is converted to heat.
As you can see the efficiency is hence a function of the typical wavelength profile you expect - and also material used (and nowadays multijunction solar cells with different bandgaps are gaining popularity). Typical efficiency values of solar arrays lie in the order of 0.25 (multijunciton cells are not yet used in real application for space a lot).
Conclusion
So concluding, which is more efficient (and hence creates less waste heat):
For RTGs we get around:
$$\eta_{t,n} = 0.98 \cdot 0.10 = 0.098$$
For solar we get around:
$$\eta_{t,s} = 0.94 \cdot 0.25 = 0.235$$
So yes, solar cells generate a lot less heat. And for heat management alone nuclear power cells are a bad idea. However Stirling engines could tip the balance. (Though similarly multijunction could also increase solar cells).
Now as for the total mass - this is an even broader topic I will not touch here, one has to take into account that you also need a minimal power to operate, and having more solar panels also requires more mass / increasingly complex designs.
