Fins are not really bad for radiative heat transfer. They just face an inevitable point of diminishing returns. Those diminishing returns place a limit on how many fins should practically be included. The optimum is greater than zero but less than a dense packing of fins.
Draw a boundary around any isothermal object radiating heat energy into space. The amount if radiation passing through the boundary is always less than or equal to what the boundary would emit if it were a black body having the temperature of the source inside. In particular, if we draw a tight-fitting convex hull of the emitter as our boundary, then the radiant heat transfer is limited to the black-box radiation that could emerge from the area of the hull. We can approach this convex-hull limit in principle in either of two ways:
1) Use a nearly black-box material. In principle, if the material is perfectly black the radiant heat transfer through the convex hull will reach its limiting value.
2) Put in a lot of concavities and increase the surface area. If we could increase the surface area of the emitting body under the convex hull to infinity, we could get black-body radiation through the hull.
The convex hull depends on geometry, so there is an extra way to increase the limit:
3) Within available space, setting up your geometry to allow a convex hull with a larger surface area enables more heat transfer. As we shall see, this plays a major role in the design of finned radiators. Even if you have a perfectly black material and the radiation then no longer depends on the detailed structure of the emitter, changing the convex hull to an increased surface area opens the way for more heat transfer.
Black-body radiators designed to calibrate pyrometters take the second concept nearly to that extreme. The emitter is an isothermal cavity with a small hole from which the radiation emerges. The area of the emitting surface is so much larger than that of the hole (which is serving as the boundary) that even with common industrial materials the hole comes close to achieving bkack-body heat transfer. The pyrometer is then placed so that (within some tolerance) it receives only this nearly black-body radiation from the hole, providing a standard for the calibration.
In the case of finned RTGs we now have materials available that come close to black surfaces (#1 above), so adding more surface area to the emitting surfaces (#2) has little effect. But we do gain an effect from the area of the convex hull itself (#3). Suppose your RTG has a radius of 0.25 m and a height of 2 m, and you place four 0.5-m long fins around it. The convex hull is a square prism, whose surface area is 10.74 square meters. Now try six fins: your convex hull has changed to a hexagonal prism, and its surface area is increased to 11.92 square meters. You've added 10% more heat transfer just from the convex-hull area alone. You eagerly add more fins, only to discover that you don't get as much bang for the buck (or for the kilograms of additional mass): the convex hull never gets bigger than a cylinder whose area is 12.96 square meters.
We got as much additional area for heat transfer with just two additional fins from four to six as we can ever hope to get with infinitely many more than six. Given this sharp dropoff in return, in real life we should expect an optimal design for heat transfer versus weight, ease of fabrication and testing, etc to involve several but not many fins.