Good SE questions deserve good answers. I'm not a PV expert but I can at least get the ball rolling.
At some point far from the Sun does the output of solar panels start dropping much faster than inverse square?
Yes!
If so, how fast?
tl;dr: efficiency at Saturn's 10 AU might be one third that at 1 AU, and about a quarter at Neptune.
I recall reading somewhere that solar power - already marginal at Jupiter - will cease to produce any power whatsoever at distances much further out than Saturn.
This, not unlike Mark Twain's death while he was still alive, seems to be greatly exaggerated. :-)
Single crystal silicon as an example
It depends on the semiconductor material in question (silicon, III-V (GaAs InP), perovskites(?), exotics etc.) material quality, defect density/radiation damage, etc. but basically at a low rate of carrier (electron-hole pairs) production there are competing processes that allow them to recombine locally inside the material such that they can't make it out and be used to produce a current.
The simplest way to characterize these processes is with a single parameter called shunt resistance that tries to lump all of these loss mechanisms into a number.
Mathematical models used for terrestrial PV applications are going to focus on sunny to cloudy/overcast days:
...and may not be very accurate at Saturn (10 AU) and beyond, and experimental data for cells optimized for terrestrial applications may be worse at Saturn than those optimizes for very low light levels, but with those caveats, let's take a look at some data anyway.
From Crystalline silicon cell performance at low light intensities September 2009 Solar Energy Materials and Solar Cells 93(9):1471-1481 DOI:10.1016/j.solmat.2009.03.018 Figure 2b shows a measured efficiency curve of some single crystal silicon PV cells. Eyeballing the measured efficiency curve I get about 7% at 1 AU (1361 W/m^2) 4% at Saturn's 10 AU (13.61 W/m^2) and 2% at Neptune's 30 AU (1.5 W/m^2):
Fig. 2 (a) Apparent shunt and series resistances ( R Se , 1 D and R Sh , 1 D ) as obtained from irradiance intensity measurement, (b) open-circuit voltage, fill factor ( FF ) and efficiency at maximum power point voltage measured between 0.01 and 1000 W/m 2 irradiance intensity for the mc-Si cell.
See also Evaluating Crystalline Silicon Solar Cells at Low Light Intensities Using Intensity-Dependent Analysis of I–V Parameters (paywalled)
Need more? Concentrate!
If you're going to be wandering out at Saturn and beyond and need to get back to PV efficiencies similar to those at 1 AU, then concentrate! By thinking harder, you'll realize you can concentrate the light as well.
While you're out there at Saturn hopefully you've either brought Wikipedia along or can still link to it and read its Concentrator photovoltaics article.
You can build one giant concentrator; say a 100 square meter aluminized polymer film reflector (like a solar sail-looking thing) for every 10 square meters of PV area, or use a high granularity system with small lenses or reflectors on thousands of small cells.
Further reading:
and related:
Yes colder is better so there's a "hot" penalty circa 1 AU
Answers to Temperature of photovoltaic cells vs efficiency - is it ever actively controlled? explain that the recombination losses are increased at elevated temperatures. Closer to Earth at 1 AU this will actually show a drop in efficiency going towards the Sun if the panels are not provided a good way to cool (they're "in the sun" so will get toasty). But out near Saturn at 10 AU they're going to be quite cold, this will not be a significant factor.