The answer here by Organic Marble explains the ISP math. This is a supporting answer on why the ISPs are different. High ISP comes from high exhaust velocity, and getting that is easiest with lighter elements. Hence nuclear thermal rockets normally assume pure hydrogen and liquid fuel tables starting with Hydrogen-Oxygen, then Methane-Oxygen (methane being mostly Hydrogen with some Carbon).
So making a high performance solid rocket involves trying to stuff as many light atoms in there as possible, with the difficulty that light atoms tend to make molecules that are not solids at room temperature, meaning otherwise valid solid rocket chemistry becomes a liquid mono-propellant instead (see proposal mentioned in Ignition by Clark for an Oxygen/Methane mono propellant).
The next challenge is mechanical, the solid fuel grain needs to stay put during handling and use, so most compounds need binders of sub optimal rocket performance to keep them in place. For space assembled solid rockets this might be less relevant.
Solid rockets are a subclass of mono propellants which means the single fuel grain needs to be an optimal fuel/oxidizer mix, which may be chemically complicated. Where a liquid engine can just tweak the pump pressures to tune things you cannot remove/add fractional hydrogen atoms from your compound. Composite propellants try to solve that by cocktailing compounds, but at the cost of adding lower performance compounds to the mix.
Finally the solid rocket needs to be stable enough to not burn or explode early. Hydrogen and Methane will happily explode with Oxygen, but can be used in rockets by careful injector and ignition design. Solid fuel needs to prevent detonation with chemistry rather than engineering.
All of this constrains solid rocket performance unless someone finds some really oddball chemistry, restricting solid rockets to applications where high thrust, shelf life or simplicity matter more than raw performance numbers.