Solid fuel does not burn unless it has been vaporized by heat from combustion. The boundary between solid and gaseous fuel is located at the point where the temperature equals the fuel vaporization temperature. When this boundary penetrates to the SRB wall, the fuel extinguishes.
This means the burn can heat the wall up to, but not beyond, the vaporization temperature of the fuel grain. The walls could never get red hot despite the 5000F flame nearby.
This is true only for core burner designs, which include all large SRBs
It is not true for end-burners.
The lower part of the casing is exposed to flames from the upper portion of the fuel. But end-burners are limited to smaller rockets.