In determining safe viewing distance, seems to me there are discrete threats to consider:
First is the rocket blowing up on the pad, or just above it. That would have no warning. There would be a distance where you would be out of range for the fire, shockwave and flying debris.
Next is an out of control event where the rocket goes sideways and is detonated. This would give a little warning. Videos I have seen of this show the detonation in a few seconds. So I don't imagine it could get very far.
There might be a possibility of an out of control rocket that cannot be detonated for some reason. I don't know if that is ever considered, or if they presume it can always be detonated. I don't know if that has happened in modern times.
I expect they would not allow people in the downrange direction until they are past the first stage, 300 miles or so.
I imagine there would be a large grey area where there is a possibility, but extremely small one, of a hit by flung debris. I wonder if any launch sites allow people in those areas.
Does anyone know how launch facility people calculate safe distance and what they are? Besides Florida, other sites may have different policies.