tl;dr The Range Safety Officer watches the vehicle trajectory to ensure that it stays within pre-determined trajectory limit lines. If the vehicle strays outside of those lines, the Officer sends the destruct signal (or signals the crew to tell them to shut off the engines, in the case of the Shuttle second stage). The inner lines are chosen so that if the vehicle is destroyed when they are crossed, the debris won't fall outside of the impact limit lines. Obviously these lines are different for every vehicle and mission.
In the case of the shuttle, they could be directed to destroy the vehicle by code word transmission from Mission Control (see Wayne Hale's blog post below).
Here are the criteria for destruct at the Eastern Test Range (Florida)
1.4.1.3 Control of Errant Vehicle Flight
a . Range Safety shall verify that all launch vehicles launched from or onto the
Ranges have a positive, range-approved method of controlling
errant vehicle flight to meet the objective of minimizing
risks to the general public and foreign countries. NOTE: Normally,
control systems on launch vehicles using the Ranges shall
consist of an airborne Range Safety System that shall meet
all the requirements of Chapters 2, 4, and 7 of this document. A thrust termination system may be considered as an
alternative to a Range Safety System; however, quantification
of risks must be determined, and the requirements in
Chapter 2 shall be met. The alternative thrust termination
concept and design shall be approved by the Range Commander.
b. Range Safety shall establish flight termination criteria and
Range Safety mission flight rules to ensure that operations do
not exceed acceptable public safety limits.
c . Range Safety
shall establish and control Hazardous Launch Areas and
procedures to protect the public on land, on the sea, and in
the air for each launch and launch vehicle using the Ranges
and to ensure the following criteria are met:
- No intact launch vehicle, scheduled debris, or payload, or launch vehicle and payload subsystems shall be allowed to
intentionally impact on land except in the launch area inside the
impact limit lines.
2 . Flight paths and trajectories shall be
designed so that normal impact dispersion areas do not encompass land.
3 . Safety margins shall be used to avoid overly
restrictive flight termination (destruct) limits.
d . Range Safety
may allow errant launch vehicles to fly to obtain maximum
data until they would present an unacceptable risk to the public or
until Range Safety can no longer control the launch
vehicle
The Impact Limit Lines are defined as
Impact limit line
- a Hazardous Launch Area; the
boundary within which trajectory constraints and
FTSs are used to contain an errant launch vehicle
and vehicle debris. Mission-essential and Wing-
essential personnel are permitted within the ILLs;
with Wing Commander approval, non-essential
personnel may be permitted within this area. How-
ever, the collective risk will not exceed acceptable
standards for non-essential personnel.
From here
A little more on the lines from here
A really good writeup on Shuttle range safety and the interaction with Mission Control from former Flight Director and Shuttle Program Manager Wayne Hale's blog.
By long standing jointly signed Flight Rules, if the shuttle were to
veer off course, spin out of control, or break up, my responsibility
as Shuttle Ascent Flight Director was to transmit those Code Words on
my loop. On hearing those words, the FCO would depress the two
buttons in front of him to – as we say – ‘terminate the flight’. That
means exactly what you think it means. I don’t have to spell it out.
This blog post also explains the role of the shuttle pilot and commander in a 2nd stage range safety scenario, as I mentioned above.
You see, the shuttle Commander and Pilot are designated Agents or
Deputies of the Range. The destruct package is built into the Solid
Rocket Boosters and those are jettisoned two minutes into an eight and
a half minute powered flight. After that, should the shuttle go off
course toward a populated area, the FCO can do nothing about it. The
responsibility which the President of the United States has given to
the FCO cannot be accomplished – except to call the crew and tell them
to do what is necessary.
So we practice these scenarios – far fetched as they may be – to
ensure that the crew knows what to do. Steer out to sea; shut down
the main engines, protect the population along the eastern seaboard.
One small problem – that procedure puts the shuttle crew into what is
delicately labeled a “black zone”. If the shuttle is high enough – as
it is for much of the boost phase – but with forward velocity
significantly below orbital speed – then an unpowered entry will
result in the g-loads and heating which builds up too fast, faster
than the wings can generate lift. And the result? Well.
So the Commander and the Pilot are designated Deputies of the Range.
If the really bad thing happens, they are sworn to protect the
population of the east coast, even at the expense of their crews’
lives.
It takes courage to fly in space.