What about their design seems to make them inflate and blow off? Did
they have some sort of air scoop built in?
The final version did (history below). They were made of Tyvek and had a pocket/parachute built in. I can't find a great picture but this at least shows the parachute.

- If they didn't blow off what would the consequences be?
If they stayed on for the entire ascent, that would be OK. The first jet firing would dislodge them. The fear was that if they detached at the wrong time, they could damage the Orbiter tiles or windows.
After the STS-107 failure a huge effort went on in the Space Shuttle Program to examine all possible sources of ascent debris and reduce the risk wherever possible. Pieces of the original "butcher paper" covers had been found jammed in the front window seals. This caused a switch to the Tyvek parachute covers. That didn't go so well at first, on the first couple of flights after STS-107 some covers got caught at the edge of the thrusters and fluttered against the thermal protection system or released later than desired.
- Have they ever stayed on into orbit?
Not as far as I know, at least not after we started scanning the thermal protection system on orbit (again, this was after STS-107). But again, I don't think anyone would care if they flew to orbit, it was the critical release period during which they could theoretically damage the Orbiter that people cared about.
Here is the history of the covers from Wings in Orbit p. 174 which is also the source of the picture.
During operations, Orbiter engines needed rain protection after the
protective structure was moved away and protective ground covers were
removed. This requirement protected the three upward-facing engines
and eight of the left-side engines from rainwater accumulation on the
launch pad. The up-firing engine covers had to prevent water
accumulation that could freeze in the injector
passages during ascent.
The side-firing engine
covers prevented water
from accumulating in
the bottom of the chamber and protected the chamber pressure sensing ports.
Freezing of accumulated water during ascent could block the sensing port and
cause the engine to be declared “failed off” when first used. The original design
concept allowed for Teflon ® plugs installed in the engine throats and a combination
of Teflon ® plugs tied to a Teflon ® plate that covered the nozzle exit. This concept
added vehicle weight, required special procedures to eject the plugs in flight, and
risked accidental ejection in ascent that could damage tiles. The solution used
ordinary plastic-coated freezer paper cut to fit the exit plane of the nozzle. Tests
proved this concept could provide a reliable seal under all expected rain and wind
conditions. The covers were low cost, simple, and added no significant weight.
The thruster rain cover material was changed to Tyvek ® when NASA discovered
pieces of liberated plastic-coated paper beneath the cockpit window pressure
seals. The new Tyvek ® covers were designed to release at relatively low vehicle
velocity so that the liberated covers did not cause impact damage to windows,
tile, or any other Orbiter surface.
I don't have a great reference for the other stuff; I worked in the Space Shuttle Program Systems Engineeering and Integration Office during the 2nd return to flight era. While not directly involved with the debris issues, it was impossible to not hear about it.
There is some discussion of the Tyvek covers and problems with their releases in the book Return to Flight by James Peters. Here's one (context is the STS-114 flight):
One of the Tyvek covers on the forward reaction control system
released much later than designed...They posed a transport and impact
threat ... if the Tyvek hit a window with high velocity. They were
designed to release during ascent before the vehicle reached 170 miles
per hour...
The book is a very detailed accounting of the struggle to deal with ascent debris after the STS-107 failure. But unfortunately it does not include an index.