The proposed orion propulsion system used nuclear explosions to push against a giant shock absorber and propell it, but from my understanding, nuclear explosions only produce heat and rely on the surrounding atmosphere to produce the destructive pressure. My question is, would such a propulsion system need a reaction mass, say hydrogen, to provide the actual explosion, and if it did, would it still be worth it or would a thermal engine be more efficient, or, is there something else providing pressure, such as radiation?
The reaction mass is included in the bomb a.k.a. "pulse unit"
The original project designed bombs with a reaction mass made of tungsten. The bomb's geometry and materials focused the X-rays and plasma from the core of nuclear explosive to hit the reaction mass. In effect each bomb would be a nuclear shaped charge.
A preliminary design for a nuclear pulse unit was produced. It proposed the use of a shaped-charge fusion-boosted fission explosive. The explosive was wrapped in a beryllium oxide channel filler, which was surrounded by a uranium radiation mirror. The mirror and channel filler were open ended, and in this open end a flat plate of tungsten propellant was placed. The whole unit was built into a can with a diameter no larger than 6 inches (150 mm) and weighed just over 300 pounds (140 kg) so it could be handled by machinery scaled-up from a soft-drink vending machine; Coca-Cola was consulted on the design.
Source: Wikipedia article on Orion
The material of the bomb is the reaction mass.
When a nuclear explosive is detonated in vacuum, the material of the bomb, and anything immediately adjacent to it, gets vaporized and accelerated at high velocity in every direction away from the center of the explosion.
Organic Marble's answer describes how this can be optimized to make most of the vaporized material, which is in the form of plasma, go in one direction to hit an Orion pusher plate.
If one adds additional inert mass to the explosive, one gets a lower specific impulse but also gets more impulse for the same amount of (very expensive) nuclear explosive. This actually is true of rocket propulsion in general.
Let us imagine that small atomic bombs are periodically exploded behind the apparatus, at a considerable distance from it. Then the blast wave, striking the bottom platform of the vehicle equipped with a shock-absorbing device, will impart acceleration to it. The fundamental possibility of such a movement has already been tested on a model, behind which TNT was exploded.
Y. Novokshonov "Rockets in Space" 1967.
Представим себе, что позади аппарата, на значительном от него расстоянии, периодически взрываются небольшие атомные бомбы. Тогда взрывная волна, ударяя в донную платформу аппарата, снабженную амортизационным устройством, будет сообщать ему ускорение. Принципиальная возможность подобного движения уже проверена на модели, позади которой взрывали тринитротолуол.
Ю.Новокшонов Ракеты в космосе.1967.