Ascent and descent are relatively dynamic. Large amounts of energy are being transformed and redistributed very rapidly and violently. On ascent in particular, there is the potential for the booster to disassemble itself in an uncontrolled manner, which could easily cause major damage to the crew capsule; descent and reentry is a little safer, but the forces the capsule is being subjected to are still rapidly changing and non-uniform.
Once in orbit, out of the atmosphere, and coasting in unpowered flight, the risks of depressurization are rather reduced. A puncture of the hull due to space debris is the main risk. In most cases this wouldn't lead to immediate total depressurization (particularly on a station with a lot of internal volume relative to its impact-target cross section).
Since it's impractical to remain in a bulky and uncomfortable pressure suit for days at a time, the suits are worn only for the short periods where the risk is highest.
In at least one case, on Apollo 7, when the crew was suffering head colds, they opted not to wear their helmets for the descent so they would be able to equalize their middle-ear pressure by pinching their noses and blowing; they must have judged the risk of losing cabin pressure to be very low.