tl;dr - each bay of the mast collapses as it is pulled into the cylinder.
The ISS solar array mast is a truly ingenious mechanical structure which can be retracted into a remarkably small canister. It was developed by ATK and is called a Folding Articulated Square Truss (FAST) mast.
This picture shows a FAST mast without arrays.
The mast is composed of aluminum
longerons, stainless steel wire rope diagonals, aluminum rigid battens, and flexible fiberglass battens.
Every second batten ring is
flexible. Therefore, the battens are able to buckle and two cells of
the boom can be collapsed.
The deployment is driven by the strain energy of the buckled battens and is controlled by a
mechanism in the container. This mechanism is a rotating nut
with four helical grooves. These helical grooves engage roller bearings located on
the corners of each mast section in a manner similar to that of a screw being pulled
through a rotating nut. It requires two revolutions of the nut to fully extend one mast
This image shows a cross section of the deployment mechanism. It's a bit complicated, but you can see the nut at the top (gray area), the collapsed truss sections at the bottom, and the drum/lanyard/cradle mechanism that feeds the sections into the bottom of the nut.
ADVANCED DEPLOYABLE STRUCTURAL SYSTEMS FOR SMALL SATELLITES
Static Stability of the Space Station Array Fastmast structure
Static Stability of a Three-dimensional Space Truss