The Ku-band system consisted of a directional antenna mounted in the orbiter payload bay and provided high BW communication with a ground station. This page: KU-BAND SYSTEM describes the system as having uplink at 13.755 GHz and downlink at 15.003 GHz. The various modes and data rates are described as well, but there is no mention of the modulation / coding that was used. There seem to be a few IEEE articles that may explain the system in detail, however they require purchase. My guess is it should have been PM, but if someone could confirm it would be great.
I am not very comm literate but hopefully I can parrot this information about TDRSS Ku-band modulation I found and point you to the source so that you can confirm whether or not it actually answers your question.
This document states
The K-band SA forward link is the only service for command data greater than 300 kbs and its carrier is BPSK modulated without range channel transmission.
Where SA stands for Single Access and BPSK stands for BiPhase Shift Key. (Italics mine).
For Multiple Access (MA) service this table is provided: