Wikipedia has a good summary of this.
STS-1 to STS-9 were numbered sequentially by order of launch, then it switched to a number based on fiscal year (5 for FY 1985), launch site code (1 for KSC, 2 for Vandenberg), and letter. The letters were assigned sequentially at the time of planning, but schedule slips etc meant that they were not launched in this order.
After Challenger they dropped this system, and returned to a numeric STS-xx name. As with the letter codes, these were assigned sequentially but got out of sequence as time went on and schedules were moved around - the first flights were 26, 27, 29, 30, 28...
(The STS-xx numeric codes were still used internally during the "letter sequence", referring to the specific payload/mission rather than the launch, but these wouldn't usually be public. This meant that there were some duplicates on internal paperwork, eg two STS-26s, and so the new ones were sometimes listed as STS-26R.)