This is largely an opinion-based answer, because natural language is a matter of opinion.
I capitalize Space Shuttle because as far as I know, that's the actual name of NASA's spacecraft, collectively and individually.
Is that right?
NASA's official terminology is "Space Transportation System", but if you use the term "Space Shuttle", everyone knows you mean STS. NASA does use the term "Space Shuttle", especially in public-facing material (and maybe they've entirely given up on calling it STS by now).
The spaceplane part of STS, alone, which I think is what you're referring to, is called the Orbiter (or Space Shuttle Orbiter, or STS Orbiter), but again, colloquially, if you refer to a Space Shuttle, most people will probably think of the Orbiter rather than the full stack. "Orbiter" does run into a bit of a problem: five spaceworthy shuttles were built, plus Enterprise, which wasn't equipped for spaceflight, and was used in atmospheric approach-and-landing tests. It could be argued that it's not strictly accurate to call it an "Orbiter" or a "spaceplane" or a "space shuttle", but "Space Shuttle Glider" might be unnecessary pedantry.
Is "Space Shuttle" a proper noun?
When used as a substitute for STS, yes. Improper/uncapitalized, it would be ambiguous whether you meant STS or some other reusable Earth-to-orbit system such as Buran or a hypothetical future system.
Should we capitalize Space Shuttle when referring to
each one?
all of them as a group?
the project itself?
- "A space shuttle": some surface-to-orbit reusable spacecraft and/or system.
- "A Space Shuttle": one of the spacecraft constructed and operated by the STS program; context may establish whether the orbiter or the stack is being referred to.
- "The Space Shuttles", "the Space Shuttle Orbiters": the group of orbital vehicles constructed for the STS program.
- "The Space Shuttle Program" (informal), "The Space Transportation System Program", "The STS Program" (more formal): the program as a whole.
I've even seen very informal use of the capitalized word "Shuttle" to refer to the entire program in constructions of the form "Apollo be like this but Shuttle be like that."