Space Shuttle was designed to be partially reusable but was later shut down because it turned out to be very expensive to run compared to disposable spacecraft.
Now Falcon 9 is suddenly reusable to a large extent and presumably not that expensive.
How does Falcon 9 do it after Space Shuttle failed to do the same?
There are many contributing factors to this problem.
The Shuttle was designed in the 1970's and technology has matured since then. Additionally, the issues the Shuttle ran into (While in truth may have been predictable at the time) are now more obvious and a new design can try to avoid them.
Consider the simple case of the heat shield on the Shuttle. It was composed of 10's of thousands of small tiles. It was not clear they could make large tiles, of the right shape, that could handle the bending loads and whatnot. But that added a monstrous manpower cost on refurbishing the orbiter after every launch. You saw that NASA tried replacing some of the tiles where they could with larger blankets later in the program.
SpaceX went with a known workable system (PICA) refreshed for the new century, and used fairly large pieces. (Also a capsule is much smaller to coat than something the size of the Orbiter)
The engines on the Shuttle were top of the line, possibly highest performing engines ever built and used in production. But along with that comes maintenance costs which again were being mitigated somewhat at the end of the program with various upgrades to the SSMEs.
SpaceX went with the most reliable design they could find in the Merlin family, a pintle injector and spent a lot of time and effort on making the engine affordable and performant. They also iterated faster. While it is true elements of the SSME were upgraded over the life of the program, the Merlin family progressed in a much shorter time from the Merlin 1A, 1B, 1C, and 1D with a further uprating of the 1D pending soon.
The SSME are in the 660Klb thrust range, whereas the Merlin family started in the 70Klb range and has upgraded to the 195Klb (and on its way higher again with the Block 5 Falcon 9), which is a much simpler problem, and easier to design, and easier to reuse in theory.
Consider size. The STS system is huge, with 7 million lbs of thrust on liftoff vs a Falcon 9 with only 1.3 million lbs of thrust. Obviously mass of the system is vaguely similar, since you need a T/W of greater than 1 to liftoff. (Else you sit on the pad burning propellant).
SpaceX's system is much smaller, and thus much much easier to recover. Consider the problem of recovering a capsule (Dragon) vs the Orbiter. Then consider reuse. (Which to be fair, SpaceX has yet to reuse a Dragon capsule, regardless of plans to do so, Edit: 2 years later, 2 Dragons have reflown and the remaining 10 CRS missions are expected to use reused Dragons).
From a design decision the choice of wings vs vertical landing is a huge differentiator. Only time will tell if SpaceX made the right choice. NASA clearly did not make the right choice with their overall approach. (SNC would argue that wings can be done properly, time would tell, but not clear they will ever launch). (Edit: 2 years later they have landed 23 first stages without wings pretty much without fail. So probably a good design choice.)
The Shuttle needed a standing army of something like 24,000 people to maintain the system. SpaceX entire employment is still under 5000. Consider the cost of salaries for 24,000 people vs 5,000 and divide that over the number of launches a year. This is a large contributor to system costs.
For political reasons the Shuttle was built across many states, much like the European Ariane 5 is built. This way different senators and congress-critters would be willing to vote for it, as it provided copious numbers of jobs in their districts. This is a recipe for inefficiency. As has been shown by those two programs.
SpaceX builds almost everything in one site (Hawthorne, CA), tests in a second (McGregor, TX), and launches from a third (LC-40). (Soon to be fourth (LC-39A) and fifth (Brownsville, TX). For Polar orbits they launch from Vandenberg Air Force Base in California (LC-4). When they discuss their next generation launcher, they suggest they plan to build and test it next to the launch site. Mostly because it will be too big to transport.
I could probably go on for longer. Suffice to say this is comparing peanuts to watermelons.
To avoid the hazard of going off-topic, I will start by re-framing the question into 2 different parts. This is done so that all the evidence directly builds to an answer to the question. In this case, the question is:
Firstly, we need to define "success". For this definition, I will use the same reference which is used to make the overall argument. My own reference also uses a reference to make its argument, which is much more esoteric, so here they both are:
You can see in their tables that different scenarios were analyzed and compared to the alternative option of the Titan III. For our purposes, Titan III is the alternative expendable launch system. Thus, I can reasonably claim that success was evaluated (and should be evaluated) on the basis of the cost of the reusable launch system versus the alternative expendable system. The degree of success is defined as the rate of return using standard financial analysis. A rate of return of 0% or less is a "no go", but the competitive rate of return is a matter for debate.
I will volunteer the fact that, at that year, the design of the space shuttle wasn't finalized anyway. So this doesn't necessarily apply to the final version. However, the underlying claim is so strong that I think you'll accept that it probably won't matter.
Somewhat trivially, the economics of a reusable system depends on the number of launches. Firstly you have a capital investment of R&D and hardware, then that cost is later recouped from reduced needs of manufacturing. The studies we have from the design of the Space Shuttle reflect exactly this. They give the competitiveness of the launch system as a function of launch rate.
The actual launch rate of the space shuttle was 4.5 launches per year, as historical fact. The above study found that the system would break even (defined as 0% return rate, but still recouping the investment) if they launched a few less than 28 times per year. There were more studies than just this, but all had the same general thinking. The flight rates studied tended to start at 40-or-so per year.
We come to a simple conclusion - anyone rationally reading the literature at the time would have concluded that the Space Shuttle would be horrendously uneconomic at the flight rates for which it was used at. The fact that it was still developed probably reflects 1) over-optimism by the designers of Congress' demand for spaceflight and also 2) over-optimism by the policymakers of the technology's performance and flexibility. In the end, the USA used a solution to solve a problem that it wasn't exactly designed for. That's mostly because the intended use evolved over time, and the design was too inflexible to adapt (partially because of enormous sunk costs).
Why is the F9R different?
Let us apply the same logical approach here. What is the expendable alternative to the F9R (reusable Falcon 9)? That is simply the F9. Similar to how a "broken" escalator becomes stairs, a failed reusable launch system simply becomes an expendable launch system. This happened in January 2015 when the first stage recovery effort failed after it boosted a successful launch. The shuttle program, on the other hand, was worlds apart from its alternative expendable system.
Actually, there are lots of reasons to think the F9R will fare much better:
NASA was a massive space program which developed a reusable launch system, and then shrunk into a smaller space program. SpaceX is a tiny launch service company that is hoping that reusability will grow it into a giant space company.
Most of these arguments are robust, but the demand-side factors may still plague SpaceX in the same way they did for the Shuttle. The break-even launch rate for the F9R over the F9 is lower than what it was for the Shuttle, but it might still be greater than their current rate. In that sense their success is not guaranteed, but anyone with demand-side optimism would reasonably predict success. In the worst case scenario, this means their investment was wasted. The worst case scenarios for the Shuttle were much more dire.
Reusing the first stage or boosters alone is much easier than to reuse a spaceship going to orbit and back. The requirements to the heatshield of a reused first stage are much lower than to a heatshield necessary for return from orbit.
