The era of the space shuttle program was one of the highlights in the history of space exploration. Why did NASA decide to retire the space shuttles, or why aren't they planning to start a new program with a more reliable and efficient design like the current Dream Chaser spacecraft? Why is it that NASA relies on commercial firms to complete their tasks?
This could be a book, but here's the tl;dr version:
1) The space shuttle system never lived up to its cost and performance goals. (See this question for background info.)
2) NASA shuttle management became complacent to the point that they ignored hazardous foam shedding from the external tank. This resulted in the loss of the orbiter Columbia and its crew.
3) The accident investigation board recommended either shutting down the shuttle program or re-certifying the system.
R9.2-1 Prior to operating the Shuttle beyond 2010, develop and conduct a vehicle recertification at the material, component, subsystem, and system levels. Recertification requirements should be included in the Service Life Extension Program.
4) The 2nd Bush administration did not want to spend the money to re-certify the system (given reason #1), so they directed NASA management to make only critical safety fixes, shut down the logistic chain, and fly the shuttle only long enough to complete the International Space Station. At the last minute, a contentious decision was made to also include a final Hubble Space Telescope servicing mission.
5) The Obama administration did not reverse these decisions.
Source; Report of Columbia Accident Investigation Board, Volume I (it is a book)
As far as the why didn't NASA develop a better replacement shuttle part of your question, billions were spent on this to no avail (see X-33/VentureStar – What really happened, which could be another book). And while it is possible that Dream Chaser may, if it ever flies, be "more reliable and efficient", it will be far less capable than the STS orbiter.
You can also see this question for some ideas about a follow-on shuttle.
The vision which initiated the shuttle program was economic, routine, schedulable access to space.
All other space launch systems involved single-use essentially hand-crafted vehicles making each launch very expensive. Aside from SpaceX, this is still largely true today. The Shuttle was intended to change all that by being highly re-usable. It was also envisioned to be capable of quick turnaround so that a small fleet of say 5 or 6 vehicles could be operated on a weekly launch schedule.
The reality turned out to be very different.
A key feature of the shuttle - the heat resistant tiles - were critical to its re-usability. Unlike ablative heat shields which work by slowly vaporizing, the shuttle's heat tiles are "durable"; they don't sacrifice material during re-entry. The problem is that they are difficult and expensive to make, and somewhat fragile; any damage can compromise their thermal protection ability. They tended to suffer some degree of damage during every launch. Among other things, inspection and repair of the heat tiles drove up the cost and cycle time to prepare each shuttle for its next launch. Damage repair aside, the tiles needed a waterproofing treatment which re-entry conditions removed, requiring careful re-application for each launch - yet another deviation from the original "quick turn-around" concept.
Another "signature" feature of the shuttle was the SRBs. They weren't part of the initial concept, but were incorporated into its design in order to make the whole thing work - to provide enough thrust during the initial boost phase while adding a minimum of weight. SRBs are typically expendable; the shuttle's are recovered and re-used, but it's a costly process.
The shuttle also has inherent safety issues. A big one is the absence of an abort mode (crew escape) between SRB ignition and separation.
The two loss-of-vehicle events in the shuttle's career underscore these issues. Challenger revealed issues with the SRBs and lack of an escape option during SRB burn. Columbia showed how vulnerable the TPS was to damage during launch.
Despite all its amazing qualities, the shuttle proved to be too expensive in relation to other launch systems and involved compromises in crew safety which don't exist in other launch systems.
For Dream Chaser to pick up where the Shuttle left off, it would have to prove to be a meaningful step in the direction of the Shuttle's original concept goals - greater/cheaper/more reliable access to space. At the very least, it would have to be immune to the problems unique to a re-useable spacecraft which affected the Shuttle. Not having cryo-propellant tankage higher in the stack than your exposed re-entry shielding helps; it avoids the risk of damage from material (ice or insulation) which could be shed during ascent, both improving safety and mitigating wear and tear. Placing it atop the man-rated (and expendable) Atlas booster may avoid some of the other safety concerns associated with the shuttle, but won't be a game changer where cost is concerned.