One of the most important things for any company looking to achieve something new in the field of aerospace is to have a good budget. NASA is funded by the United States government, which has its plusses and minuses. The big plus is that its budget is rather large: Close to or over 18 billion dollars, for the last part of the 2000s. Is that good? Yes. There are a lot of things you can do with 18 billion dollars. But there's a catch. NASA competes for a share in the federal budget with many other agencies (okay, all other federal agencies), and if some of them were to need more funding (e.g. in times of war, or if there was a natural disaster requiring a lot of money), NASA's budget could be slashed dramatically.
For a private company, on the other hand, the money that goes in is dedicated completely to that company's goals. If SpaceX receives, say, 8 billion dollars (I haven't been able to find an accurate figure) in private funding, all of that 8 billion will go to its projects. This isn't the case with a government's budget: Taxpayers pay a sum of money each year, an officials divvy up the money where it is needed. You may want a certain percent of your money to go to NASA, but you have no control. If you give the same amount of money to SpaceX, you know it will go to its projects.
In short, the advantages of having a privately-funded budget are a) Relatively constant funding, and b) All money the company takes in goes to that company's projects, where the investors want it, as opposed to a government taking in money from taxes and not necessarily distributing it where taxpayers want it to be.
Note: I don't agree with Gwynne Shotwell for the reason that a private company is dependent on its investors and may do what the investors want, while a government (in theory) doesn't always have to do what the people want, but that is a personal opinion, and (I don't think) relevant to your question.
Source for NASA's budget:
http://www.gpo.gov/fdsys/pkg/BUDGET-2010-BUD/pdf/BUDGET-2010-BUD-23.pdf. Wikipedia's page on NASA's budget references this.
TildalWave recently brought up a good point regarding NASA-private company relations, and suggested I include it here, so I'll add in some more information:
A portion of SpaceX's budget comes from government contracts. This is the case with many private companies: the government needs someone to manufacture tanks, aircraft, or, in this case, rockets, and pays a private company to do so. A company such as SpaceX can earn a lot of money this way. The trouble, though, is that competing for a government contract is high-risk. Chances are, there are many other competitors with similar proposals, and the government can pick and choose.
This case was exemplified in the early conceptual stages of the space shuttle program - the pictures in the book "Space Shuttle: The History of the National Space Transportation System: The First 100 Missions" by Dennis R. Jenkins gives a great look at how dramatically the design changed from time to time, depending on the contractor and new requirements. Tens of contractors, sub-contractors, and sub-sub-contractors competed for a piece of the pie, and many failed in their efforts. The take-away message is that even if a private company competes for government contracts, it can't assume in its budget that it will get them. A private company operating entirely under its own budget doesn't have those issues.