Yet another question where the answer becomes obvious if you play Kerbal Space Program.
What is going on here is gravity loss. As you climb into the sky on your tail of fire some of your energy is going to simply holding your rocket in the sky. While there is no way to get around this on a body with an atmosphere it's energy expended for no progress, you want to minimize it.
Now, consider what orbit is: For a circular orbit you are moving fast enough that your path climbs as fast as gravity pulls you down. (If your orbit is elliptical you keep trading back and forth between speed and height but it averages out the same.) This does not magically switch on as you reach orbital velocity, any horizontal velocity produces some effect.
Thus, for the purposes of minimizing gravity loss you want to get as close to horizontal as you can (consistent with not falling down) as soon as you can. Doing this too soon on a body with an atmosphere will cause too much loss due to atmospheric drag, though. In practice you pick a path that minimizes the sum of gravity loss and drag loss. This ends up being a path that goes almost horizontal while still in the outer edges of the atmosphere.
(The reason I say it can be avoided on airless bodies is that if you build a big enough base you can launch your spacecraft with a maglev train. It's riding on rails as it builds velocity, thus you are not expending any energy countering gravity. With a properly designed track you can boost to your desired velocity without even lighting your rocket.)
I do not know how big an effect it has on Earth. I have played with the numbers in Kerbal Space Program, though, and I've seen going up almost straight costing about 10% more velocity (and remember that the rocket equation is exponential--that's a lot more than 10% more fuel.) I suspect it's worse in the real world as KSP rocket engines are cheap and thus KSP rockets tend to be a lot more powerful than real world engines and thus suffer less gravity loss. In KSP I find a thrust to weight ratio of about 2 (on the pad) to be ideal, using a less powerful engine makes for a more expensive rocket. (Going higher gets one going too fast while the atmosphere is still too thick which causes problems with trying to steer. Even at 2 I lose a bit more due to gravity because it can't tip over fast enough without having the nose pointed dangerously far from the direction of travel.) In the real world the engines are a much bigger portion of the total cost, the Space Shuttle has a thrust to weight ratio of 1.5 on the pad.