One of the things to think about is rockets don't have energy budgets. They have delta-v budgets. Try and think of it that way but: The majority of the energy in a 100km orbit above earth is in 'going sideways'. However besides the lack of numbers this doesn't answer your question.
For example how much dV is needed to gain attitude is at least the amount you would need to coast to your target altitude in a vacuum. But doing this as part of a way to get to orbit is only one (inefficient) way of getting there. The dV saving is at most this. There is no good bound in either direction.
A few things are worth considering to get a feel for the problem:
How long you have to spend fighting gravity also depends on how long you need to keep the rocket in the air before centripetal force and gravity balance each other and there's no universal answer to that question. Aero-properties and TWR affect what's optimal.
Rockets don't "fish-hop"; they don't go up until they reach orbital height, then sideways until they reach orbital velocity, instead they try and combine the two maneuvers which takes advantage of Pythagoras, but costs some aero-dynamic drag.
A little bit of extra or saved dV costs a lot. Anything you gain or loose is necessary the 'first' bit; you have to move the whole fully fueled rocket by this amount.
Let's do the simplified theory first. If dV was the only concern and there was no atmosphere. With infinite TWR, you could reach a "sea-level" orbit without any energy or dV being expended upwards. Further the theoretically most efficient way to use dV to gain altitude is a Hohmann Transfer, the Hohmann transfer from sea level to 100km around earth is small. Concretely around 60m/s, which compared to 7800m/s for the orbital velocity in the first place is tiny.
So yes, there are many meaningful ways in which your intuition and the saying about staying there are correct. If you simplify things down too much, then especially when you consider that going there then going fast enough to stay there is not a very efficient way of doing things: then 100% lifting rockets up before you launch them is silly, it gains very little for a lot of effort.
However, these are way to big assumptions to ignore. When you add these factors back in things start to make more sense. Drag and gravity losses are a significant factor in current rockets and they are designed to compromise these things. If you didn't need to worry about TWR and being aerodynamic rockets could be made much lighter, which is the holy grail as this means less fuel is need, which makes them lighter. Etc, etc.
I know this is anti-climactic but its really hard to put a number on how much mass/money this would save as it changes a lot of factors, but its hard to imagine this not being at least a 20-30% mass saving.
Another thing is wings etc don't just get you altitude, they might also get you speed. By not needing to use valuable dV just to keep you up, you can fly through the air for long enough to make use of atmospheric engines. Jet engines are a lot more fuel efficient than rocket engines at low speeds, so if you can it would make sense in theory to want to use them when you can.
If you usefully use this extra speed and knock mach 1 off your dV requirements the savings could be huge.