When the space shuttle goes out of the Earth's atmosphere, how does it then initiate going into the orbit?
You're thinking of the climb out of the atmosphere and the entry into orbit as two separate operations, but it's really one continuous thing.
It takes several minutes for even powerful rocket engines to accelerate to orbital speeds. Most of the horizontal acceleration has to happen at high altitude, above the densest part of the atmosphere, otherwise too much power would be lost to air resistance. So the strategy is to start going straight up, then gradually tip over toward the horizontal, with the rocket engines firing the whole time. As more of the acceleration goes toward the horizontal, the vertical rate of ascent slows, and ideally you hit the target altitude just as your horizontal speed reaches that needed to maintain circular orbit, and then turn off the engines.
The space shuttle did it a little bit differently; the main engines would shut down while it was on an elliptical path that would take it to its desired altitude, but not going quite fast enough for circular orbit. Once at the desired altitude, the secondary Orbital Maneuvering System engines would fire, providing the last increment of speed to circularize the orbit.
Here's a plot of the the space shuttle's altitude over time, for the main part of the ascent, from a simulation of the STS-129 mission from FlightClub.io:
The lavender arc represents the trajectory of the solid rocket boosters after they burn out and separate; the pink track is the orbiter. You can see that the vertical speed is increasing continuously until the SRBs burn out. The main engines produce less thrust than the boosters, and the orbiter is pitched over, only about 36 degrees up from horizontal, at that point, so the vertical climb begins to slow as horizontal speed builds up.
The Kármán line of 100km altitude is generally accepted as the border between atmosphere and space, reached at about 4 minutes into the ascent, but as Organic Marble points out, there is no actual dividing line; the atmosphere just gets thinner and thinner as altitude increases.
Here's the pitch angle over the same time period:
For the first fifteen seconds of flight, the shuttle goes straight up (90º pitch), then quickly pitches over; by 85 seconds into the flight it's at a 45º angle, gaining both horizontal and vertical speed rapidly. It continues slowly lowering the angle until main engine cutoff at about 510 seconds. At this point, it's coasting to a higher altitude, and the Earth's surface is curving away below it, so the plot shows the effective pitch angle is increasing as it goes. (At some point, the real orbiter would have to return to a horizontally pitched attitude to do the circularization burn, but I suspect this simulation doesn't cover that.)
Here's the total velocity of the orbiter over time, but it doesn't tell you what direction the velocity is. (The pitch angle is pretty close to the velocity direction, though.)
As the engines and solid rocket boosters burn fuel, the mass of the shuttle decreases rapidly, while thrust remains generally level, so the velocity curve increases as it goes for the first two minutes before the SRBs burn out. Velocity then increases more slowly, with the acceleration rate once again increasing as fuel is burned off.
Reentry is pretty nearly the reverse. The OMS engines would fire to slow the orbit slightly, just enough to bring the low point of the orbit down into the atmosphere. This requires a velocity change ("delta-v") of only about 90 m/s down from the 7700 m/s orbital velocity. Once the orbiter started to meet the thin air of the upper atmosphere, it would slow down further, and reentry would be assured.
