Your question indicates to me that although you aren't familiar with celestial mechanics, you do have some knowledge of physics and astronomy. You're a space enthusiast? Good for you!!
The situation you describe is very much aligned with what we call Keplerian orbits, named for their pioneer, Johannes Kepler, publishing in the early 1600's.
In the absence of perturbations by other gravitating bodies, which I assume comes under your prohibition on "obscure funky disturbances", Keplerian orbits are indeed stable.
Where did your reasoning lead you astray? Assume the orbiting object is in a circular orbit, as shown by the dark line in the figure, orbiting in the direction shown. At the large red arrow you gave the orbiting object an instantaneous gentle nudge inward, indeed not changing the tangential velocity at that point.
But you did change, even if only a little, the flight path angle, to that indicated by the small red arrow on the red orbit. The object is no longer traveling horizontally, it's headed a bit downhill. When the object was traveling strictly horizontally (circular orbit) the local gravity vector was perpendicular to the velocity vector. When those vectors are perpendicular there is no change in the object's speed. (Speed is a scalar quantity, the magnitude of velocity, which is a vector quantity with both speed and direction) In the downhill case, post-nudge, just as in the case of something rolling down a hill on Earth, there's a (small) component of the gravity vector parallel to the velocity vector. The object accelerates, i.e. its speed increases with time, so its speed doesn't remain constant after the perturbation. That change in speed corresponds to the change in gravitational potential energy due to the changing radius from the primary's center: the farther downward it goes, the faster it goes.
This increase in speed with decreasing altitude causes the orbit's radius of curvature to be larger than that of a circular orbit at that altitude, so the orbit eventually bottoms out at the periapsis, 90° away (as measured from the center of the primary) from the perturbation point. It then rises back to the original altitude, 180° away from the perturbation, with the same tangential speed and the same vertical speed, just upward instead of downward, as you see at the bottom of the diagram.
That vertical speed carries the object higher, and that decelerates it. The reduced speed decreases the orbit's radius of curvature, so it peaks out at apoapsis 270° from the perturbation and begins descending. At 360° from the perturbation — one orbit — it is back to exactly where it started, at the instant of the perturbation, with the same velocity, same flight path angle, same everything, and this repeats ad infinitum.
This orbit, like all bound (i.e., not escaping) Keplerian orbits, is perfectly stable. Given the constraints you listed, it would remain exactly as shown forever, without any kind of control.
If you made the perturbation not small, say a significant fraction of the orbit speed, then you could make the object collide with the primary. "If you push something hard enough, it will fall over."
Once you start complicating the picture — the planet isn't spherically symmetrical, the planet rotates, there are other gravitating bodies involved, part of the orbit is in full sunlight and part is eclipsed, etc. — then those perturbations make the orbit evolve (change with time), in a few cases to the point of colliding with the primary or even being ejected from the system. Orbit evolution happens to everything orbiting Earth, even the moon.
