For a sense of what would happen just as a mental picture, what you do to the ball wouldn't translate clearly into a change in its orbit until you had both moved along on your paths through a fairly large angle. At the altitude of the ISS, of about 400 km, it takes about 90 minutes to complete an orbit. A whack on a tennis ball with a racket could impart a speed of 120 km/h in the direction it was struck. Things move in straight lines unless other forces act on them. Gravity is acting on the ball, but the direction it is pulling it in is changing over time as the the ball continues on its orbit. The speed you gave it of 120 km/h is small compared to the orbital speed it already had of around 8 km/s. 22.5 minutes after you hit the ball, you and it will have moved through 1/4 of your orbits, and the direction gravity is pulling on you both is now at a right angle from the direction when you hit it. In that time, the ball has moved about 40 km because of the impulse you gave it, in whatever direction you struck it.
Hitting the ball towards zenith or nadir (directly towards or away from the center of the Earth):
When you hit the ball straight down towards Earth, it still has the orbital speed to be higher and it will return to that height - plus a little more. You added energy to it by hitting it and that will translate into a higher apogee.
22.5 minutes after you strike it, the direction you pushed it is no longer straight towards Earth, it is horizontal to the Earth and aligned with its direction of orbit, because you have both covered a quarter orbit in that time. Over that time, the ball has been getting nearer to the surface of the Earth, and so the Earth's gravity has been pulling on it slightly more than before. That has added a little momentum to it. 90o later, that extra speed, plus the speed you gave it, are both moving the ball faster in the direction of orbit. This point is where it will be closest to Earth (perigee) and moving fastest. While at first you would have seen the ball moving straight down, now you will see it moving faster and parallel to the ground, pulling out in front of you in orbit.
Beyond that point, the higher orbital speed of the ball causes it to gain altitude. 180o after hitting it, the ball crosses the altitude of its previous orbit and continues to rise. Through this quarter of its orbit, its orbital velocity was dropping. The orientation of the impulse you gave it is now in direct opposition to gravity, because the ball is now directly on the other side of the planet. But the greater speed gained as it fell towards Earth continues to push it away from the planet. Its orbital speed slows as it rises and you begin to catch up to it.
270o after hitting it, 3/4 of the way around the planet, it reaches apogee - its highest point above the planet, and also its slowest orbital speed. You are at a lower altitude and your greater orbital speed has caught you up to the ball. You pass under it.
Because the ball's total energy, and its total angular momentum, is greater than yours due to the energy you gave it in striking it, it's orbital path is now longer than yours. The average height of the ball's orbit is now higher than yours, which is a larger ellipse with a slower average orbital speed. Once both of you have returned to the spot where the impulse happened, you are ahead of it, and the ball again passes the height of its previous orbit. From your point of view, the ball completes an asymmetric loop around you 90o later. But because the ball will continue to fall further behind you will each orbit, it won't continue to loop around you. From your point of view, it will appear to continue to loop, but ever further behind you - until you eventually 'lap' it and pass it again.
Consider the impulse as movement on a straight line along the direction you hit it, a line that doesn't change orientation, unlike the changing orientation of the pull of gravity. That line is longest 90o later in your orbits, and the ball then reverses direction along it. 180o after, the ball has returned along the line to the spot where it started and begins to extend it in the opposite direction. At 270o it reaches the furthest point on the line in the opposite direction. Once back where the impulse occurred, it returns to the starting point on that line again.
Hitting the ball towards the zenith, away from Earth, would cause a new orbit that is an exact mirror image of the one just described, with apogee 90o later and perigee 270o later. Consider the exact opposite action happening in each step along the way, and that is what results. But again it is a larger orbit, because energy has been added to the ball.
Hitting the ball forwards or backwards along your orbit (a prograde or retrograde impulse):
Because the impulse here is aligned with the orbital path the ball already had, the dynamics are simpler. Impulses in these directions are what are usually used to raise and lower the orbits of spacecraft.
Hitting the ball forwards gives the ball a greater orbital speed, which causes it to pull further away from the planet. This time the arrow in the direction it was struck points away from the planet, at varying angles, through the first half of the following orbit, and towards the planet through the second half. Through that first half the ball gains height and slows, through the second half it loses height and speeds up. Again the added energy means the size of the orbital ellipse is larger and the orbital period is longer. In the first half, the ball would be in front of your position and higher, in the second half it would fall behind you and return to your altitude. Again it would appear to execute a loop from your perspective, and would continue to loop and fall further behind with each successive orbit.
Hitting the ball backwards robs the ball of orbital speed. It begins to lose altitude and reaches its lowest point half an orbit later. The dynamics are the opposite of the description above.
Energy was removed from the ball and its orbit is now smaller, and takes a shorter time to complete. That energy was transferred to you and your orbit changes in the opposite manner, but to a smaller degree because you weigh so much more. This is also true when you hit the ball in any other direction.
Coin funnels have a shape that imitates the dynamics of a gravity well. The link is to a video that may help in visualization. The ball bearings move around the hole very much as objects orbit a central body.
A great way to develop an instinct for this is to download even just the demo of Kerbal Space Program and go through the tutorial.