Your intuition is quite correct.
The Hohmann transfer orbit is a bi-tangential orbit, so at the point where the spacecraft leaves Earth, it is travelling in parallel to us.
In the case of Mars, we want to travel slightly faster than the Earth in order to lift our aphelion up to the orbit of Mars, meaning we want a little extra velocity on top of the prograde velocity. Hence, we escape prograde.
In the case of Venus, we want to travel slightly slower than the Earth in order to lower our perihelion down to the orbit of Venus, meaning we want to subtract some velocity from the prograde velocity. That means a retrograde escape.
The direction of escape certainly matters. In fact, the escape velocities to enter a Venus transfer orbit and a Mars transfer orbit are approximately the same, but the escape being in opposite directions means the resulting orbits are very different.
In practice, the directions are often going to be a tiny bit off from perfect prograde or retrograde. Angling the escape slightly outwards from the Sun, you can reach Mars faster, albeit at a somewhat greater velocity cost. Saving a week or two of transfer time in this way is close to free, while the cost increases dramatically the more time you want to save.
Earth and Mars are also not quite in the same plane, so the transfer orbit needs to correct for this. Combining the plane change with the escape manoeuvrer is more efficient than doing them separately, so the hyperbola will point slightly up or down from the ecliptic.
