...for the purpose of vertical landing especially on other planets, what is the alternative to GPS navigation?
Terrain recognition and imaging, and if there's a landing pad, some radio and optical beacons and possibly even a big "X" or other reflective (optical) or topographic (radar) marker if somebody with a company with an "X" in the ...
There are a couple of questions here, I will try to answer them all.
Q1: Is GPS only valid below GEO orbits?
A1: For the most part yes, because GPS satellites transmission are pointed at earth. (Minor Technical correction: GPS satellites are not in fact in Geosynchronous orbits. They are in MEO orbits which are only half way to GEO by altitude)
However the ...
So, you have basically 2 questions:
1.) vertical landing ... 2.) Navigation outside of earths shere of influence
Let us start with 1.) :
Typical extra-terrestial landers are equipt with radar altimeter:
A Doppler radar altimeter on the ExoMars 2016 Schiaparelli Entry, descent, and landing Demonstrator Module...
The lander's [....