Why can't I just use dot product angles to get satellite vision of a point on Earth?

Right now I'm using MATLAB and can easily get the ECEF coordinates of the satellite and the point on the Earth. Since ECEF is cartesian, why can't I just rearrange:

$$r_{es}\cdot r_{ps} = \|r_{es}\| \|r_{ps}\| \cos(\phi)$$ to get: $$\phi = \arccos\left(\frac{r_{es} \cdot r_{ps}}{\|r_{es}\|\|r_{ps}\|}\right)$$ and then just say when phi is less than the cone half-angle of the sensor on the satellite, the point is in view? When I tested this method, I found that depending on the latitude of the point (with circular satellite orbit at altitude = 1000, cone half-angle = 60), phi is between 38 and 45 as a maximum when the point is first seen at the edge of the cone.

To account for this method not accounting for the Earth's obstructing vision of the point, I also incorporate the same method to get the elevation angle from the point to the satellite, where

$$el = 180^{\circ}-\omega = \arccos\left(\frac{r_{ep} \cdot r_{ps}}{\|r_{ep}\| \|r_{ps}\|}\right)$$ and in the code I implement it by

if phi < 60 && el < 90
vision = True

However, with some testing, I need to set el < 95-105 (depending on the lat of the point) to get accurate results (verified with STK).

It makes so much sense to me that this method with the dot product angles should work in the way I explained, but I have no idea why it doesn't work out that way. • the earth is not a perfect sphere. Apart from being a pear-shaped and oblate, it has valleys and mountains?
– user20636
Aug 4 '20 at 16:37
• I realize in my diagram it is a sphere (mainly bc I used Paint to make this). My model uses oblate Earth, but does not take into account mountains or valleys, neither does my check scenario with STK, which is also oblate Earth Aug 4 '20 at 19:32