I think I solved it... I'm not sure but I think I'm on the right path.
You were very close to the right solution, and you were very correct the linked equation is useless in describing actual trajectory of a "Karman Plane" - it only works for flat Earth :) Karman's Line is an abstraction that has very little physical meaning in the real world - it's derived from a lousy hybrid of two equations:
the aviation equation for level flight: $mg = {\rho v^2 S C_l \over 2}$ (weight = lift; $W=L$); assumptions: flat Earth, constant gravity
the circular orbit equation: $v^2 = {GM\over r}$ or 'unabridged' ${G M_e m \over r^2 } = {m v^2 \over r}$ (centripetal acceleration = gravitational acceleration, $W=ma$); assumptions: vacuum. (also, $r = R+h$)
The hybrid extends the level flight equation by the gravity drop-off with altitude, but doesn't account for Earth being round, a simplification one can't afford in orbital mechanics.
Your version correctly expands the orbital equation by giving it an aerodynamic component: $W-L=ma$, where
Lift $L = {\rho v^2 S C_l \over 2}$; (with $\rho$ being a function of height; proportional to pressure approximated as $P_0 e^{- {h \over H}}$)
Weight $W = {G M_e m \over (R+h)^2 }$
centripetal acceleration $a = {m v^2 \over R+h}$
Now, why your equation won't help:
If we take standard lift, it will never vanish; its density component is dropping off exponentially with altitude, and as orbital speed also drops with altitude, the velocity component will drop with it, so it's a huge $x e^x$ drop-off rate, but never zero. This makes it impossible to find the altitude where the airplane "doesn't produce lift". Angle the solar panels of ISS right and they will still produce some millinewtons of lift.
But what we can do is to take negative lift coefficient, try to prevent the spaceplane from being ejected into a higher orbit, and find an altitude where we'll fail that. Fly the airplane "belly-up" and see how high can it go at speed slightly exceeding the orbital speed for given altitude before it gets ejected into an elliptical orbit as its dwindling lift fails to prevent that.
So let's switch the sign: $W+L=ma$
The velocity is slightly larger than necessary. Not infinitesimally but of order of just a couple m/s.
$v_k^2 = {GM_e \over r}+\epsilon$
The equation will take form:
${G M_e m \over r^2 }+{\rho v_k^2 S C_l \over 2}= {m v_k^2 \over r}$
Let's substitute the $v_k^2$
${G M_e m \over r^2 }= ({m \over r} - {\rho S C_l \over 2} )v_k^2$
${G M_e m \over r }= ({m } - {\rho r S C_l \over 2} )v_k^2$
${G M_e m \over r }= ({m } - {\rho r S C_l \over 2} )( {GM_e \over r}+\epsilon)$
Simplify it a bit, solve for r
$m = (m - {\rho r S C_l \over 2} )( 1+{ \epsilon r \over GM_e})$
$0 = - {\rho r S C_l \over 2} - {\rho \epsilon r^2 S C_l \over 2GM_e} + { m \epsilon r \over GM_e}$
$0 = - {GM_e \rho r S C_l } - {\rho \epsilon r^2 S C_l } + { 2m \epsilon r}$
$0 = (2m \epsilon - {GM_e \rho S C_l })r - {\rho \epsilon r^2 S C_l }$
$r = {2m \epsilon - {GM_e \rho S C_l } \over {\rho \epsilon S C_l } }$
and reorder into aerodynamic and gravitational parts.
$r = {2m \over \rho S C_l} - {GM_e \over \epsilon } $
And so, the equation to solve would be
$R+h = {2m \over \rho(h) S C_l} - {GM_e \over \epsilon } $
Normally, the $GM_e \over \epsilon$ part should be very large - but even for quite small $\epsilon$, $\rho(h)$ diminishing exponentially should create growth of the aerodynamic part rapid enough to give a good solution.