EDIT October 15, 2017: the problem in this question is cast as an optimal control problem and reformulated here.
The spacecraft I am designing is flying in an elliptical orbit (8 x 3000 km) around the Moon. At periapsis, the non-impulsive bi-propellant retromotor mounted upon the spacecraft will cancel the orbital velocity while at the same time increase the altitude from 8 to 28 km (the spacecraft will then start its free-fall phase to the surface, but that's not part of this problem). The Moon is assumed to be a perfect sphere with $r_{Moon}=1737.100$ $km$ and non-rotating.
The state vector consists of 7 components (position and velocity vectors, plus the mass). The initial state vector is defined as:
$$ \textbf{y}_0 = \begin{bmatrix}x \\ y \\ z \\ \dot{x} \\ \dot{y} \\ \dot{z} \\ m \end{bmatrix}_{t=0} = \begin{bmatrix}1745100 \enspace m\\ 0 \\ 0 \\ 0 \\ 2026.813 \enspace m/s\\ 0 \\ 70 \enspace kg \end{bmatrix} $$
The dynamics are given by:
$$ \ddot{\textbf{r}}=-\mu\frac{\textbf{r}}{r^{3}}+\frac{\textbf{T}}{m} $$
So:
$$ \dot{\textbf{y}} = \begin{bmatrix}\dot{x} \\ \dot{y} \\ \dot{z} \\ \ddot{x} \\ \ddot{y} \\ \ddot{z} \\ \dot{m} \end{bmatrix} \quad \textbf{f}(t,\textbf{y}) = \begin{bmatrix}y_4\\ y_5 \\ y_6 \\ -\mu\frac{y_1}{r^{3}}+\frac{T}{y_7}\cdot (k_x ?) \\ -\mu\frac{y_2}{r^{3}}+\frac{T}{y_7}\cdot (k_y?)\\ -\mu\frac{y_3}{r^{3}}+\frac{T}{y_7}\cdot (k_z?) \\ -\frac{T}{I_{sp}g_{0}} \end{bmatrix} $$
With $r=\sqrt{y_1^{2}+y_2^{2}+y_3^{2}}$ and constants:
$\mu = 4.90487E(+12) \enspace m^3/s^2$; $T = 200 \enspace N$; $I_{sp} = 300 \enspace s$; $g_{0} = 9.81 \enspace m/s^2$
The $k$'s in the set of equations should determine the thrust magnitude (and sign) in that particular direction ($x$, $y$ or $z$) at each integration step: $\textbf{k}$ is the unit vector of the thrust vector. The solution I'm seeking should be in the form of a vector set containing thrust vectors at each integration step of the breaking/climbing phase.
The final state vector should be:
$$ \textbf{y}_n = \begin{bmatrix}x_n \\ y_n \\ z_n \\ 0 \\ 0 \\ 0 \\ m_n \end{bmatrix} $$
Where $r_n=\sqrt{x_n^{2}+y_n^{2}+z_n^{2}}=1737100+28000=1765100 \enspace m$ and $m_n$ must be maximum.
Any ideas on how to find an optimal solution (read: optimal thrust-vectoring strategy, final mass $m_n$ should be maximum) for this boundary value problem?
Thanks a lot!