A little bit of background in case you are interested. I am running a re-entry simulation, and I am dealing with two different reference frames. The one with the higher fidelity dynamics is in the simulator, whilst the lower fidelity dynamics are used in a closed loop algorithm that calculates the optimal trajectory that is calculated in the trajectory based on a guidance frequency. The former uses the NED (North up, East Right, and Down positive; it is almost like the local vertical and local horizontal) convention for a 3DoF Rocket. The guidance algorithm instead calculates the trajectory in a flat earth environment ($X',Y',Z'$), thus no curvature is considered. The Launch pad is located at a certain known latitude and longitude, and the simulation starts with a displacement only in latitude (no crossrange, just altitude and downrange).
Here is where the problem begins. For some reason the simulation does not follow very well the results of the algorithm. Note that this is even in the most extreme case where there are no aerodynamic forces but there is only thrust and gravity. The guidance algorithm returns feasible (thus optimal) solutions, meaning that there are not any problems to suspect with the algorithm itself. Here is a sample of solutions superimposed over the simulator's result.
As you can tell, the guidance solution correctly calculates a curve that returns at the origin (the launch pad location) in multiple guidance steps, until there are no new solutions because the simulated trajectory has deviated so much from past solutions that it cannot find a way to go to the launch pad from there.
The way that the output reference thrust is converted from the NED frame from the guidance algorithm's flat earth frame is by the following transformation matrix:
This is fairly intuitive as can be seen from the conversion of the NED frame at the launch pad to the flat earth plane which is really just rotating the axis at that point:
However this does not seem to be the solution because the trajectories do not reflect the optimality of the solution of the guidance algorithm.
So one of my alternative hypotheses was that the thrust acceleration should be evaluated with respect to the local NED frame at the point where the rocket is. This would mean converting from the $X'',Y'',Z''$ frame to the $X',Y',Z'$ frame and back to the NED frame when feeding the reference thrust acceleration to the simulator. My question is how this is done.
Alternatively, it could be that the coordinate transformation is fine and something else is going on, because if you take a look at the point where the rocket is meant to start doing a toss-back (that hook at the top), it does not do it in the same plane but rather it starts building up a positive crossrange.
Any suggestions or comments are appreciated.