I had never noticed it before, but when I was watching Apollo 13, the path of the rocket looked like this. Obviously since it's going around the moon, it won't actually be elliptical, but that doesn't even look close around the other parts. Is this an accurate model of the orbit? And if so, how can that path be modelled mathematically? Sorry if its a basic question, I'm new to this corner of stack exchange.
Orbital paths are elliptical in the case where only one massive body is affecting a spacecraft's path gravitationally. In this case, the Earth and the moon each have significant gravity, so you could approximate the path as a partial ellipse from Earth to the point where the moon's gravity dominates, then a different partial ellipse going around the moon, then another partial ellipse returning to Earth. As Hohmannfan notes, this is the "patched conic" approximation, because conic trajectories are patched together.
The trajectory in your image is actually elliptical. It is just projected in a way that makes it appear differently.
How? This is a rotating frame of reference; that is, our point of view is rotating along with the Moon's orbit so both the Earth and the Moon appears stationary.
Here is another example of how this works: Stand in the middle of the room and start spinning. Then place your hand on your face and move it outwards. From your point of view it is moving in a straight line away from you, but for another person in the room that is not spinning it looks like your hand is moving in a spiral.
The simplest way of modelling this orbit is the patched conic approximation. It is sufficient in most cases, and assumes we are only affected by the gravity of one object at a time. In this case, the first part of the trajectory is an ellipse with respect to Earth, then it is a hyperbola with respect to the Moon, and then an ellipse again. A more advanced model would be to always consider the gravitational influence of both the Earth and the Moon. This, however is known as the three-body problem and can only be solved by numerical simulations. (Though restricted versions can have usable analytical approaches.)
TL;DR Patched Conics solutions look WEIRD when viewed in synodic frames. Since they do not reflect gravitationally correct trajectories, representing them in rotating frames will accentuate their "rough edges."
The kernel for this answer is here.
From page 437 of An Introduction to the Mathematics and Methods of Astrodynamics by Richard H. Battin, as seen in google books:
An adequate approximate trajectory may be had by matching in both position and velocity at the junction points (1) an ellipse from Earth to the sphere of influence of the moon whose focus is at the center of the earth, (2) a hyperbola around the moon, and (3) an ellipse from the sphere of influence back to Earth. The simplified problem, though itself fairly complex, is tractable when then relevant parameters and independent variables are identified.
The faint line in the drawing in the question and the drawing from the linked reference (the line drawings below, not the photos), is shown with the moon fixed. This is a frame rotating with the moon's orbit around the Earth.
The hand-drawn yellow-orange arc to the right of the moon in the drawing highlights what bothered me, that there does not seem to be an easy way to smoothly patch a hyperbolic trajectory around the far side of the moon to either of the elliptical conics with the Earth as the focus.
The drawing shows corners where the elliptical segments are patched to the hyperbola.
In my opinion the drawing highlights a problem understanding how this orbit can be approximated with two elliptical segments and one hyperbolic segment that are smoothly patched - preserving both continuity and velocity at the patch points simultaneously.
The resolution to this apparent problem is that at each patching, there is another reference frame shift. Another way to say that is that "the moon is moving!" So when you patch back to the 2nd elliptical segment, you have to remember that you've shifted reference frames twice!
The drawing in the question is provocative, it certainly gave me something to think about, but without the accompanying talk, or text, or subtext, it is does not teach us much more than to dig in to the issue more deeply because drawings can be deceiving and those Apollo engineers were really clever!
Take a look at Exploration of Lunar Free-return Trajectories by Brian Michels, ASEN 5050, University Colorado, Aerospace Sciences where there is a nice review of the patched conic analysis of the free-return orbit, and includes a lot of raw, historical data from that analysis: