Orbit propagation (calculating the trajectory of an object under the influence of, at a minimum, gravity of a central body) is at the heart of astrodynamics, and high precision propagation means numerical solutions of ordinary differential equations. This, in turn, requires relatively deep understanding of numerical analysis so you can implement your algorithms on a computer.
Probability theory can also come in handy, particularly when coupled with the estimation/determination problem. We have the ability to take very precise measurements of a satellite on-orbit, and these may be used to update the state of the satellite (via filtering and least squares, among other techniques), but inherent uncertainty in the measurements and the dynamics models will lead to uncertainty in the state estimate. Understanding how to characterize and propagate this uncertainty is currently a very hot topic of research. Additionally, most of the processes involved in orbit propagation and determination are nonlinear, which adds complexity to the estimation process.
Of course, good ol' multivariable calculus always comes in handy, both for propagation (calculating Jacobians) as well as for understanding the underlying effects of different perturbations, such as the asphericity of Earth. In fact, General Perturbations (GP) theory, while not used much for high precision propagation any more, still proves useful in the orbit design process, and involves analytically averaging equations of motion.
While I wouldn't classify it as "advanced" per se, a deep understanding of geometry is an absolute must. It's manifested largely in coordinate system transformations.
Low thrust trajectory design, another hot topic, often boils down to an optimization problem, which is a whole field unto itself.
I know someone will get mad because I didn't include their sub-specialty here, but I think I captured the topics that come up most often. It's a very interesting, diverse field.