Since nobody has jumped in so far to answer except @DavidHammen's incredibly helpful comment I'll write an answer to my own question.
Other/different answers are more than welcome!
His link http://messenger.jhuapl.edu/About/Mission-Design.html#final-extended-mission points to the final extension of the Messenger mission. This is a really nice summary and explains things so well that I don't know how to summarize.
Instead I will point out certain aspects of these strange wiggles shown in the screenshot.
Messenger was in a highly elliptical polar orbit around a low mass body near an extremely massive body (the Sun) so its orbit is constantly being pulled-on and reshaped, and there were many orbit correction maneuvers as well.
The maneuvers were used to
a) offset the perturbation effects that would have prematurely ended the mission by moving the periapsis down to the surface, causing end of mission.
b) keep the periapsis over latitudes (and longitudes) of interest to better study certain features. While the periapsis altitudes were of order 100 km or less, apoapsis was way up at roughly 10,000 km altitude.
It's important to note that proximity to the Sun had other impacts on mission design. Messenger had a sunshade and that needed to be kept pointing towards the Sun to avoid overheating while the antenna needed to be pointed towards the Earth.
Also, communications was lost whenever the line-of-sight came to close to the Sun as seen from Earth, when Mercury would be at superior conjunction.



Maneuver Calendar date dV(m/s) Purpose
MOI 18 Mar 2011 861.7 insert spacecraft into orbit around Mercury
OCM-1 15 Jun 2011 27.8 lower minimum altitude to 200 kilometers
OCM-2 26 Jul 2011 4.1 increase orbit period to 12 hours
OCM-3 07 Sep 2011 25.0 lower minimum altitude to 200 kilometers
OCM-4 24 Oct 2011 4.2 increase orbit period to 12 hours
OCM-5 05 Dec 2011 22.2 lower minimum altitude to 200 kilometers
OCM-6 03 Mar 2012 19.2 lower minimum altitude to 200 kilometers
First Extended Mission
OCM-7 16 Apr 2012 53.3 Decrease orbit period to 9.1 hours; deplete oxidizer
OCM-8 20 Apr 2012 31.5 Decrease orbit period to 8 hours
Second Extended Mission
OCM-9 17 Jun 2014 5.0 Target 25-km minimum altitude on 12 September 2014
Met 25.2-km minimum altitude on 12 September 2014
OCM-10 12 Sep 2014 8.6 Target 25-km minimum altitude on 24 October 2014
Met 25.4-km minimum altitude on 24 October 2014
OCM-11 24 Oct 2014 19.3 Target 25-km minimum altitude on 21 January 2015
Met 25.7-km minimum altitude on 21 January 2015
OCM-12 21 Jan 2015 9.7 Target 15-km minimum altitude on 1 March 2015
Met 14.9-km minimum altitude on 1 March 2015
Final Extended Mission
OCM-6 03 Mar 2012 19.2 lower minimum altitude to 200 kilometers
OCM-13 18 Mar 2015 3.1 Target 5.7-km minimum altitude above terrain on 2 April 2015
OCM-14 02 Apr 2015 3.0 Target 13.1-km minimum altitude above terrain on 6 April 2015
OCM-15 06 Apr 2015 1.8 Target 6.9-km minimum altitude above terrain on 14 April 2015
OCM-15A 08 Apr 2015 1.9 Target 6.9-km minimum altitude above terrain on 14 April 2015
OCM-16 14 Apr 2015 1.0 Target 9.0-km minimum altitude above terrain on 24 April 2015
OCM-17 24 Apr 2015 1.5 Raise minimum altitude to delay Mercury impact to 30 April 2015
OCM-18 28 Apr 2015 0.45 Raise minimum altitude to delay Mercury impact to 30 April 2015 at 19:26:01 UTC
Here is some data from the simulated orbit in JPL's Horizons database. This solution was revised in 2015. Messenger's orbit around Mercury was every 12 hours. I downloaded the osculating orbital for Messenger in orbit around Mercury at a cadence of 1 hour. Each data point is the median of 24 values over one day, which smooths the tiny wiggles do do the fact that the orbit is not purely Keplerian and so osculating elements wiggle.
The strange curved undulations in periapsis distance are likely due to the highly elliptical orbit bringing Messenger far from Mercury so that the Sun's influence is stronger. You can see the periapsis wiggles decrease when the apoapsis drops down, However, I am not sure why those wiggles pick up again towards the end even though apoapsis only increases a little.
I believe that these changes will correlate with the angle between Messenger's orbital plane and the direction of the Sun's gravitational attraction, but I've run out of steam here...
This is a very complicated and interesting Mission!
