The Lagrange points are not some physical objects, just locations where gravity of other bodies coincides to be of specific value/"shape", and as the strength of gravity field changes, so do the point move; obviously whatever affects the gravity does influence these points, like trajectory of given body that generates them, its mass (influenced by capturing meteorites or losing mass due to massive impacts), other passing bodies etc.
Of course such changes are rather infrequent and/or slow (or minuscule) and so the points change rarely and by not much; also, L4 and L5, being stable, will drag whatever fell into them as they shift following the changes, unless the changes are very drastic (say, given celestial body split into multiple pieces by a massive impact.) and other bodies passing by rapidly shift them too.
Also, never to forget, they permanently travel along with the orbit of the body - they are "almost immobile" in relation to the Earth-Sun system, but that system is mobile!