To add to other answers, the L1-L2 Lagrange points are unstable because they need to follow radial velocity of the two parent bodies as they orbit each other, in our case the Earth in a heliocentric orbit around the Sun, but none of them are really at the orbital altitude matching their radial velocity (too slow in L1 and too fast in L2). Since the Earth's orbit isn't exactly circular (orbital eccentricity of ~ 0.017), their altitude will also slightly change during one orbital period. This is less of a case with L4-L5, and arguably also L3, depending on orbital eccentricity of parent bodies, and is where we might find Trojan and Hilda family of asteroids, respectively. All these Lagrange points can also be perturbed by gravitational influence of other celestials in the system, for example Jupiter's and even Moon's orbit in case of L1-L2 Sun-Earth points. And this is of course about instabilities in the velocity vector along the M1 parent body. Orthogonal to it and towards the M1 and M2 bodies, it's just the tipping point in direction towards M1 or M2.
Slightly simplifying, what I'm talking about is that the heliocentric velocity (using $v_o \approx {2 \pi a \over T}$) will be $\text{SEL1} \approx 29.49\ \text{km/s}\ \ $ and $\text{SEL2} \approx 30.08\ \text{km/s}\ \ $, where Earth's orbital velocity is $v_o \approx 29.78\ \text{km/s}\ \ $. This difference will be maintained by the L1-L3 saddle points, not too dissimilar to surfing at the tip of a wave. Any lateral movement will tip your balance towards one of the two parent bodies (M1 or M2).
So Lagrange point satellite orbits need to be managed, what's usually referred to as orbital station-keeping. Effects of these perturbations and instabilities can be somewhat offset by placing Lagrange point satellites into Halo or Lissajous orbits and using precision orbital insertion, but not even these will be stable without using onboard propellants and corrections to their orbits. Any orbital debris or whole defunct satellites will eventually spiral towards the Earth (see update to this answer, but they hold too much orbital momentum to really fall towards the Sun, as their orbital period actually matches that of the Earth's, but their semi-major axis towards the Sun doesn't for about ± 1.5 million kilometers or roughly ± 1%).
TL;DR: All this means that the L1 and L2 points would essentially be free from long-term orbital debris. And with L3-L5, unless the bodies there formed from the same protoplanetary disk and share same radial velocity that's required to stay at that altitude, there is nearly no chance that any other body with significantly different orbital energy would be captured at those points. But if we deliberately placed satellites there, any debris from them would stay there for a lot longer than with L1 and L2 points (and perhaps L3, as mentioned before).
Edit: Apparently, I've misread the question initially and was answering for the Sun-Earth lagrange points instead of the Earth-Moon ones. OK, no problem, most of the problems remain the same in theory, only the L1-L3 saddle points are even more unstable. Moon's orbital eccentricity is ~ 0.055, so L1-L3 points move even more along the Earth-Moon axis. On average, EML1 is 326,380 km away from Earth and 58,019 km away from the Moon, EML2 448,914 km and 64,515 km respectively, and their velocities would be (again, average) ~ 0.87 and 1.2 times average orbital velocity of the Moon (1.022 km/s). They are even more perturbed, especially by the Sun itself and the complexity of the Moon's orbit relative to the Sun (it never curves on itself in loops though, contrary to popular beliefs).
Here's a nice image depicting Earth-moon lagrange points:
The Lagrange points for the Earth-moon system. Credit: David A. Kring, LPI-JSC Center for Lunar Science and Exploration
And this is how Lissajous orbits of the ARTEMIS (Acceleration, Reconnection, Turbulence and Electrodynamics of the Moon’s Interaction with the Sun) mission's P1 spacecraft's EML1 and EML2 orbits looked like:
The view from above of the ARTEMIS orbits as they make the transition from the kidney-shaped Lissajous orbits on either side of
the moon to orbiting around the moon. Credit: NASA/Goddard Space Flight Center
Illustration of Artemis-P1 liberations orbits, side or ecliptic view. Credit: NASA/Goddard
Sources: