The space elevator basically consists of a base station that acts as an anchor, a long cable for movement of 'climbers' and a counterweight to keep the cable in place.
The length of the cable is determined by the intended purpose. Below the altitude of geostationary orbit (~ 35,786 km), the gravity is more than the centrifugal force and above that the centrifugal force is more than the gravity.
Source: Wikimedia Commons: User:Booyabazooka
A space elevator constructed as shown above will pass through the atmosphere, the earth satellite orbits and the Van Allen radiation belts.
The debris in the low earth orbit is but one of the hazards faced by the space elevator. Aircrafts, meteorites and orbiting satellites all can potentially snap the cable.
It is possible to design the satellites so that they avoid the cable(of course you would have to replace every one out there). The orbital debris is another thing.
There are about 6000 tonnes of space junk in orbit and most of them are in the LEO orbit.
Image Source: NASA
There are more than 21,000 orbital debris of diameter 10 cm or more in orbit, which can cause serious damage to the cable as they usually travel in speeds excess of 10 km per second.
Considering the fact that the elevator cable will be practically stationary and the very high number of debris in orbit, it is certain that one of them will hit the elevator cable.
In that case, the cable will snap and more importantly, the counterweight will simply fly out of the earth orbit. Considering the probable impact height (~1000 km) and the total cable height (>36000 km), the entire system will have to be rebuilt again.
This is a serious safety issue for the vehicles using the space elevators during that time as well as they most probably will not have a propulsions stem of their own (this being the most important advantage of space elevator)
In conclusion, the safety of the elevator cable from impacts is one of the most important challenges that has to be solved if the system is to become practical.