(with considerable help from "Why we'll probably never build a space elevator")
You have laid out a good foundation, for the first, and largest challenge, namely the material for the cable itself. Carbon Nanbotubes are the best substance we know of to build a Space Elevator. In their purest form, they have a tensile strength of over 100 GPa. The exact number varies, but the general consensus is 130 GPa would result in a safe elevator. The next challenge beyond that is to build one that is the required length, and combine them into a workable cable, which isn't an easy task.
The second significant obstacle is how to keep the cable stable. It would vibrate with time, which could make managing the cable quite difficult. Add in space debris already in orbit (a ton), and it would be extremely challenging. Keeping the cable steady while climbing would take a considerable amount of time, perhaps up to a month for the trip to orbit.
Then there are the issues of where to build the space elevator, security, and other related issues. It's an extremely challenging problem.
There is some hope, principally in constructing an elevator at the Moon, Mars, or similar locations. In fact, one could be built around the Moon using technology easily available, and there are far fewer societal issues with that than would exist around the Earth. Around Earth would be quite difficult, at least, in the foreseeable future.