It's the exponential nature of the equation determining the thickness.
Let's discretize the problem to make it simpler to picture: you have a thread of very strong material, that has a self-breaking length of two kilometers. That means, if you hang more than two kilometers of this rope on iself, it will break under own weight.
Let's cut it in kilometer long pieces, so each piece can hold another time its weight. Then let's try to extend it.
First two segments are single strings, each a kilometer long, first one with 50% redundancy, the other at breaking limit.
The third kilometer must hold the weight of two lengths - so it must be two such pieces in parallel.
Fourth kilometer holds weight of four lengths (the initial two and the segment with two), so four ropes in parallel.
Fifth - eight times the thickness. Sixth - sixteen. Seventh - thirty-two. To add the twelth kilometer, the thickness must be 1024 times the initial.
It's not actually AS bad, as with altitude gravity starts dropping both with distance of the central body and with fraction of orbital velocity. But it's a long time and a huge stress before we get there and the thickness starts stabilizing, then dropping as it extends past GEO. Still, with current material engineering, the amount of material needed is huge - and in case of materials less durable than nanotubes the exponential explosion means there's not enough material on Earth to build it, as it quickly gets into continent-sized thicknesses..
One of earlier estimates for using nanotubes gave the thickness of 1km around GEO for 1 ton payload. There are newer ones that give more promising estimates, but that should give you a clue why this is a problem.