Electromagnetism
To be as practical as possible, the Space Shuttle mission STS-75 experimented with a space tether. For the record, the tether did break. To be fairly exact, it did break for reasons that wouldn't have broken it if the line had been shorter. So in a certain sense (but only a limited sense), we have already seen a space tether break because of its size.
That was because of wire current interacting with trapped air from manufacturing in the line. The current was caused by movement within the Earth's magnetic field. That was 20 km long. We can't call that a limit in any meaningful sense, because NASA could make a better tether now, and the entire point of the mission was to experiment with induced current. So an insulator would fair better.
Nonetheless, induced current is still size specific. It's also specific to velocity, magnetic field, and probably a few other things.
Tidal Forces
The reason there was tension in the wire to begin with was tidal forces. There are abundant tether proposals for various cislunar systems that take advantage of the gravity gradient. It was even considered to make structures for the ISS that would use the tidal forces to help maintain orientation.
There are some qualifiers. Tidal forces only extend in one direction, so it may not actually stress your structure depending on geometry and how it's pointed.
For a basic mathematical treatment, the tidal field grows linearly with movement from the central point. If you have a mechanical member, then that is integrated over the length to give a tensile force on the order of (length)$^2$, I believe.
In a meaningful sense, a space elevator has to fight against tidal forces, and it would require futuristic materials. My poor $l^2$ approximation would be insufficient on that scale.
Self-gravitation
Something can collapse under its own weight. Of course, we'll have to assume that it's not relying on internal pressure. Planets rely on their own internal pressure to hold itself stable against self-gravitation. This is fairly unspectacular, and since we're looking for man-made structures, I'll imagine something like the Death Star.
If a spherical space station has a constant density, the gravity grows linearly from the center. Because of that, it would have the same (length)$^2$ scale of structural requirements, but this would be compressive. You can push materials limits further with tensile loads than compressive loads in general.
It's important to note that self-gravitation depends on the average density of the structure. Actually, it's a (density)$^2$ relationship. The argument for the square is that on Earth your weight is just (mass), because it's gravitation between you-Earth. Self-gravitation is between you-you. Thus, your mass term gets entered twice. This means that a very sparse structure could theoretically span a size greater than that of Earth, while not collapsing under its own self-gravitation.
Others
There's another way you could push this even further - use kinetic forces. You could have a rigid structure that rotates in a big ring, which avoids some self-gravitation compression. You could push that idea very far, with huge self-gravitating Dyson swarms, or something along those lines. But maybe that would fail the requirement for not being "rigid". There may be other workarounds. My creativity fails me at this point.
At this point, reaching absurdity, there are some odd and even comical limits you can think of. For instance, if you assume that our energy growth continues to grow at 1% per year, we will cook the Earth in 1000 years or so. It's not a complicated argument. Just assume we continue to exponentially grow and the conclusion is obvious. This can be applied on any boundary, including the solar system or the galaxy.
Ultimately, yes, the limit would be a becoming a black hole. It's difficult to see how that would happen sooner than the thermal limit, because it's highly sensitive to the matter density. Theoretically you could make a black hole without a catastrophic event like a supernova, because large black holes can have a "density" (defined with the event horizon) less than water. So if you were flying giant blocks of lead around in space in a carefully planned formation over many light years, you could turn into a black hole in a very novel way. But why?