Not touching upon how "useful" this could be as a stepping stone, as that's difficult to answer, but the more narrow question "Will a pile of rocks stick together at EML4, or drift apart due to tidal forces?" is quite answerable.
An object near EML4 is acted upon by several forces, mostly the Earth's gravity balancing inertia in the rotating frame of reference, with a slight pull from the Moon completing the stability. In addition there are some complications due to the Sun's perturbations.
As a first order approximation, the tidal gradient can't be larger than the Earth's gravitational field. And it turns out that's enough to give a straight-forward answer.
The Earth's gravitational gradient at that distance is $1.4 \cdot 10^{-11 } s^{-2}$
For a reference asteroid, let's use an object like Deimos. Deimos has a surface gravity of $3\cdot 10^{-3} m/s^2$. With a radius of 6km, the tidal gradient of Earth would result in an acceleration of $8.4 \cdot 10^{-8} m/s^2$, far less than the surface acceleration provided by the object.
Surface gravity of objects scales with the radius, provided they have the same shape and density. Similarly, the tidal acceleration also scales with distance, so the question simply boils down to density. Deimos density (1.5 kg/l) already provides about 4 orders of magnitude of safety, so:
Objects, of any reasonable density, placed right next to each other near ELM4, will stick together purely by gravitational forces, provided they are large enough to not be blown apart by radiation pressure