For the purpose of $\Delta v$ planning, an orbit at Earth-Moon L4 or L5 is about identical to just any orbit with a radius of 380,000 km. At L4/5 distance to Earth and Moon are identical and as the gravitational force changes with $r^2$, the pull of the Moon is just 0.01% of that of Earth at this point and can be neglected. It's sufficient to keep the orbit somewhat stable, but doesn't change the $\Delta v$ to get there.
As Steve Linton already commented - If you are in LEO and raise your speed (to be precise, energy) to C3=0, you have enough energy to escape Earths gravity well, but not enough energy to get into a high Earth orbit. If you check the chart, even going to GEO takes a lot more $\Delta v$ than reaching C3 = 0.
Adding 3.2 km/s (just short of C3=0 to avoid infinities) to your speed in LEO (200 km height) brings you to an orbit with a huge apogee (millions of kilometers), but the perigee is still in LEO. To get into a circular orbit in any height, you need to spend more fuel to raise the perigee to the same height, which will cost you another substantial amount of fuel. To get to L4/5 you have to spend 3.13 km/s in LEO, and another 0.83 km/s to raise the perigee.
And, indeed you are right, that L4 can be reached via a swing-by maneuver around the Moon, increasing the orbital speed at apogee and therefore decreasing the magnitude of the second burn (in the ideal case, by about 60%). In fact, the same could be used to get to L5: L5 is trailing the Moon by 60°, but at the same time can be thought of leading the Moon by 300°. I.e. you can "overtake" Moon, take the "long way" around the orbit and finally end up at L5. In both cases you need yet another, tiny, burn to adjust the orbital velocity to the right value once you reached L4 or L5.