Generally, I think that there's this mythos about how the Soviet Space Program was superior to the US one, or, more specifically, "red"washing on this topic that paints them as far more competitive with the US than they actually were in reality.
While it is true that the Soviets were able to eeke a handful of notable "firsts" like "satellite in orbit" or "man in orbit" against the USA among others, these calendar-victories almost always came at the cost of scientific rigor and long-term sustainability. While both nations placed great value on the prestige aspects, the Soviets repeatedly showed they were willing to accept more risk, achieve less scientifically, and expend less capital resources, to take the lead in the milestone-footrace.
One great example of this is Sputnik itself. Yes, they beat the US and shocked the world, however Sputnik was pure "PR-machine". It had absolutely no scientific instrumentation, and all scientific value gained from it could've come from putting literally anything into orbit.
Meanwhile, the USA's very first satellite, Explorer 1, contained a multitude of important scientific experiments including multiple temperature sensors, a comprehensive micrometeorite detection system, and a radiation sensing system that allowed Van-Allen to detect (and name) the Van-Allen radiation belts around the Earth.
Now consider that Explorer 1 was launched less than four months after Sputnik.
So, yes, the Soviets "won" the sprint, but even in the earliest moves of the space race, their moves show that they're willing to sacrifice the marathon performance for a shiny medal.
Now, this plays into why they lost the space race and were never able to get the N-1 and many of their more ambitious fantasies to work: the foundation they were building on was simply too focused on short term public-relations victories and military applications, and not focused enough on scientific and engineering rigor to succeed at what is still, to date, probably the most complicated and technically impressive feat in the history of humanity (landing on the moon). This, of course, compounds with what other answers have pointed out like lack of funding, interest, and the death of key figures.
Here's a video (warning, long) on the topic that I believe takes a rather fair and only slightly biased take on this topic.