To offer a direct answer up-front, this document published by NASA's chief historian in 2000 examines NASA's missions and history from 1950-2000. I admittedly have only skimmed this document, but this is probably one of the best documents you'll get showing NASA's pre-Apollo plans. An important factor to consider with your question of "what if NASA focused on something else back then" is the historical context - the US and USSR were basically trying to prove to each other that they could nuke each other more than the other party. So the Apollo program wasn't predicated on "let's eventually get to Mars", but moreso on "how well can we master rockets for the purposes that we need them today?", meaning that the program itself wasn't built with sustainability or long-term technology development (relating to deep-space exploration anyway) in mind. Now, putting the historical context aside for a bit, let's talk about why it wasn't sustainable.
The Apollo Program is considered a "flags and footprints" program in which the goal was basically to go to the moon for the point of saying that it could be done. Generally speaking, this type of program is not sustainable for long-term exploration because once you have achieved your objective, there is little incentive to continue sending more missions. While the later Apollo missions had expanded science objectives compared to the first, the financial cost and human risk of each mission quickly outpaced the marginal science and prestige returns. After the first few missions, NASA already had enough lunar samples to a great deal of analysis. Getting more samples would be nice, but they wouldn't give much more information than what was already had by the first samples. Also, saying "I can go to the moon and come back" is extremely impressive. Being able to do it twice proves it wasn't just a fluke. Doing it a 7th time instead of a 6th time just isn't as big a deal. Considering the relatively huge amount of the US budget that was going towards NASA at that time, Congress eventually decided that the point had been made and there was no need to do more missions.
Just because Congress pulled the plug on the program doesn't mean that an incredible amount of technological progress was made in many fields: some examples and more examples. The problem wasn't that the program wasn't productive, it was that there wasn't a reason to continue doing it once the single, clearly defined objective was achieved. So what kind of program would be more successful in the long run?
Three major factors to consider are the goal(s) of the program, the technologies available to achieve the goals, and the funding available to support technology development and mission operations (plus everything else required to make those happen). First, NASA would need to develop an appropriate goal. The goal itself would need to be indefinite, or at least provide concrete progress towards a more advanced goal. These could be something like: long-term LEO/cislunar colony, human asteroid landing, permanent lunar colony, permanent Mars (or Martian moon) colony, or something similar. Now, if NASA can create an efficient technology roadmap to achieve this goal AND it can create a plan to effectively develop the required technologies within a reasonable annual budget, then the program could have a shot at surviving for a long time.
A series of comprehensive studies by NASA that looked at the long term exploration of Mars are the Mars Design Reference Missions. The first one was completed in 1993, and the most recent one was completed in 2009. Each iteration took into account lessons learned from previous versions of the study, as well as new and changing technologies. Another solid study done by JPL in 2015 was the Minimal Mars Architecture (I couldn't find a public version of their paper). There have been many other studies of varying magnitude by other authors, but the examples I gave are from some of the biggest brains in NASA.