But they also knew that the mission was likely to be extended, like all missions.
Like all what missions, exactly? Voyager 1 was initially planned as Mariner 11. By the time it flew, all other Mariners were already dead.
I made a list of US solar-system exploratory missions (pioneer, mariner, ranger, viking) up to the launch of Voyager 1, showing durations from launch until last transmission or destruction, and for those that were still alive in 1977, I've put in bold how long they'd been alive and added what their power source was. My "launch failure" includes failure to separate, etc. For fairness and completeness's sake, I've also included satellites, where they were RTG-powered.
- 1958: Pioneer 0: launch failure
- 1958: Pioneer 1: launch failure
- 1958: Pioneer 2: launch failure
- 1958: Pioneer 3: launch failure
- 1959: Pioneer 4: 4 days
- 1959: Pioneer P-1: launch failure
- 1959: Pioneer P-3: launch failure
- 1960: Pioneer 5 (P-2): 3 months
- 1960: Pioneer P-30: launch failure
- 1960: Pioneer P-31: launch failure
- 1961: (sat) Transit-4A: 16 of >63 years, still going - 1st RTG sat, US Navy
- 1961: (sat) Transit-4B: 16 of >63 years, still going - RTG sat
- 1961: Ranger 1: launch failure
- 1961: Ranger 2: launch failure
- 1962: Ranger 3: 2 days
- 1962: Ranger 4: 5 days
- 1962: Ranger 5: 1 day
- 1962: Mariner 1: launch failure
- 1962: Mariner 2: 5 months
- 1963: (sat) Transit 5BN-1: 14 of >61 years, still going - RTG sat
- 1963: (sat) Transit 5BN-2: 14 of >61 years, still going - RTG sat
- 1964: Mariner 3, launch failure
- 1964: Mariner 4: 3 years
- 1964: Ranger 6: 4 days
- 1964: Ranger 7: 3 days
- 1964: (sat) Transit 5BN-3, launch failure - RTG sat
- 1965: Ranger 8: 4 days
- 1965: Ranger 9: 4 days
- 1965: Pioneer 6 (A): 12 of 35 years - solar
- 1965: (sat) SNAPSHOT: 12 of >59 years, still going - RTG sat
- 1966: Pioneer 7 (B): 11 of 29 years - solar
- 1967: Pioneer 8 (C): 10 of 29 years - solar
- 1967: Mariner 5: 5 months
- 1968: Pioneer 9 (D): 9 of 25 years - solar
- 1968: (sat) Nimbus B (Nimbus-B1): launch failure - RTG sat
- 1969: Pioneer E: launch failure
- 1969: Mariner 6: <2 years
- 1969: Mariner 7: <2 years
- 1969: (sat) Nimbus 3 (Nimbus-B2): <3 years - First NASA RTG sat
- 1969: Apollo 11 (EASEP): N/A - RT passive heaters
- 1969: Apollo 12 (ALSEP 1): 8 of 8? years - RTG Lunar experiment
- 1970: (sat) Nimbus IV, 1970: 7 of <11 years - RTG sat
- 1970: Apollo 13 (ALSEP *): 5 days - RTG Lunar experiment
- 1971: Mariner 8: launch failure
- 1971: Mariner 9: <2 years
- 1971: Apollo 14 (ALSEP 3): 6 of 6? years - RTG Lunar experiment
- 1971: Apollo 15 (ALSEP 2): 6 of 6? years - RTG Lunar experiment
- 1972: Pioneer 10 (F): 5 of 31 years - RTG
- 1972: Apollo 16 (ALSEP 4): 5 of 5? years - RTG Lunar experiment
- 1972: Apollo 17 (ALSEP 5): 5 of 5? years - RTG Lunar experiment
- 1972: (sat) TRAID-01-1X: 5 of >52 years, still going - RTG sat
- 1972: (sat) Nimbus V: 5 of 11 years - RTG sat
- 1973: Pioneer 11 (G): 4 of 22 years - RTG
- 1973: Mariner 10: <2 years
- 1975: (sat) Nimbus VI: 2 of 8 years - RTG sat
- 1976: Viking 1 Orbiter: 1 of <5 years - solar
- 1976: Viking 1 Mars Lander: 1 of <7 years - RTG
- 1976: Viking 2 Orbiter: 1 of <2 years - solar
- 1976: Viking 2 Mars Lander: 1 of <4 years - RTG
- 1976: (sat) LES-8: 1 of >48 years, still going - RTG sat
- 1976: (sat) LES-9: 1 of >48 years, still going - RTG sat
- Voyager 1 (Mariner 11), 1977: - >47 years, still going - RTG
Now, I'm cherry-picking, showing only stuff from the US NASA's own probes, and omitting any non-interplanetary spacecraft which lacked RTG.
Still, it's clear that no interplanetary probe missions had, by this point, been extended. Even the four solar-powered Pioneer craft set to study solar weather were still well within their mission specifications, and they didn't have RTGs. The Apollo ALSEP experiments left on the moon were the only extended missions I could find, as they had a planned operating duration of one year. But even all their experiments would be terminated and turned off 25 days after Voyager 1 launched (they were left transmitting the carrier frequency, though, so continued transmitting carrier frequencies for an unknown number of years afterwards).
Overall, there was nothing at this point to suggest that interplanetary spacecraft would last into the next century, let alone for a half-century.
What basis could they have had for imagining such astonishing duration? This was only the third interplanetary mission with RTG power, and the others had only been going for 5 years by launch, likely far less at the time the design was being thought up.
In the late 1970s, a lot of technologies that have since plateaued were then on a rapid, even exponential rise. At least in the common imagination, our then-current rocketry technologies were expected to be completely superseded by the year 2000, and in 2020 we anticipated that we'd be zooming about in jetpacks, flying cars, and spacecraft on the regular. It was all unimaginably far in the future. The first flight of the Space Shuttle still four years away, and that was just the first step to our inevitable path to the stars.
We hadn't realized yet what having flown our last mission to the moon meant, in terms of space exploration. We didn't even know that it was our last mission there: to the designers of spacecraft in the 70s, the idea that we might never return to the moon in their lifetimes was both unimagined and unimaginable.
These probes were designed for one purpose, and that was to do some planetary flybys. By 2020, obviously, we'd be visiting those planets in person, with our ion-drives, launching from our rotating space stations or a moonbase, perhaps using nuclear ram-drives to blast across the solar system in days. The propulsion technology was not expected to stagnate. The flow of multiple probes a year was not expected to dry up.
In 1977, given how much the probes were increasing in sophistication, the idea that it would still be worth extending the mission of the very early ones to more than 50 years, was near unthinkable.
There wasn't really thought to be anything particularly worthwhile for the probes to study once their flyby mission of finished. The idea, made famous by Star Trek: The Motion Picture, was just to cast them out into space as a postcard to any aliens who might find them.
And there's no real point in powering a postcard.
But you also ask not just "should they", but also the secondary question of "would we", if we were to do it all over again, with the benefit of hindsight, knowing as we now do that the components could and would survive essentially indefinitely, and that this experiment in particular wouldn't blow up on the launchpad nor die quickly so we needed to get as much info as we could as fast as we could from it... I think it might arguably be worth the extra radioactive fuel. But as others have pointed out, that might then mean a cost in weight, and hence a tradeoff against the shortest-lived instrumentation.
And the engineers back then were right. There's not a LOT out there in interstellar space to study. So any tradeoff of more years of that very lightweight information density, compared to the insights the instruments gave us on their primary, high-information-density mission of planetary science, is likely not worth the cost.