You are correct a high launch location could help with rocket efficiency. Engines with nozzles optimized for lower atmospheric pressure, usually resulting in higher specific impulse primarily, also lower dynamic pressure and atmospheric drag. The "distance" gain is minuscule - maybe half a second worth of fuel usage, but the pressure change would be beneficial.
The problem is a set of other restrictions and requirements of a launch site.
- Safe "downrange" area - a couple hundred kilometers preferably completely uninhabited, both for planned booster fall zone and in case of mid-flight explosion. Sea is optimal for that.
- Politics. Not launching over another country's territory. Technically after getting over the Karman line there's no issue... but if the rocket fails to achieve orbit, it's better for it to fall onto international waters or own territory, than onto territory of a country you're in tense relations with. Also try to imagine USA launching spy satellites from a spaceport in Bolivia... The spaceport should definitely be on "home territory." (Russia is definitely not happy about Baikonur being located in Kazakhstan.)
- Relatively flat and safe to land on, for manned launches. Launch Escape System is a great device that assures in case of trouble at launch the astronauts will be brought safely to the ground... but doesn't help a bit about where exactly and what happens afterwards, e.g. if they land on a steep slope and the capsule starts rolling down rapidly... Mountains aren't safe for landing.
- Logistics. Many of rocket parts are too huge for land transport over large distances, and delivery by sea is often the only viable route. Also, good land transport network for all that can be transported by land - usually best developed in coastal areas.
- Employee comfort, health and safety. People qualified to work for the rocket industry are top professionals who'd easily find work in more hospitable areas had their workplace turned out to be too inhospitable - like high altitude settlements often are. You can bet warm beaches of Florida are more welcoming than Bolivian highlands.
Economy and economical politics. The spaceport is supported by a whole lot of industries big and small and stimulates the economy immensely. So you want it to be domestic for this reason. Bolivia can't afford own major spaceport. USA could have one in the Rocky Mountains, but - range safety! It would be very hard to find enough safe uninhabited terrain to assure safe 'drop zone' for the spent boosters etc.
Local development, economy, industry and infrastructure. It's much easier to have a spaceport near existing manufacturing facilities of firms that operate or are capable of operating producing the rockets, than to have the rockets transported across half the country - or build the entire production infrastructure from scratch at the site. Again, coastal areas tend to be most developed in these domains.
As result, the benefits of high altitude don't outweigh the costs of otherwise awkward location.