There are a number of reasons for this, and as is often the case, it is the result of a trade-off.
Some of the factors that were included in the trade-off:
- The launch site should allow launching to the east
- Availability of (a lot) of land for development of launch pads and industrial complex
- Availability of civil infrastructure (electricity, water)
- Nearby cities for staff and workers to live
- Availability of skilled workforce and/or willingness of such workforce to move to the vicinity of the complex
- Accessibility via water for delivery of rocket stages
Especially the distance of American Samoa to the US main land would be an issue: it is unlikely that a lot of people would be willing to move that far away (making staffing a problem) and shipping rocket stages across the ocean is - while possible - not ideal from a risk point of view.
Moonport is a good reference for a more detailed exposition.
Note by the way that Europe does not have a lot of alternatives: most of Europe is too far North and there is no large enough body of water or uninhabited land on the east to launch over. Overseas territories are basically the only option.