(Started writing this before your edits — decided to keep going since it still answers the questions)
Here are the major problems as I see them.
A lunar colony would need a lot of help from Earth before it could be self-sufficient. The bare necessities are a habitat, oxygen, food, and water. It would take a long time to get just the habitat up there — look how long it's been since we started making the ISS, and getting things to Low Earth Orbit is much easier than landing them on the moon. Similarly, we'd need ways to get oxygen, food, and water up there until the colony can become self-sufficient — if it ever gets that far. We're currently working on growing food in low-gravity environments, so that could become sustainable. There is a lot of water ice at the lunar poles, but that would require more help from Earth to get a mining operation going.
Long story short, technically it would require advancements in "space survival", meaning generation/recapturing of food, water, and oxygen in environments not naturally conducive to survival, as well as (probably) advancements in rocketry that would make it cheaper to launch very large machinery and parts to the moon. Which we're working on, but we aren't quite there yet.
We've been to the moon. We have rocks from the moon. We've been doing science directly on rocks from the moon for almost fifty years. You know what we haven't been doing science on directly? Rocks from Mars. Rocks from asteroids. Comets. The best we have is remote-control robots on 2/3 of those. From a scientific standpoint, there is more to learn from going to Mars than the moon.
As I said above, we've been to the moon. It's not new anymore. It's not exciting. Andy Weir summed it up pretty well in The Martian (paraphrased because I don't have the book in front of me):
When the crew of the Ares I returned to Earth, they were greeted with parades all around the planet. When the crew of Ares II did the same thing four years later, they were greeted with a firm handshake and a hot cup of coffee.
We like new things more than we like things that have already been done. So, frankly, people aren't as likely to support another moon mission as they are to support a Mars mission. Mars is new. Mars is cool. The moon is sooo 1969. The benefits to society would be there for both, obviously, but it's easier to sell something new and exciting.
So How Do We Deal With It?
Frankly, we already are. The simple fact that SpaceX and such private space companies exist is good news for interplanetary space travel, and the way you worded it in your question makes me think that you might have a major misunderstanding of what's going on. SpaceX is currently doing Low Earth Orbit launches; they're currently able to run resupply missions to the ISS. They've been in the news a lot more than NASA, that's very true. But SpaceX is currently stuck in Low Earth Orbit — if LEO is like running a 5K, getting to the moon is running a marathon, and getting to Mars is an ultra-marathon. SpaceX will not get us to Mars.
But the fact that they exist, that they can do resupply missions, means that NASA doesn't have to worry about resupply missions and can instead focus on longer-term plans.
NASA doesn't have to focus on resupply missions every few months, so they can devote more time, money, etc to development of new programs that can actually get to Mars or beyond, like Orion. It's just that the general public doesn't really care about the many, many phases of testing that a spacecraft has to go through before it can be declared ready for launch, so they don't get as much coverage.
Add in the fact that that lack of coverage tends to lead to a cycle of budget cuts ("What have you done to deserve all this money? Here's a 3% cut!" "Okay, so now we have less money to work with… We're going to have to push testing back to spread over more time to deal with that." "Well you keep pushing back the delivery date, so this part of the budget looks like a nice place to cut back a bit more….") and you end up with programs that end up taking a long time to come to fruition. Budgetary constraints are huge for an organization like NASA, but infinite money won't solve problems instantaneously.
TLDR, it's a combination of everything that leads to a lunar colony being unlikely any time soon. But we have our eyes on the rest of space, and we are actively working on getting there.