One big factor is Earth's gravity.
To reach space from Earth, any spacecraft needs to be huge. It requires a huge crawler and good crawlerway. It requires a solid, big launchpad to support the weight of the craft and infrastructure to keep it upright, fueling it with hundreds or thousands of tons of fuel, keeping it powered, and safe against the environment (including undisciplined bystanders). The mighty engines require flame trenches to contain the blast of the exhaust and protect the craft from it. There's a big water tower and a massive sprinkling (more like flooding) system to extinguish the sound waves from the launch which are enough to destroy the craft.
This all weighs a lot and needs structural infrastructure to support it and keep it from simply sinking into the soil or breaking under own weight. That's why Earth spaceports are so huge.
On the Moon, the delta-V needed to reach Moon orbit is way, way smaller, and taking the Rocket Equation into account, the needed size of the craft is exponentially smaller. Smaller engines, as not nearly as much thrust is needed. Way smaller tanks. Supply pipes and cables that will not snap under own weight. That means far less - exponentially less - infrastructure needed to support the launch. Your craft will be maybe 20 ton instead of 20,000, for similar payload to the orbit. Even completely independently of the same structural construction being able to support six times the mass it could support on Earth, simply due to everything being lighter!
Mars lies somewhere in the middle mass-wise, but again, the rocket equation with its exponent is on our side, reducing the mass exponentially in relation to the gravity. You might need more infrastucture, but really, a simple slab of concrete (or something similar) will suffice where the launchpad with flame trenches and water dump system was required.