Firstly, "rocket science" is a bit of a misleading term. Rockets are a type of propulsion system performing a function within larger systems such as launch vehicles, spacecraft, missile systems, etc. Engineering these complex systems is a holistic, multidisciplinary process. There's an entire discipline dedicated to this type of engineering originating from the 1930s: Systems Engineering.
Systems Engineering and modern software development practices are similar, despite their somewhat independent evolution. Design process models familiar to the IT and software engineering world such as Waterfall and Vee have played their roles, and approaches like Concurrent Engineering have more than a bit in common with Agile software development practices and are intensively iterative. See ESA's description of Concurrent Engineering as an example.
The specific nature of iteration over a system's life cycle is going to depend on the project, who's doing it, who's paying for it and what the life cycle stage is, but the "design, test, fix" paradigm isn't really much different (though, as implied, the scale might be).
Mathematical models, simulation and design patterns are important in keeping costs down, particularly in early design phases. Components and subsystems can be developed and verified against specification, with integration being tested through ground tests as covered in other answers. In some regards it could be argued the physical nature of components makes it easier to quantify and verify against specification than the abstract nature of software [systems] but -- as pointed out -- there are additional difficulties and costs associated with engineering in the physical domain.
Software engineering is a comparatively young discipline and there is a significant degree of convergence occurring as it matures (and software systems get more complex), but engineering is engineering.
Suggestions for further reading: