Side note: The US does not use imperial units. There are subtle differences between imperial and US customary units.
I have seen multiple claims that von Braun despised US customary units. But he pretty much had to live with those US customary units, at least as far as hardware was concerned. The same pressures that made von Braun hate but be stuck with US customary units exist to this very day. Elon Musk also apparently despises US customary units. Despite this dislike, there are some aspects of the rocketry at SpaceX that still use US customary units.
For example, from the 1930s through the 1960s, there were very few, if any, US suppliers of space qualified metric fasteners. The European suppliers had been bombed out of existence by World War II. To this day, hardware developers, especially those who deal with "bending metal", fasteners, plumbing, structures, physical support systems, ..., tend to lean strongly toward US customary units. This has turned out to be a very tough nut to crack (pun intended). On the other hand, software developers, especially those who deal with physics-based software such as the on-board guidance, navigation, and control software and ground-based simulation software, tend to lean strongly toward the metric system.
Those on the physics side of things have tried to get NASA to finally go fully metric. This has not been not easy. Metric fasteners and customary fasteners, and the tools used for them, do not mix and match. The efforts to get NASA to go fully metric has resulted in blowback from the hardware developers, so much so that they tried to force the software developers to switch to US customary units. The end result is a detente.
This mishmash of units can lead to disasters such as the Mars Climate Orbiter and can lead to statements such as this from NASA's Space Launch System fact sheet:
Initial 70-metric-ton Rocket Development
The 70-metric-ton SLS will stand 321 feet tall, provide
8.4 million pounds of thrust at liftoff, weigh 5.5 million pounds and carry 154,000 pounds of payload.