Not much savings would be had by replacing the engine. In general, rocket engines are a surprisingly small piece of the spacecraft. In this specific case, the SPS engine makes up about 0.7% of the total mass of the Apollo CSM/LM stack at Earth departure. Fuel tankage, instead, dominates the mass of a ship that does significant maneuvering.
According to this NASA overview document, the weight of the engine including the 2.5 m nozzle extension is about 300 kg (650 lbs). The earlier AJ10-118 variants were around 90–100 kg with nozzles ranging from 0.84 m to 1.4 m in diameter.
Simply providing the same engine with a smaller, lighter nozzle would reduce thrust, but would also reduce specific impulse (fuel efficiency), so any mass savings would be obliterated by the need to carry more fuel. The AJ10-118E with a 1.4 m nozzle provided 278 s of specific impulse, 13% less than Apollo's 314 s; something like 3 more tons of fuel would have been needed.
An all new engine could have been specified for the project. Masses of hypergolic pressure-fed engines tend to be proportional to (very roughly) the 0.6 power of thrust, so you'd expect an engine of half the thrust and same specific impulse to mass about 200 kg. That reduction in total mass would also allow the fuel load to be reduced proportionally, saving another 50kg or so. In principle, then, a 150 kg savings should have been relatively easy to achieve with a scaled down engine, while still preserving the efficiency. Reducing the thrust further might start to make the lunar orbit insertion and trans-Earth injection burns significantly less efficient; the Oberth effect peaks with very short impulses.
However, the AJ10 family was (and still is) one of the most mature rocket engine systems in existence, with about a decade of flight experience behind it at the time of the Apollo missions. Its specific impulse with the big Apollo nozzle was already about as good as a pressure fed hypergolic can get. There was no other appropriate engine available at the time having anywhere near the performance and proven reliability of the AJ10.
The Apollo LM's ascent engine is an interesting point of contrast. It was developed specifically for the project. At 1/6 the thrust of the SPS, it masses 82kg. Its specific impulse is only slightly lower than that of the SPS. Its development was badly delayed by problems with the injector design, endangering the project schedule; Aerojet had to step in to replace the injector when Bell was unable to develop a stable one. The ascent engine used an ablative nozzle that wouldn't have stood up to the repeated firings required to maneuver the CSM.
Saving 150 kg on the service module would have been nice, but there were other ways to save mass that didn't risk compromising reliability.
The AJ10 would go on to become the heart of the space shuttle's Orbital Maneuvering System, racking up hundreds of flights, thousands of starts, and no failures. Fifty years after Apollo, the Orion spacecraft is supposed to use surplus OMS engines from the shuttle program.
