The general shape of fixed nozzles is pretty close to a solved problem. Per Huzel, even a conical nozzle of sufficient length and 15º half-angle gives better than 95% efficiency. Bell nozzles can exceed 99% efficiency in much shorter form.
The actual engineering design of a nozzle for a large liquid-fueled rocket is more complex; cooling the nozzle becomes a large challenge, and in regeneratively cooled engines, it interacts with the propellant pump system.
In solid rockets, regenerative cooling isn't an option, so I believe ablative cooling is more common, helped by the fact that burn times are often shorter -- typically around 2 minutes instead of 4-6 minutes for liquid first stages. Substantial engineering issues involved in interfacing the casing to vectored nozzles, but again, shape-wise, it's generally a near-optimal bell nozzle.
Either the Huzel or Sutton book will take you through the next couple of levels of complexity on the topic.