How can I solve for the amount of radiation that hits a Mars transit vehicle, in transit? What formulas should I use? I am mostly focused on solar radiation.
It depends when the journey is made and many other things.
High energy solar protons are more of a problem during the solar maxima of the 11 year solar cycle and less so during the solar minima. The flux of high energy particles also varies over much shorter periods during solar storms which themselves produce variable levels of radiation. In fact solar storms produce a wide (and variable) spectrum of energetic protons. Solar storms also follow magnetic field lines and are directional. So this is not an easy question.
To add further complication if you are interested in radiation as a hazard for crews in deep space, Galactic Cosmic Radiation (GCR) also varies over the 11 year solar cycle being at a maximum during solar minima and at a minimum during solar maxima. It also comes in a spectrum of different particle types and energies.
High energy solar radiation tends to be easier to shield from than GCR, (although easier does not imply easy). I doubt that there is a simple formula that would provide what you want. There are too many variables in terms of timing, random effects and multiplicity and variability of radiation types.
The radiation situation in interplanetary space and on the surface of Mars is significantly different from the situation in near-Earth orbits. It is more dangerous and under certain conditions (with large solar flares) can pose a real threat to the life and health of astronauts. Radiation conditions in the Martian expedition will be mainly determined by galactic and solar cosmic rays, as well as secondary neutron radiation arising on the surface of Mars and during the interaction of cosmic radiation with the materials of the ship and the radiation shelter. The total doses that the cosmonauts will receive during the expedition may turn out to be about 4-10 times higher than in long-duration orbital flights (the dose that cosmonaut V.V. Polyakov received during the 14 months of the flight was 14 Centisievert (cSv)), but they should not exceed the maximum allowable exposure dose standards, except in cases of major solar events.