The only equation needed to produce these curves is the thrust equation in a couple of different guises.
- $$F = \dot{m}v_e + (p_e-p_{atm})A_e $$
- $$I_{sp}= F /(g_0 \dot{m})$$
You also need isentropic flow charts or tables to calculate the nozzle parameters that change with changing area ratio. An example isentropic flow chart is shown here, from 1953 book "The Dynamics and Thermodynamics of Compressible Fluid Flow" by Ascher Shapiro, page 87.

Since the X axis of the graph in the question is nozzle exit diameter, which sets nozzle exit plane area, and therefore the area ratio, I rearranged the Shapiro chart to show pressure ratio and velocity ratio plotted against area ratio. Both axes are dimensionless, area ratio is on the X axis. I started with an isentropic flow table spreadsheet from here. The spreadsheet didn't have velocity ratio in it, so I calculated it from pressure ratio using equation 3-26 in Sutton (4th edition).

Knowing the area ratio gives us these two ratios which give us all the information we need to calculate the exit plane parameters needed for the thrust equation - if we know the throat properties. We will need to make some assumptions about the engine for that.
Assumption 0 - The Raptor's throat area is 0.6 ft2. This comes from old information giving the exit diameter as 1.7 meters and the expansion ratio as 40. I have found no reference to the expansion ratio on the latest iteration of the engine, so I used this.
Assumption 1 - The exit plane pressure where the question graph of sea level $I_{sp}$ reaches a maximum is 15 psi. A sensible assumption because when the exit plane pressure matches ambient, the losses are minimized.
Assumption 2 - The Raptor's sea level thrust is 380,000 lbf (Wikipedia)
With these assumptions we can calculate the mass flowrate using equation 2. I read the maximum sea level $I_{sp}$ from the chart as 334, this gives a mass flowrate of 35.3 slugs/sec.
Using that mass flowrate we can use equation 1 to calculate the exhaust velocity. The delta pressure term vanishes when the exit plane pressure is 15, so the exhaust velocity is 10,750 ft/s.
With these assumptions we know the area ratio, exhaust velocity, and exit pressure at the peak of the sea level curve. This lets us calculate the throat properties using the isentropic tables and gives
$V_t$ = 4898 ft/s
$P_t$ = 640,900 lbf/ft2.
We can now calculate specific impulse for any nozzle exit diameter by the following process:
- Calculate exit plane area based on diameter; divide this by the
throat area to get the area ratio.
- Look up the pressure and velocity ratios in the isentropic tables
based on area ratio.
- Calculate the exit plane pressure using the pressure ratio and the
throat pressure.
- Calculate the exhaust velocity using the velocity ratio and the
throat velocity.
- With the mass flowrate, exhaust velocity, exit plane pressure, and exit plane area, use Equation 1 to calculate the thrust.
- Use Equation 2 to calculate the specific impulse.
You may certainly change the assumptions and get different values. However, this demonstrates that by holding those parameters constant and calculating the isentropic flow properties solely based on area ratio, you can generate curves that are shaped like the ones in the question.
This doesn't match the graph in the question exactly; I presume the maker of that graph had better/different info about the engine.

Re: the "effective ISP"
It's supposed to be the average ISP for a booster as it goes from
sea-level to main engine cutoff. I don't know enough about the flight
profile to really calculate the effective ISP so it's just the sea
level and vacuum ISP averaged together with the vacuum weighted as
twice as important.
Source- reddit thread in the question