I am attempting to calculate the oxygen to fuel ratio (O/F) for my hybrid rocket engine (paraffin wax and gaseous oxygen). I know I can use the oxidizer to fuel equation to calculate this, but my question is, why can I not just setup a complete combustion reaction of the paraffin wax and solve for the mass of oxygen? I know this does not work, but I have yet to understand why.
Rockets frequently (potentially even usually) are not operated stoichiometrically, as this tends not to be the most efficient operating point.
The most extreme example is that of the H2/O2 rocket, which is normally run very rich so that the exhaust contains a lot of unburned hydrogen -- this reduces the burning temperature and the very light hydrogen molecules generate thrust more effectively than having a bit more thermal energy in the jet.
Hydrocarbon-oxygen rockets are run closer to the stoichiometric ratio, but still generally somewhat rich, as this leads to one having lighter CO molecules in the exhaust rather than CO2.
However, you're talking about a hybrid, which is more complicated because you don't really control the rate at which fuel enters the reaction. This is determined by many factors including the temperature, heat transport dynamics in the fuel grain, disintegration mechanics of the fuel grain, flow rate and turbulence of the oxidizer and the fire region, and of course the surface area of the fuel grain (which will change as it burns for most fuel grain geometries).
Additionally, just because the fuel grain is eroding does not mean it is burning -- you might be spraying droplets of unburned melted fuel out the nozzle.
If you set it up with a specific amount of oxidizer and a specific amount of solid fuel, in stoichiometric ratio, it would be easy to either use up all the oxidizer without burning more than a little bit of the fuel, or to blow most of your fuel unburned and partially burned out the rocket.
As a result, it's significantly more complicated to control the stoichiometry of a hybrid motor, and what the stoichiometry actually is will be somewhat variable, empirical, and dependent on the exact details of your implementation. It's not just based on the ratio of mass (flow) to mass (flow). Clark's Ignition! details the history of difficult engineering needed to make the first successful hybrids (even though hybrids are much more sensible and achievable for amateurs than liquid bipropellants).