The most basic answer is that pressure gives you velocity, and velocity gives you energy.
A rocket engine becomes more efficient the faster the particles making up the thrust gases are leaving the rocket through the nozzle. Simple projectile physics:
$$E = \dfrac{1}{2}mv^2$$
The energy of a mass, such as a stream of rocket exhaust (and, by Newton's third law, the rocket and whatever's strapped to it), is one half the product of the mass of that exhaust times the square of its velocity. If you need double the energy per second in a rocket (to accelerate faster), you can either double the rate of mass you're ejecting from the nozzle by doubling your fuel consumption, or you can increase the velocity of the same mass being ejected by $\sqrt{2} \approx 1.414$. By increasing the pressure inside the ignition chamber, you will increase the velocity of gases escaping it through the nozzle.
Assuming the fluid is incompressible and frictionless (it isn't, but we can make rough calculations under these assumptions), Bernoulli's Equation will give us the velocity of the rocket exhaust, if we know the pressure and density of the gas in the chamber and the pressure and density of the environment outside the nozzle:
$$\dfrac{P_a}{\gamma} + \dfrac{v_a^2}{2g} + z_a = \dfrac{P_b}{\gamma} + \dfrac{v_b^2}{2g} + z_b$$
where, for the subscripts $a$ and $b$ corresponding to the environment inside and outside the ignition chamber respectively, $P$ is the pressure in Pascals (N/m2), $\gamma$ is the specific gravity (relative density to that of water) of the fluid, $g$ is the acceleration of gravity (we can effectively ignore it), $v$ is the velocity in meters per second, and $z$ is the relative elevation of the two fluids. In our circumstance, internal pressure and external velocity dominate the equation, to the point where we don't care much about internal velocity (effectively zero), external pressure (effectively zero), gravity or relative elevation, and so the equation simplifies to:
$$\dfrac{P_a}{\gamma} = \dfrac{v_b^2}{2}$$
This equation rearranges very similarly to the energy equation - $P_a = \dfrac{1}{2}\gamma v_b^2$ - and for similar reasons (A pressure differential is a form of potential energy). We see from this equation that the higher the pressure, or the less dense the gas, the higher the velocity. Because lowering the density would lower mass, thus lowering energy of the moving gas, increasing pressure is the way to go.
And how do you increase pressure? well, according to the ideal gas law:
$$P = \dfrac{nRT}{V}$$
where $P$ is pressure in Pascals, $n$ is number of moles of gas, $R$ is the gas constant for the units of measure we're using (for SI units it's 8.3144621 J/(mol*K)), $T$ is temperature in degrees Kelvin and $V$ is volume in cubic meters. Assuming a constant volume (the "combusion chamber" of an SRB is in fact always increasing as fuel burns away to create a larger cavity inside the casing, but most liquid rockets have a fixed-volume combustion chamber), increasing either the amount of gas or its temperature are your options. Again, this mainly comes down to either reacting more fuel to produce more gas, or increasing the temperature at which the fuel burns (and better insulating the chamber to prevent heat loss).
Given a specific design and a specific fuel, the thrust energy of the rocket is dictated by your fuel flow rate. By the same token, given a specific fuel and a desired thrust energy, the required fuel flow rate is dictated by your design, so you'll design for the highest velocity exhaust, and that means the highest pressure you can get with the strength of the materials you're using (which you choose for the best tradeoff of high strength to low weight).