These are vernier thrusters! They're small (relative to the main engine) rocket engines used to finely control the heading and alignment and form part of the Mercury-Atlas' attitude & roll control system. These days, systems tend to use gimbaling of the main motor to maintain attitude control; but if you couldn't gimbal, introducing changes in the spacecraft's or rocket's attitude by pushing the vehicle with smaller rockets is preferable.
Specifically, the vernier engines are LR-101's developed by Aerojet Rocketdyne and certified for use in 1958:
The LR-101 was a single-start, fixed-thrust engine with an expansion ration of 6:1. In its Block 2 version, ignition occurred by means of pyrophoric (hypergolic) fluid (a mixture of tri-ethyl aluminum and tri-ethyl boron) which ignites spontaneously in the presence of oxygen.
http://heroicrelics.org has some additional specific information about the engine, but its variants usually produced between 528 and 1000 lbs of thrust each. Mercury-Atlas had 2 of these at 180 degree intervals around the booster.