As suggested by OrganicMarble in a comment, nitrogen is miscible with oxygen (you can thus make liquid air). According to NASA Technical Paper 2464, this is a major concern because using "enriched air" instead of pure oxygen as the oxidizer degrades the performance of the engine:
The transfer of liquid oxygen (LOX) from a storage vessel to a rocket engine generally requires the use of a pressurizing gas at high pressures. The primary criteria for the choice of gas are low cost, safety, and immiscibility with liquid oxygen. Among the common gases, helium, nitrogen, and oxygen itself
have been considered. Helium is expensive, and oxygen is hazardous at high pressures. The remaining gas, nitrogen, unfortunately is miscible with oxygen
and causes dilution and loss of engine performance. (emphasis mine)