I can imagine a satellite with a solar sail it rolls when on the night side of Earth, then unrolls near "afternoon", propelled on each past of its "sunset side" of travel. Would such system be a viable method of keeping the satellite in orbit indefinitely (or until actuators fail) or would some other factors (e.g. atmospheric drag) make it impossible?
First, there are several forces acting on a sailcraft.
- Aerodynamic drag
- Solar radiation pressure
- Gravity field non-sphericity
- Electrodynamic drag
If you go anywhere lower than 770 km, atmospheric drag is by far the largest force that pulls your apoapsis down.
Quoting Vulpetti (2008):
When unfurled in LEO, the drag on the sail produced by its flight through this residual atmosphere can be very high; larger in magnitude than the thrust the sail experiences by reflecting sunlight. Simply put, a sail flown in LEO will very quickly lose energy by interacting with the ionosphere (despite the fact that it is getting accelerated by reflected sunlight), and find itself on a reentry trajectory. It is easy to compute that a sail shall operate beyond 700 km (nominally); if one takes the upper-atmosphere changes into account, the previous lower limit increases to 750 to 770 km.
Second, deploying a sail is a risky and not fully understood maneuver. You don't want to stow and deploy it repeatedly.
Third, when there's no solar pressure, you don't need to stow the sail. Provided you operate beyond the 770 km limit, the only thing you have to do is rotate the sailcraft. Now, ensuring necessary rotation rates (attitude control authority) is a problem if you take into account that the whole sailcraft must be very light to rely on solar radiation pressure.
Between atmospheric drag out through almost 1000km, there's the issue that solar light and solar wind are both cut by the passing into the shadow of a planet.
Since this puts more of the energy on the solar side, even for high orbits, the sail is, if passively deployed, making the orbit more ellipsoid and thrusting it planetward on the sunward lobe.
Let's assume a flat sail perpendicular to wind: as you come sunward, it slows you, lowering your periapsis by both slowing you down and by direct thrust, and increasing your apoapsis slightly, but less, due to the shadow.
Now, we tilt it, for outward tack on the inbound: you still slow, widening and shallowing the inbound arc. On the outbound, it's pulling you towards the planet, narrowing it and accelerating you back slightly. At the chunk closest to the sun, it's accelerating you counter-orbitally; that's slowing your orbit, and lowering your periapsis. At away-from sun, it's getting increased speed except in shadow - so it doesn't equal the amount from them sunward arc. This skews the orbit's long axis, but still lowers periapsis.
Tilting the other way drives the approach downward and anti-spinward, lowering periapsis more radically, and the sunward side and half the retreat is acceleration; the away-from sun is slowing but interrupted, slightly increasing apoapsis. It also skews the long axis of the ellipse of the orbit.
Now, even if it's tacked properly (inbound arm outward, ourbound arm outward, solar-side arm spinward) it's still consistently pushing towards the planet. It might be able, if the orbit is high enough, to push the apoapsis up and maintain enough acceleration to overcome the outward force of solar energies.
As a passive system, it's going to eventually push you into atmosphere, and at that point, it is a liability.
As an active system, it can theoretically be used to increase orbital velocity for the majority of the orbit, and slowly accelerate one outward. This could even — in theory — be used to accelerate to breakaway, provided one starts in an orbit well above 1000km, and can tack the sail fast enough to not be a problem on the sail-turn at the sunward extreme.