I am working on the on-board computer for a CubeSat. Our computer will be vulnerable to radiation, hence single event upsets, e.g. bit flips are likely to occur. Would a lighter, smaller OS like FreeRTOS bring more stability and a lower probability of failure over a full-blown Linux operating system?
In general, you want as simple of an OS for any embedded systems application. You want as few features as you can get away with. You are less likely to run into problems that way, don't really need all of the features, and it'll reboot quicker, and use less memory.
Two things you want for sure. You want a realtime operating system, you probably want threads and similar protections, but you probably don't want device drivers, for instance, especially for GUIs. Rodos is one OS I found in my search that meets the criteria, I'm sure there are others. RTOS is a reasonable choice as well.
BTW, as a bonus tip, there are several architectures used to keep satellites safe in case of an emergency. See How does one build software for a Satellite such that a new build doesn't break it? , for instance.
I don't know about the probability, but the RTOS will reboot a lot faster. When you run into a problem, and you will run into a problem, you reboot, and you want that to be fast.
I do know that FreeRTOS, on an Pumpkin MSP430-F216 microprocessor platform, has been working successfully on the NASA Firefly Cubesat project that I wrote the flight and ground software for, and has been communicating for over 2 years.