I worked directly on several university cubesat missions that ran linux in a semi-custom main flight computer, as well as more traditional embedded software on small processors such as the MSP-430. I don't want to say which university I supported because I no longer work there and I don't know how open they are currently about their system design.
Through interacting with other universities and small businesses I know that there are several fully COTS solutions available that are capable of running linux. Here and here are the two most comprehensive lists I could find of suppliers marketing towards micro/small sat missions. There is also a wide selection of products that not satellite oriented that are equally suitable for running linux and commanding a satellite (such as the Intel Atom, Raspberry Pi, and other computers designed for used in more typical industrial applications).
I can't really address the specifics of hardware selection as it is so tightly coupled with other missions design concerns. What I can say is that running linux has the potential to simplify the software design (increased clock speed, cashe, memory, higher level interfaces like USB, etc.) and shift complexity into interfacing with the hardware and designing for reliability (timing accuracy, linux complexities, multiple threads, etc.). Look for a system that is physically designed to be fault tolerant, or where it can be added on in another way (internal & external watchdog timers).