I have knowledge about basic control systems, mathematical modelling and deriving state space and transfer functions. I'm a part of a team designing a small satellite. Control will be achieved using RCS thrusters and reaction wheels. How do I design an attitude control system for such a satellite? References to lectures and/or books would be helpful. To be more specific, I'd like to learn more about the mathematical and analytical part, where the controller algorithm is designed, analysed and tested.
Take a look at Space Mission Analysis and Design by Larson and Wertz, it's standard literature on systems engineering. Additionally Spacecraft Attitude Determination and Control by Wertz will give you more information on the design of an ACS.