I need to determine the torque that should be provided by the RCS thrusters for a slew maneuver of a 3axis-stabilized spacecraft (say 50x50x60cm and 50kg). Too much torque can cause unwanted structural vibrations and too little results in a very high thruster activity. Is there a method or systematic/analytical approach to determine the torque output required by the Reaction Control Subsystem?
If you are asking about how to determine the torque output from a given system with reaction wheel and/or thrusters, then you need to start with the specifications of those devices. Reactions wheels will generate torque only along their specific axis of rotation, otherwise you can simply look up what the rated and maximum torques are for that wheel. Thrusters generate torque based on their offset distance away from the vehicle centre of mass. To compute the torque you need to look up the rated thrust for that thruster then use the cross-product of the thrust vector and the position vector from the centre of mass to the thruster position. You can then combine all of the torques from each device to determine the next torque in all three directions.
On the other hand, if you are asking about how to determine how much torque you need, then you will have to follow Deer Hunter's comment and break down your problem some more so that you can determine what motions are needed and then convert that to torque requirements based on angular acceleration. For example, you could assume you'll be slewing from one orientation to another using a bang-bang maneouvre (apply some constant torque to start moving, then do nothing while moving, then apply some constant torque to stop moving). Then you can define how quickly you want to accelerate/decelerate and determine how much torque is required.