Does the gravity assist get affected by any of the forces by other planets as they proceed towards the required orbit? Moreover what can be the other setbacks of gravity assist flyby?
Arthur Dent caught the basic problem with gravity assists - you can't decide where and when you can use them - you must plan your mission around opportunities for gravity assists, and that means a very complex and often very long trajectory. It takes a lot of computational power to plan. It sets a rigid launch window; the planets won't wait for new budget for your probe or time you need for extra testing. The time spent maintaining the probe from the ground station while waiting for the assists is not free - the ground station and crew need to be funded. With each day in space the risk that something fails on the craft increases. And botching an assist will send your probe into a trajectory which is very unlikely to give you another chance to encounter the target.
You gain "free" delta-V. That's one huge advantage. But the two primary disadvantages of rigid mission plan, and increased mission duration lead to so many small secondary disadvantages many missions are designed with no gravity assists simply to save the headaches.
Referring to the first (and only) answer to this question so far by SF, it's not quite correct that increased mission duration is a disadvantage per se, because for most missions that have used gravity assist the gravity-assist technique enabled a mission profile with much shorter trip times than conventional methods.
Case in point: using the most efficient conventional orbital transfer technique known — the Hohmann Transfer — to travel from Earth to Neptune would take 40 years. Using gravity assists at Jupiter, Saturn and Uranus, Voyager 2 arrived at Neptune 12 years after launch. The more recent New Horizons mission to Pluto combined a gravity assist at Jupiter with an oversized launch rocket (Atlas 5) and got to Pluto in about 9 years. (Pluto is about the same distance from the Sun as Neptune.)
True, you have to launch during windows when the planets are aligned to meet mission needs, but in most cases the trip times are much shorter than using conventional methods, and in some cases the mission is simply impossible or extremely difficult without using gravity assist.
Until very large, oversized rockets (e.g., BFR) and/or more exotic in-space propulsion systems (e.g., nuclear thermal) are available gravity assist will be the most efficient way in terms of trip time to bounce among our planets when Hohmann-like transfers won't do. Current ion propulsion systems are very efficient propellant-wise, but mission trip times are typically much longer than gravity-assist missions for the same start/end locations.