Any mass added to the vehicle "hinders" a ion engine, in the sense that it reduces the acceleration for a given thrust.
Ion engine thrust is proportional to power, more or less, the goal is to produce as much power per unit mass as possible. An suitably shaped mirror can focus the sun to make an image on a photocell array with the same angular diameter as the Sun has. Using a very curved sail, we could hope to get a focal length close to the radius, So close to Earth, where the Sun is about 10 milliradians across, we could hope to get an image diameter about 1/100 the radius of the sail which would receive about $10 MW/m^2$ of incoming light (basically the mirror is concentrating the light by a linear factor of 100, so 10000 times in area.
So in principle (ignoring many engineering issues) a 100m diameter sail and a 1m diameter 30% efficient photocell array could produce 3MW of electrical power (it also has to get rid of 7MW of waste heat, which will take some engineering).
Continuing our "in principle" calculation, the sail might be made of 50nm thick aluminium, for a mass of about 10^4 kg/m^2 or roughly 1kg for the whole sail. The photoarray and its associated cooling system sure masses much more. Let's be widly optimistic and say 10kg. This would power 12 DS4G ion thrusters with a total mass of 60kg and total thrust of 30N, so the whole system (with no payload or Xenon reserve) could accelerate at a very creditable $0.45 ms^-2$ almost 1/20 of a gravity.
So, what does this tell is in answer to the question. Basically the ion thrusters (that particular model, anyway) consume about 50 kW per kilogram of thruster. With 30% efficient solar cells, this can be harvested from about $150 m^2$ of light-collecting surface (mirror or solar cell). So scale doesn't really play a role. If you can make your light collecting and electricity producing system mass much less than 1kg/50kW then the limiting factor is the capability of the ion thrusters. If you can't then that mass is a major limitation, but it's not more or less of limitation for a bigger or smaller system