There are several perspectives to look at this from, and a good number of different assumptions to consider -- some of which are more realistic than others, and some of which are heavily influenced by your mission design and the economic and organizational / operational setting the mission is designed in -- a realistic science-fiction-style space opera tramp freighter has very different concerns from a modern day communications satellite, which has different concerns from an expendable Mars probe, which has different concerns from an expendable probe meant to travel to the outer planets in a short period of time, which has different concerns from a minimal mass Kuiper Belt or interstellar probe, which has different concerns from a "space tug" or similar vehicle.
Several principles are dictated by the mathematics of rocketry:
Based on the Rocket Equation, required mass ratios are small when delta-V is significantly less than V_e, reasonable (about 2 to 4) when delta-V is equal or a bit more than V_e, and impractically large when delta-V is larger than V_e, necessitating staging.
Achievable delta-V varies linearly with Isp / V_e, while thrust power (which is both the lower limit on energy needed to drive the engine and also a good basis for heat dissipation and energy-handling ability needed) varies as the product of thrust and the square of V_e.
Both propellant and power sources have mass and cost money, and so does fuel for generating power if your power source uses fuel (such as uranium fuel rods for a fission reactor, or deuterium for a fusion reactor.)
Propellant probably requires tanks which have at least some mass.
There's a limit to how much power you can get out of a given power source per unit mass. (for open-cycle devices like chemical rockets and nuclear thermal rockets, this limit can be incredibly high). In space, this also tends to be limited by heat radiators. Very often this limit is nowhere near high enough to drive the engine you would like. For some reason, practical solar panels and practical nuclear reactors both seem to be on the rough order of 10-20 kg per kilowatt.
Our perceptions of the mathematics tend to be a bit skewed by the very low V_e of chemical rockets and also the fact that chemical rockets combine the power source and the propellant intrinsically. However, it's easy to ignore the need for a power source when considering more advanced propulsion methods such as ion engines.
There are two basic ways that an Isp can be too high:
It wastes energy that you could have saved at the cost of some more propellant.
(relatedly) it requires too much energy to be a good fit for available power sources.
(also relatedly) To get a level of thrust appropriate to the mission, an infeasible amount of power density is needed.
Let's look at a notional ion-drive nuclear-powered ship with variable Isp, and a 10-ton engine (not including power reactor) which can process 10 MW of thrust power at 75 percent efficiency.
Let's also look at the reactor: if we go with 10 kg/kW, this reactor will weigh about 133 tons. Let's say it's 50 percent efficient.
Let's give the ship 10 tons of structure and tankage (probably optimistic), 20 tons of payload, and 519 tons of propellant, rounding it out with a wet mass of 692 tons and a dry mass of 173 tons.
Let's look at this operating at an Isp of 30,000 S (V_e of 294300 m/s):
Thrust: 68 N
Initial Accel: 9.82E-5 m/s^2
Delta-V: 408 km/s
Time to accelerate that delta-V: 71 years
Time to accelerate 5 km/s: 589 days
Propellant expended for 5 km/s: 12 tons
Energy used for 5 km/s: 6.9E14 J
Amount of uranium fuel burned for 5 km/s: 16.6 kg
That's a lot of delta-V, but it takes forever just to use a small part of it such as one would use to make a cislunar or interplanetary transfer that shouldn't even take more than 2 years! Meanwhile the vast majority of this ship's dry mass is a huge nuclear reactor -- and it burns over 16 kg of uranium fuel.
This ship makes sense if you're doing a very high delta-V, very long duration mission, probably into the Kuiper Belt, or a series of outer-planet transfers that take years at a time, without any ISRU.
If you had a reactor that was vastly lighter for the power produced, you could make this ship much faster -- but there's only so far you can do that until you reach torchship territory where you are struggling hard against the ability to handle absurd amounts of thrust power.
Let's look at the same ship operating with an Isp of only 3000 s (29430 m/s), with the same thrust power and a much higher propellant consumption:
Thrust: 680 N
Initial Accel: 9.82E-4 m/s^2
Delta-V: 40.8 km/s
Time to accelerate that delta-V: 260 days
Time to accelerate 5 km/s: 54 days
Propellant expended for 5 km/s: 108 tons
Energy used for 5 km/s: 6.24E13 J
Amount of uranium fuel burned for 5 km/s: 1.5 kg
That's very different: By reducing the Isp by a factor of ten, you get a ship that can instead make a couple large interplanetary transfers without ISRU and with burns measured in weeks rather than months, and also use a factor of 10 less expensive uranium fuel for the initial 5000 m/s burn. The nuclear reactor is still very large and still the majority of the dry mass of the ship (and this is a somewhat optimistic specific power).
This isn't universally applicable -- consider that if you only need 5000 m/s and the ship can be expendable or have a high mass ratio and arrive empty, you can make this much faster and cheaper with chemical fuel. Meanwhile, if you have access to a very high specific power source with cheap or free fuel (possibly beamed power?) you might as well just use high Isp and avoid having a high mass ratio. (Remember, however, that a present-day jet airliner has a mass ratio around 2, so moderately high mass ratios don't necessarily mean flying fuel tanks). If you're doing something like transport ISRU-sourced fuel over a long delta-V chain to get it in the place needed and can take your time, you might want very high Isp ion thrusters so as to avoid a huge multiplier of the amount of fuel needed at the source. However, this should provide a pretty clear example of a case where an Isp can be much too high.