From On the Shoulders of Giants, ch. 14:
Schneider called James McDivitt and Scott, who were in Los Angeles, and asked them to go to the Douglas plant and look at a duplicate target vehicle shroud to see if the wires could be cut or the shroud removed in any way during orbital flight. The astronauts soon reported that the wires could be clipped, but there were many sharp edges that might tear the astronaut's suit as he worked.
So a) it sounds like there were indeed significant safety risks, and b) they were assessed as dangerous by about the most experienced people available - McDivitt had flown on Gemini 4 and supported White's spacewalk, and Scott had trained to do his own spacewalk on Gemini 8 before it was cut short.
In addition, from a risk-aversion perspective, it's important to remember that this was only the second ever American spacewalk - and White's one a year before had been very challenging. So there's already a limit to how much you would want to push things and add complex new unplanned tasks, even without the danger of those tasks injuring your crew!
There is also the issue of familiarity. The Gemini 9 mission had been planned to rendezvous and dock with an Agena. This was changed at short notice (after the Agena launch failed) to be an ADTA docking adaptor. This was not the same hardware, and probably (though I haven't been able to confirm exactly) would not have had any of the EVA equipment that the Agena did. It seems unlikely they planned to use it in the EVA, even in normal circumstances, so the underlying hardware would also have been unfamiliar.