According to this article, those are insulation panels, and they're simply lightweight enough to not damage the rocket at the relatively low speeds it's moving in the initial seconds of flight.
The insulation keeps the upper-stage liquid propellants from boiling or freezing as the ambient temperature changes -- the oxidizer, N2O4, in particular, is liquid between -11C and 21C at ambient pressure, although it's pressurized in the tank, so boiling is limited. The UDMH fuel remains liquid over a much wider range.
In some other rocket launches, you'll see more irregular pieces falling off at launch; those are ice chunks, condensed and frozen from water vapor in the air against cryogenic propellant tanks (liquid oxygen, liquid hydrogen, etc). PSLV, however, uses non-cryogenic hypergolic propellants, so it doesn't condense ice.