The SRB's, made of much heavier material than the Falcon 9 first stage, would hit the water, tail end first, with the heavy motor casing at the bottom trapping air inside it.
It should be noted, the SRB's did NOT impact gently, they hit at terminal velocity under their parachutes, and usually would bend the SRB segements some amount out of round that had to be fixed in the factory.
This air inside would suffice to keep it floating mostly vertical until the recovery ships could find it. Only about 20 feet or so stuck up out of the water.
To recover it, they would have divers swim down with an inflatable plug, to close the hole the engine makes, and pump air into it until it was floating on it's side, then they could tow it back to Port Canaveral for processing.
You can watch this in a great video SRB Recovery
The Falcon 9 first stage is much lighter and less structurally sound. Additionally, it would come to a 'stop' relative to the surface and then fall down, somehow.
In the soft landing attempts SpaceX made before they had an ASDS they reported that the stage survived landing and was destroyed by wave action once on the surface, which indicates it likely was floating.
It does seem that the fall from standing tall, on the surface, down to the surface likely did much of the damage as well.