According to the standard, the power source shall detect if a device is PoE capable and only then apply the voltage, so no raspberry pis can not be damaged IF everything works according to spec.
I've heard about power sources from major manufacturers not working according to the spec, so while unlikely, it's still possible, especially with older, more unusual equipment.
On initial power up Power Sourcing Equipment is designed to detect, and only supply power to the network device if it is identified as a PoE-enabled device. When first connected the PSE initiates a detection process that involves low probing voltages to sense the type of PD connected while avoiding damage to non-PoE network devices.
Source: http://www.bb-elec.com/Learning-Center/All-White-Papers/Ethernet/Power-over-Ethernet-PoE.aspx