Have the rovers on Mars encountered anything unrecognizable to geologists?
Yes, sort-of: These are similar to structures on Earth that are produced by life, so they have people scratching their heads!
See my Earthscience SE question What are Ordovican trace fossils, and what do they look like? (I've just added a bounty there.)
There are some unusual structures in recent images from the Mars Curiosity Rover's MAHLI (Mars Hand Lens Imager) shown in the Space.com article Curiosity Rover Spots Weird Tube-Like Structures on Mars.
An astrobiologist (not at NASA) is quoted as saying that they looked "...remarkably similar to Ordovician trace fossils..." while others are quoted as suggesting that these are visually reminiscent of bioturbation processes on Earth.
below: NASA's Mars rover Curiosity captured this image on Jan. 2, 2018, with its Mars Hand Lens Imager (MAHLI). Using an onboard focusing process, the robot created this product by merging two to eight images previously taken by MAHLI, which is located on the turret at the end of the rover's robotic arm. Credit: NASA/JPL-Caltech/MSSS
Cropped section are also included for better viewing, in the first one, sunlight can be seen below the structure near the bottom, showing that it extends somewhat upward from the rock below it.
below: Curiosity Mastcam right image taken on Dec. 15, 2017. Credit: NASA/JPL-Caltech/MSSS