There are many cameras currently orbiting Mars, taking pictures of rocks and sand and things.
What is the highest (in the visible spectrum) sensor resolution of these? Are there higher res ones planned?
Mars Reconnaissance Orbiter has a camera called HiRISE (High Resolution Imaging Science Experiment), launched in 2005-08-12, which is the highest resolution camera in orbit of Mars at an altitude that varies from 200 to 400 kilometers (about 125 to 250 miles) above the martian surface by Carrier rocket Atlas V-401.
The High Resolution Imaging Science Experiment camera is a 0.5 m reflecting telescope, the largest ever carried on a deep space mission, and has a resolution of 1 microradian (μrad), or 0.3 m from an altitude of 300 km. In comparison, satellite images of Earth are generally available with a resolution of 0.5 m, and satellite images on Google Maps are available to 1 m.
HiRISE collects images in three color bands, 400 to 600 nm (blue-green or B-G), 550 to 850 nm (red) and 800 to 1,000 nm (near infrared or NIR). Red color images are 20,264 pixels across (6 km wide), and B-G and NIR are 4,048 pixels across (1.2 km wide). HiRISE's onboard computer reads these lines in time with the orbiter's ground speed, and images are potentially unlimited in length. Practically however, their length is limited by the computer's 28 Gigabit (Gb) memory capacity, and the nominal maximum size is 20,000 × 40,000 pixels (800 megapixels) and 4,000 × 40,000 pixels (160 megapixels) for B-G and NIR images.
Each 16.4 Gb image is compressed to 5 Gb before transmission and release to the general public on the HiRISE website in JPEG 2000 format.To facilitate the mapping of potential landing sites, HiRISE can produce stereo pairs of images from which topography can be calculated to an accuracy of 0.25m.
To answer the second part of your question, the highest resolution camera proposed in Mars orbit would be one built off of one of the two NRO donated spy satellite frames. As of yet, it has not been determined what the second of these will be used for. The Mars proposal, called MOST, would allow for a camera with higher spatial resolution then the HiRISE camera, and the proposal was put forward by the same Principal Investigator as HiRISE, Alfred McEwen.
The truth is, however, that we very rarely need higher resolution photos then HiRISE can provide, and in fact it is quite common to bin the HiRISE images to effectively lower it's resolution, which allows it to take more photographs in slightly lower resolution. The highest resolution images are saved primarily for mission support tasks. Data is really the limiting factor, not the resolution. The original 2 year primary mission of MRO called for imaging 1% of Mars's surface with HiRISE, I'm sure the number now is closer to 10%, but I don't have that number handy.