edit: Since I'm also the OP, after accepting @BobJacobsen's answer I've slightly revised the question to allow for another, very informative answer, which addresses testing of a higher frequency band through the Earth's atmosphere, though the testing was done from LEO and not deep space. This answer here is about the highest frequency used or tested from deep space.
I'll post this as a tentative answer. I'm not 100% sure a higher frequency hasn't been tested but it looks like the Deep Space Ka-band frequencies around 32 GHz is the highest that's received substantial, systematic testing and use in deep-space.
I should also point out that the Transiting Exoplanet Survey Satellite (TESS) will be using Ka-band from cis-lunar space (roughly speaking) as discussed in the question Will there be “Near Space” Ka band allocations for TESS?
From the DSN Telecommunications Link Design Handbook 201, Rev. B, Frequency and Channel Assignments, Released December 15, 2009:
I should not that all of these seem to be downlink. I haven't found any corresponding Ka-band uplink information.
The Kepler Telescope has been using Ka-band to downlink processed data regularly over the distance of roughy 1 AU.
Mars Reconnaissance Orbiter (MRO):
According to the abstract of this paper Mars Reconnaissance Orbiter Ka-band (32 GHz) Demonstration: Cruise Phase Operations (also this presentation):
Among the highlights of these was setting a single day record for data return from a deep space spacecraft (133 Gbits) achieved during one 10-hour pass; achieving the highest data rate ever from a planetary mission (6 Mbps) and successfully demonstrating Ka-band DDOR.
I'm not sure of the distances, the experiments seem to have been done during the cruise-phase of the mission before orbital insertion.
The Ka-band transmissions from the Juno Spacecraft have been systematically studied for an extended period of time. In the IPN Progress Report Cassini Downlink Ka-Band Carrier Signal Analysis. Here's the complete abstract, there is plenty to read in this report:
Lower frequency telemetry bands are becoming more limited in bandwidth with more competition between flight projects and other entities. Higher frequency bands offer significantly more bandwidth and hence the prospect of much higher data rates. Future or prospective flight projects considering Ka-band (32-GHz) telemetry data links are interested in past flight experience with received Ka-band data. Over 10 years of Cassini closed-loop received Ka-band carrier data involving over 2 million individual measurements were acquired at all three Deep Space Network (DSN) sites. We analyzed these data to characterize link performance over a wide range of weather conditions and as a function of elevation angle. Based on this analysis, we have derived a recommendation for telecommunications
link margin for preflight planning purposes. These results suggest that a 4-dB margin will ensure a ~94 percent data return at a minimum 20-deg elevation angle under 90 percent weather conditions at 32 GHz (Ka-band).