TL;DR: won't work with any spacecraft further away than one third of the larger semiaxis of the earth's path, even for fully-armed battlestation sized spacecraft.
Radar (more exactly, time-of-flight) is limited by two things:
- speed of light: The nearest star is 4 light years away. So anything that we send into that direction could, earliest, return in 8 years. Now, this is not really the problem at hand, because
- free space attenuation is the fact that for any wave front, the power density decreases by the same amount the sphere surface increases with radius. I.e., you get $\frac1{4\pi d^2}$ of the original power density at distance $d$.
Now, the result is the so-called Radar Equation:
$$P_r = {{P_t G_t G_r \sigma \lambda^2}\over{{(4\pi)}^3 R^4}}$$
with
$$\begin{align}
P_r && \text{received signal power}\\
P_t && \text{transmitted signal power}\\
\lambda && \text{wavelength}\\
G_t,\,G_r && \text{directional gain of the transmit, receive antennas}\\
\sigma&&\text{radar cross section, "effective reflection area"}\\
R && \text{the distance between you and the radar target}
\end{align}$$
Let's plug in a few numbers.
First of all, let's assume your radar spacecraft has enough power, and sends 1 MW. It's also got an excellent receiver and lots of signal processing, so that it can even detect a reflected signal at far below thermal noise level at 20°C. Let's say it can work with -180dBm of power – that's $10^{-21}$ W. Pretty much nothing. (in fact, we're getting close to action quantization here)
Then, we come to the following reasoning for our maximum distance $R$:
$$\begin{align}
10^{-21} \text{ W}&= \frac{10^{6} \text{ W} G_t G_r \lambda^2 \sigma}{{(4\pi)}^3 R^4}\\
10^{-27} &= \frac{{ G_t G_r\lambda^2 \sigma}}{{(4\pi)}^3 R^4}
\end{align}$$
Let's furthermore assume your spacecraft has something slightly smaller than the Arecibo Observatory (72dBi) as antenna – something with a gain of 60 dBi, and let's also assume you use that for both transmitting and receiving, $G_t=G_r=G$
$$\begin{align}
10^{-27} &= \frac{{ G_t G_r \lambda^2\sigma}}{{(4\pi)}^3 R^4}\\
&= \frac{{ G^2 \sigma \lambda^2}}{{(4\pi)}^3 R^4}\\
&= \frac{{ 10^{12} \sigma \lambda^2}}{{(4\pi)}^3 R^4}\\
10^{-39}&= \frac{{ \sigma \lambda^2}}{{(4\pi)}^3 R^4}\\
\end{align}$$
The question remains: What's a good estimate for the radar cross section of your target? So, we need to pick a target.
I arbitrarily chose the Imperial Death Star. Which is nearly spherical, so we can analytically determine its RCS based on its radius $r$, assuming they have a nice, flat, metal surface freshly polished for the visit of the emperor (first Death Star had a $r=70\text{ km}$
$$\begin{align}
\sigma &= \pi r^2\\
&=\pi {(7\cdot 10^4)}^2\text m^2\\
&\approx 3\cdot 50 \cdot 10^5 \text m^2\\
&= 1.5\cdot 10^7 \text m^2\text{ .}
\end{align}$$
Back to our maximum distance:
$$\begin{align}
10^{-39}&= \frac{{ 1.5\cdot 10^7 \text m^2 \lambda^2}}{{(4\pi)}^3 R^4}\\
6.67\cdot10^{-47}&= \frac{{m^2 \lambda^2}}{{(4\pi)}^3 R^4}\\
\end{align}$$
Let's assume we're doing some 1 GHz as frequency, so we have a wavelength of
$$\begin{align}
\lambda &= \frac cf\\
&=\frac{3\cdot 10^8 \frac{\text m}{\text s}}{10^9\frac1{\text s}}\\
&=3\cdot 10^{-1}\text{ m .}
\end{align}$$
Why not a lower frequency, you ask? Simply because the size of an antenna of 60 dBi gain scales linearly with the wavelength. We need to get that antenna to space, so we can't have it being arbitrarily large (and as you've noticed, I'm overly concerned with realism).
It follows that
$$\begin{align}
6.67\cdot10^{-47}&= \frac{{m^2 \lambda^2}}{{(4\pi)}^3 R^4}\\
&= \frac{{9\cdot 10^{-2} m^4}}{{(4\pi)}^3 R^4}\\
0.74\cdot10^{-46}\text{ m}^{-4}&= \frac{1}{{(4\pi)}^3 R^4}\\
{(4\pi)}^3 \cdot 0.74\cdot10^{-46}\text{ m}^{-4}&= \frac{1}{ R^4}\\
R^4 &= \frac{1}{{(4\pi)}^3 \cdot 0.74\cdot10^{-46}}\text{ m}^{4}\\
&\approx \frac{1}{2000 \cdot 0.74\cdot10^{-46}}\text{ m}^{4}\\
&\approx \frac{1}{2 \cdot 0.74\cdot10^{-43}}\text{ m}^{4}\\
&\approx \frac{1}{1.5\cdot 10^{-43}}\text{ m}^{4}\\
&= \frac{1}{1.5}10^{43}\text{ m}^{4}\\
&= \frac{2}{3}10^{43}\text{ m}^{4}\\
R&=\sqrt[4]{\frac{2}{3}10^{43}}\text{ m}\\
&=\sqrt[4]{\frac{2}{3}10^{3}}\cdot\sqrt[4]{10^{40}}\text{ m}\\
&=\sqrt[4]{\frac{2}{3}10^{3}}\cdot 10^{10}\text{ m}\\
&\approx 5\cdot 10^{10}\text{ m}\\
&\approx 0.334 \text{ AU .}
\end{align}$$
Since from the formula we see that radius of the target only contributes to maximum range with the square root, to get a max distance of 5 AU, we'd need to increase the radius by a factor of $\left(\frac5{0.334}\right)^2\approx 15^2=225$, ie. that body would need to have a diameter of 31,500 km at least – about one fourth of the diameter of Jupiter!
