tl;dr: This is not meant to be a at all a complete answer, but it's a start, and I think it gives a flavor of what will come; that in general the stuff is not very radioactive as (at)Gregg has already predicted.
But if your material is in contact with breathable gasses or perhaps even drinkable liquids, don't forget to buy your lunar Radon protection kit because just like here on Earth the U/Th decay chains will be emitting radon and if your regolith or rubble is porous enough it will fill up your cozy shack on the Moon with radon gas.
Source 
Source click for larger
I found Preliminary Examination of Lunar Samples from Apollo 11 in 19 Sep 1969 Science 165 (3899) 1211-1227.
A quick read suggests that these samples were not particularly radioactive compared to terrestrial rocks. The short half-life stuff (induced by protons) was already decaying a way and would be irrelevant for a construction project.
Here are some snippets:
Because of the complex operations involved in handling lunar material, preparation of samples for analysis in the RCL was low, and analysis of the firstRCL samplecouldnotbeginuntil 29 July 1969. As a result, radioactive species with half-lives of less than a few days were undetectable. In addition, the intense interferences from the gamma-ray spectra of the Th and U decay series in the samples made detection of weak gamma-ray components very difficult.
and
Low-background gamma-ray spectrometers used in these studies were located 15 meters below ground in a room supplied with radon-free air and shielded by 0.9 meter of compacted dunite inside a welded steel liner of low radioactivity. The principal detector system consisted of two NaI(Tl) detectors (23 centimeters in diameter and 13 centimeters long) at an angle of 180 degrees with the sample between them. Data were recorded in the singles and coincidence modes. Response of the detectors was enhanced by use of a surrounding anticoincidence mantle. Back- ground was reduced further by surrounding the detectors and inner mantle with a thick lead shield, the exterior of which was covered by a thin, meson-sensitive anticoincidence mantle. Spectra were also recorded with a Ge(Li) detector of large volume inside a lead shield 10 centimeters thick. However, all data reported here were obtained from the NaI(Tl) scintillation spectrometer.
and
The results are summarized in Table 5. Twelve radioactive species were identified, some tentatively. The nuclides of shortest half-life characterized were 52Mn (5.7 days) and 48V (16.1 days).
and
Yields of cosmogenic 26Al are generally high, and the ratio of 22Na to 26Al is considerably below unity in both the lunar rocks and soil. The high yields of 26Al may be understood in terms of the chemical composition of the lunar material, if the samples in question have been exposed to cosmic-ray bombardment for several half-lives of 26Al (0.74 x 10^6 years) so that the production of 26Al could reach saturation. Such reasoning suggests, in agreement with the rare-gas analyses, that the material analyzed in these studies has been exposed to cosmic radiation for at least several million years.
Here's Table 5. for ppm numbers it's some work to estimate a dose, you have to track down the isotope and do all the dosimetry calculations, but for the dpm/kg that's decays per minute and it's really, really, really small. No wonder they had to go to such extraordinary measures to measure in a low background environment and use background suppression techniques.
There will be much better and quantitative follow up papers using high resolution Ge(Li) detectors and of course five more Apollo missions and hundreds of kg of rocks and regolith.
click for larger
.svg){kind=link}