The Mission Rules as of Apollo 11 don't seem to have a hard cutoff time for TD&E. The pilot can cancel TD&E if they don't like the look of things:
TD&E will not be performed for--
A. Pilot's evaluation of rates and attitudes, and SLA configuration not acceptable.
The real limiting factor for repeated attempts would be fuel in the RCS system:
3-91 | SM RCS
A. The redline for TD&E is 630 lbs and is consistent with a CSM only lunar orbit mission
According to this overview of the service module, a full load of RCS propellant was 1362 lbs.
Normally TD&E would take only an hour or so start to finish (everything being done slowly and cautiously). Apollo 14 had trouble with the capture latches and took about two hours before getting a good docking. After they successfully docked, there was this exchange:
005:17:21 Fullerton (CAPCOM): Have some RCS quantity numbers - just - it should make you feel good. You used 131 pounds so far, which is 62 pounds below nominal, but you're still 211 pounds above the redline. Over. [Long pause.]
This is Apollo Control, Houston. You heard those...
005:17:48 Mitchell (LMP): Roger, Houston. We got - We used 131 pounds, which is 62 pounds below nominal. And say again how much above redline?
005:17:57 Fullerton: You're still 211, 211 pounds above the redline.
This math -- 342 lbs to redline -- doesn't square with the Apollo 11 rules and rated RCS fuel capacity; there seems to be a different value in the A14 Mission Rules called the "CSM rescue redline" being referenced here but it's not clear to me what that is.
Assuming the RCS consumption rate was generally constant during TD&E attempts, this 131 + 211 lb redline would be reached after something like 5 hours of attempts. The crew of A14 discussed suiting up and having someone EVA to look at the docking hardware from the outside; this would have taken quite a bit of time but not expended much RCS.
The S-IVB might have been a time-limiting factor; it was supposed to be disposed of in a controlled manner, and I believe it was battery-life limited, but I don't see any specific references in the mission rules.
The next significant "dealbreaker" in the mission timeline is a midcourse correction at about 26 hours; obviously this couldn't be done by the CSM without losing the LM/S-IVB if the LM hadn't been extracted by that point.