I'm looking for a formula can be used to calculate the magnitude of a passing satellite (e.g. the ISS) based on it's TLE
In general, this is a very hard problem. You might take a look at this paper, which looks at heuristic models for estimating satellite visual magnitude as a function of phase angle, range, object size and object type. Though IIRC, it's only looking at geosynchronous objects.