Some of the factors constraining launch vehicle size are the same as for any other technology-based product, namely fixed and incremental costs, the learning curve, time to market, and economies of scale.
A larger launch system will have larger fixed costs (not necessarily per unit of payload mass per launch). Gathering the funding for a larger research and development effort is more difficult, in part because the risk is higher (the payoff is likely over a longer period which increases the risk from inaccurate prediction and increases the required magnitude of the payoff by the compound interest effect).
Larger systems also have larger fixed costs with respect to manufacturing.
When a technology is relatively immature and has relatively limited demand (both from cost and lack of familiarity), fixed costs tend to dominate.
With respect to the learning curve, since a larger launch system will have fewer launches, more payload will be attempted to be launched with a lesser understanding of the system. E.g., if the first two launches each have an 90% success probability, the next four 95%, the next sixteen 98%, then to launch 20 units of payload, a four-units per launch system will expect to require 5.37 launches, losing 0.37 vehicles and 1.5 units of payload while a single-unit system will expect to require 20.69 launches losing 0.69 vehicles and 0.69 units of payload. Just counting incremental costs, if the payload value is twice the single-unit system vehicle cost, then to break even at 20 units of payload launched the heavy lift vehicle must have an incremental cost only 3.56 times that of the single-unit vehicle (11% less expensive per unit of payload capacity).
The learning curve also applies to manufacturing costs and even rocket efficiency (which can adjust payload size upward slightly or fuel costs down slightly).
Time to market interacts with funding and the learning curve. Being the first to begin making a profit (or even having a firm prospect of making a profit) reduces the cost of attracting capital (which funds additional R&D). Being the first to successfully launch will bring in more customers, increasing revenue (and profit) and increasing the number of launches. (A similar phenomenon for programming languages has been called "Worse Is Better".)
Economies of scale give a modest advantage to smaller systems beyond what would be expected from a simple model of the above factors. As volume increases, competition becomes more practical (increasing innovative pressure and opportunity [outsiders are more able/inclined to contribute] and reducing profit margins [but also risk]) and standards tend to develop which reduce production and design costs.
Another factor favoring smaller systems is that multiple types of systems become more economical to run. If an unexpected failure mode grounds a particular type of system, other systems will be available for launch.
Launch systems have some specific features that constrain the minimization of payload size. The launch itself has costs associated with use of a given area that must be cleared as an aviation hazard and the ground area cleared for launch failure safety issues. The monitoring cost of a launch is also relatively fixed per launch rather than per unit of payload.
In addition, a larger launch vehicle will tend to be somewhat more efficient due to area versus volume effects.
There are also limits on how many devices can be tracked, communicated with, and maintained in non-conflicting orbits, so there is some benefit to having a larger payload. Without in-space assembly, the size of a system that works effectively as a whole can be a limiting factor at the lower end of size.
Scheduling of launches is also a factor. With a multi-payload launch, all the cargo must be able to accommodate launch at the same time. With multiple launches, the delay of one launch would not necessarily impact the schedule of other launches. Furthermore, with more launches, there would likely be more launch sites, so localized effects like weather would have a reduced impact on launch capability.
A multi-payload system would also tend to increase the number of failure modes and might make specialization for specific components less attractive.
(The economics of space launches is one of the arguments in favor of fuel depots, despite the fact that a significant fraction of cryogenic fuels would be lost due to heating while in orbit.)