Liquid hydrogen makes a poor first stage fuel. First stages operate with the vehicle full of propellant and lifting itself directly against gravity, and need thrust more than specific impulse. That means a high mass flow rate, and liquid hydrogen's lower density makes it more difficult to pump enough of it through the engine. Solid rockets are from the high thrust/low specific impulse end of the spectrum, and serve to get the rocket moving when it's too heavy for hydrogen engines to do the job. A couple rockets additionally use variable numbers of SRBs to allow a "dial-a-rocket" functionality, adding more of them to support heavier payloads or higher orbits.
There's been many liquid fueled rockets that do not use either hydrolox first stages or solid boosters, instead using kerosene (Soyuz, Falcon 9/Heavy, Electron) or hypergolic propellants (Proton, various Long March rockets). Instead of adding boosters, they just build the rocket big enough to handle the largest payloads in the target market, and sometimes add additional upper stages. This has generally been more commercially successful: the Russian rockets have done a couple thousand launches over the years with relatively low launch costs, and SpaceX's Falcon 9 has recently been able to undercut them. In the case of the Falcon rockets, the booster returns for reuse and its cost gets spread over multiple launches, so it doesn't matter so much that it's oversized for many payloads. SpaceX's Starship is to carry this further, reusing both booster and upper stage, using methane as a fuel.
For some numbers: each SRB added to an Atlas V adds about $7M to its price. The GEM-63XL to be used on Vulcan makes changes to improve its economics, but it's also larger, so that's probably a decent estimate of its cost and matches up fairly well with the expected price range. For comparison, the Falcon 9 first stage is estimated to cost around \$20-30M to build, but is expected to do at least 10 flights, and SpaceX recently contracted a flight (the IXPE) for just \$42M.
One major reason for the continued use of SRBs is politics. SRB technologies are shared with ICBMs, and the manufacturers of such have substantial political influence. For examples, look at the events surrounding how Shuttle SRBs ended up being made in Utah (ultimately resulting in the Challenger disaster), or the fights around the Ariane 6 design, which originally was going to have first and second stages consisting entirely of solids.