Outside of games like Kerbal Space Program, application for such an algorithm is limited, because components like tanks and interstages are geneally purpose-designed and built for a particular rocket, rather than being selected off-the-shelf.
However, your problem description strongly hints that you're talking about KSP, so I'm going to run with that. I don't believe there's a general magic algorithm for optimal rockets, but there are several heuristic shortcuts that can allow you to reduce the size of the brute force search space. Wikipedia has some relevant hints on optimal staging.
Take the initial mass limit divided by the payload mass to get the overall mass ratio you're working with. For an n-stage rocket, the nth root of the overall mass ratio gets you a stage-to-stage mass ratio target; now you can break the problem down into building individual stages that fit approximately those target numbers.
For example, if your payload is 10 tons, your launch mass limit is 200 tons, then a 2-stage rocket should have a $\sqrt 20 = 4.472$ stage mass ratio; 10 tons payload, 45 tons 2nd stage+payload; 200 tons 1st+2nd+payload. Thus your first stage would be aiming at 155 tons, and your second 35 tons. You'd probably want to give some wiggle room to each stage search, say +/- 20% of the mass target.
Brute force search for an individual stage is pretty straightforward. Iterate over the engine types available, figure out how many of that engine you need to give the stage-plus-upper-stages better than 1:1 initial thrust-to-weight ratio, allocate the remaining mass after engines and decoupler to tankage, compute the stage ∆v, repeat and sort, discard the low performers. There might be better solutions using heterogenous engines (e.g. 1 of engine X and 2 of engine Y) it's up to you if you want to accept that complexity in the search.
Then you just take each combination of high-performing stages that remain under the mass limit and find the winner.