The ideal launch site
The previous answers point out that there are many factors that go into choosing a suitable launch site. However, assuming a strategic requirement for long term space access, the predominant factors are good clearance devoid of populated areas east of the launch site and as close to the equator as possible.
Why launch eastward
The earth's rotation can assist a rocket launch by boosting its payload capacity. The surface of the earth has a tangential velocity (parallel to the surface of the earth) which is towards the east. This is the prime reason why companies such as Sea Launch developed a sea platform that they pull out to the equator to get the maximum speed boost.
I found a mathematical demonstration as an answer to a similar question on this thread using Tsiolkovsky Rocket Equation:
$v_f=v_eln(\frac{m_i}{m_f})$
which allows you to calculate the initial mass of your rocket $m_i$
as a function of your final mass $m_f$ (empty rocket + payload), the final orbital velocity $v_f$ and the exhaust velocity of the rocket $v_e$.
Keeping in mind that this equation doesn't take into consideration the gravitational force acting on the rocket, we can still get an understanding of how the rotational speed of the earth plays a role in the economics of rocket launches. $v_e$ and the final mass $m_f$ are constants for a given rocket + payload combination. The final orbital velocity $v_f$ actually changes depending on the position of the launch pad relative to the equator.
The tangential speed of the Earth at the equator is its circumference ($2\pi r$, $r$ =6000 odd kilometres) divided by the time it takes to go round, a little less than 24 hours, giving a total speed of 465 m/s.
So plugging in all the numbers into the equation allows one to calculate the initial mass $m_i$ of our rocket if launched from the equator or from a position with a lower tangential speed. The closer to the equator, the lower $m_i$ to produce the same total $v_f$.
There are 2 alternative ways to calculate this, by keeping $m_i$ constant increasing $m_f$ allowing for a larger payload for equatorial launches, or to keep both $m_f$ and $m_i$ constant and calculate a larger $v_f$ allowing for higher orbits.
The original thread actually calculates these for the Soyuz rocket launched at different latitudes.
The equatorial launch does make a difference to the payload a rocket can carry because of the exponential nature of the equation. If we compare LEO orbits (regardless of their orientation be it equatorial or polar or in between), gains are quite small, on the order of hundreds of kg.
However, if we compare GTO orbits, where the majority of commercial payloads are sent, then the gains can go up by as much as 25%. The main reason for this is the fact that an orbital plane change needs to take effect, which is the reason why Sea Launch developed their sea-platform launch pad as the economic savings were worth it.
Furthermore, a NASA study dating back to 1959 made a detailed calculation of the cost reduction of launching a Saturn rocket from the equator as compared to Cape Canaveral and concluded that equatorial LEOs are 80% less expensive propellant-wise due to required orbital plane changes.
A more detailed derivation of the rocket equation with gravity force acting on the rocket can be viewed online on the MIT website. Note also that staging rockets reduces the actual payload capacity. If we say that a 2 stage rocket is 80% fuel, 10% rocket and 10% payload, the first stage's payload is the 2nd stage, therefore the actual payload is 10% of the 2nd stage, or 10% of 10% of the initial rocket, ie 1%.
Why did Russia launch so many satellites from Plesetsk
Plesetsk is not the ideal launching site, however, the Russians needed to develop alternate orbital solutions to meet their needs. Accessing GSO orbits for communication/weather/spy satellites was too expensive, and they therefore developed an alternative solution which instead leveraged the disadvantages of the Plesetsk site location into an opportunity.
High latitude launch sites are better suited for accessing polar orbits or highly elliptical orbits (or HEO orbits).
Russia's need for communication and weather satellites were met by perfecting the HEO orbit known as the Molniya orbit (Lightning in Russian). Molniya orbits are variable speed orbits, slower on the apogee (point furthest away from the Earth) and faster at the perigee (closest approach to Earth).
The Russians perfected this orbit to the point that the satellite would orbit twice the earth in a day, with an apogee centred on Russian and another one centred on the USA. This led to dual function satellites, spying on the USA 9 hours of the day, then zipping round the bottom of the Earth and serving as communication/broadcasting/weather satellites for the Soviet Union for another 9 hours of the day (see this illustration).
The downside of this orbit is that it requires several satellites to provide a continuous cover for the entire surface of Russia. The satellites are also exposed to the Van Allan radiation belts 4 times a day, resulting in shorter life-times. The Russians upgraded the satellites regularly, refining the orbits as well as the satellite technology.
Plesetsk is actually ideally situated to launch satellites into Molniya orbits, and as a result saw many more launches than Baikonour.