Reductio ad absurdum
If you could choose freely on which circle to orbit, the most convenient place to take off from would be the North pole. That would set the circle diameter to zero. You would then climb to whichever altitude you pleased and remain there, immobile in space, for as long as you wanted. How cool would that be?
An attempt at analogy
In reality, the trajectory of a circular orbit is like that of a stone spinning in a sling: Earth pulls you toward its centre and the centrifugal force pulls you away in the opposite direction. Since the forces balance each other out, all you can do is follow the circle.
Gravity acts like the rope preventing the stone from flying off. If the planet suddenly disappeared, you would shoot straight into outer space as if released from that sling.
Just like the stone circles around your hand (where the strings of the sling are held), the satellite circles around the centre of the Earth (where the "gravitational rope" is tied).
Now why would we want to launch a rocket due east?
Simply standing on Earth's surface, you spin with the planet (one full circle per day), and this gives you a rotational speed that translates to an initial flight speed in the referential of the orbit (i.e. the Earth immobile in space).
This time, the speed is proportional to the distance to earth's rotation axis.
If you stand on a pole, you just spin around on a zero radius circle, and your speed is zero. If you stand on the equator, your speed is maximal and amounts to about 1,700 km/h or 0.5 km/s.
As it happens, the speed that must be reached to go into a low orbit is roughly 8 km/s. In practice you will consume more fuel due to air drag and other technical issues, for a total equivalent of about 10 km/s. That means the rotation of Earth gives you an initial impulse of about 5% of the speed you need.
This initial speed clearly points due east (the direction toward which the Earth spins). So if you follow that course, you simply add to this initial speed until you reach orbit, spending 9.5 km/s worth of fuel.
If you went due West, you would first have to cancel that initial speed, and then start accelerating in the opposite direction, going from -0.5 to 10 km/s for a total of 10.5 km/s.
This 1km/s difference might not look like much, but as it happens, the fuel cost of each extra km/h is exponential, which means the 10% or so speed difference might translate into a much more costly increase in mass and needed fuel.
Why does the inclination of an orbit launched due east match the launch site latitude?
The rocket is flying straight East until it reaches its orbit. Going East means remaining constantly at the latitude of the launch site.
During this part of the flight, the rocket follows roughly the trajectory you describe: a circle in a plane perpendicular to Earth's rotation axis. In reality it's rather a flat spiral, due to increasing altitude, but it would be a circle if the rocket continued applying thrust after reaching its orbit.
However, this is only possible during the powered part of the flight, because the rocket is actively fighting against gravity.
While the rocket's speed vector points due east, the engine nozzles (controlling the thrust vector) are tilted slightly northwards, to counteract the gravity trying to pull the rocket southwards into this earth-centred circle (or rather ellipse slowly turning into a circle as the rocket comes closer to the orbital insertion point).
As a side note, the guy in the video is slightly wrong when he plays around with Kerbal Space Program. Pointing the engine nozzles of a rocket due east will produce a trajectory that veers slightly southwards.
The drift is not huge because the ascension time is fairly short in comparison with the duration of an orbit, but still...
Anyway, soon as the engine cuts, the trajectory becomes only subject to orbital mechanics laws and starts following this circle centred around Earth.
There is no way gravity could pull the rocket further North. The orbital insertion point is the highest latitude the rocket will ever reach (unless the engine is re-ignited). The orbit must then be a circle centred around Earth that reaches its highest point at the launch latitude.
This is the very definition of an orbit whose inclination matches the launch site latitude.