This video may help to answer your question. Starting at about 00:24, you can see an astronaut running around the "exercise wheel" of Skylab (an early NASA space station program, which followed the Apollo moon landings). Basically after some time, NASA told the astronauts to stop running around like this because it was causing more propellants to be used to maintain Skylab's correct attitude (orientation) in space (at least this is what I've been told...it would be nice to find a reference to back this claim up).
Anyways, the short answer is yes, an astronaut's physical activity within a spacecraft can absolutely affect its orientation and even its orbit. As to whether it is a large or small effect, this depends on the amount and type of activity and (most importantly) the size of the spacecraft. Even on the ISS, however, astronauts are instructed to avoid or even stop altogether some activities because they can have an adverse affect on the amount of propellants used for attitude control (I have some memory of such instructions during my time working in the ISS Program). Another issue is that an astronaut's activities can affect the microgravity experiments being conducted on the ISS. These are typically conducted over a specified period of time, so an astronaut may be instructed to avoid motion in a certain part of the ISS for a certain period of time, in order not to disturb a particular experiment.
It's not really a simple matter of analyzing starting and ending velocities of an astronaut and assuming that the net change is zero. If you think of it in terms of kinetic energy, an astronaut is all the time transferring small amounts of kinetic energy to his or her spacecraft (the energy being converted from chemical bonds in the food he or she eats into kinetic energy from the work his or her muscles do). Changes in kinetic energy, of course, imply a change in velocity [E = (0.5) m * v^2]. On average, these velocity changes most probably cancel each other out, if they are not applied in a consistent direction. This is why there is no large overall change in a spacecraft's orbit due to these motions. However, the motions can absolutely cause small changes in the orientation of the spacecraft, and these are most often controlled by an active attitude control system (usually small rockets that fire in small bursts to counter unwanted changes in attitude).
I will add, however, that these effects (i.e. astronaut motion) are probably not the largest source of attitude disturbance on a spacecraft. I'm not really an attitude control expert, so I can't tell you the relative magnitude of the different disturbances, but one of the largest (for low Early-orbit spacecraft) is actually aerodynamic drag, caused by the thin layer of atmosphere present at such altitudes.