tl;dr: Alas, Spaceman Spiff will not have a very good viewing experience. This turns out to be a really interesting problem by the way! The only directions you can point the orbital telescope and hope to have it continue to point in the same direction at a star without an attitude control system are the perpendiculars to the orbital plane. For an Earth equatorial orbit for example, that would be Celestial North, near polaris, or Celestial South.
The Earth's gravitational potential falls off as 1/r, the force as 1/r^2. For any orbiter of non-zero size, some parts of it will be closer to the Earth and be pulled by the Earth more than other parts. This gravity gradient has been both a problem and a solution.
First let's think about the orientation with respect to the Earth's center, the nadir or local vertical direction. For a long thin object like a tube or telescope, there are two ways that it can be oriented so that it remains in an equilibrium position in this rotating frame.
Vertical, or pointing straight down. If you orient the telescope in this direction and nudge it so that it is slowly rotating at the same rate that it revolves around the Earth, it will remain in that nadir-pointing direction. There is no net torque, except for higher order problems because the Earth does not have a perfectly spherically-symmetric potential.
This is a stable equilibrium. What that means here is that if you nudge it slightly; say a few degrees, it will slowly oscillate back and forth with a fairly constant amplitude.
Tangential, or pointing exactly sideways. If you orient the telescope so that it points exactly 90 degrees away from the local vertical or nadir direction, then there is also no net torque. It could be pointed "forward" in the direction it is orbiting, or "sideways" meaning perpendicular to the orbital plane, or anywhere else within the plane defined by the perpendicular to the local vertical.
This is an unstable equilibrium. If you nudge it slightly towards the vertical, it will start rotating in that direction and accelerate, and ultimately exhibit some oscillatory motion. It's like a valley between two hills; if you nudge a ball down from the top of one hill (the unstable equilibrium point), it will then continue to oscillate between the two.
However if you nudge it in the plane of the perpendicular to the vertical, there is no torque to counter it and it will just slowly and steadily spin.
The only directions you can point the orbital telescope and hope to have it continue to point in the same direction at a star without an attitude control system are the perpendiculars to the orbital plane. For an Earth equatorial orbit for example, that would be Celestial North, near polaris, or Celestial South
For some math, see this question and especially @Litho's answer. For a thin rod of mass $m$ and length $l$ in circular orbit, with a perpendicular moment of inertia $\frac{1}{12}ml^2$, rotating in-plane around the short axis, the torque (to first order) is given by
$$L_G = -\frac{GM_Eml^2}{8R_C^3}\sin 2\theta,$$
and the instantaneous angular acceleration is simply
$$\ddot{\theta} = -\frac{3GM_E}{2R_C^3}\sin 2\theta.$$
That's a pretty amazing result! With $GM_E$ of about 3.986E+14 m^3/s^2 and an altitude of 400 km, $\ddot{\theta}$ can be as large as 0.4 degrees per minute^2 at 45 degrees, and that is independent of length!
When out of equilibrium, a rod will oscillate for a very, very long time, unless your object has build-in dampers to slowly absorb the angular acceleration as it spins at first, or eventually achieves "capture" in a gravity gradient stabilization, where it will oscillate and slowly decay. The details depend on initial conditions.
Natural damping mechanisms include differential atmospheric and photon drag, and induced tidal distortions in the Earth by the telescope's gravity. These are very very small on an orbital timescale!
This is all from Gravity Gradient Stabilization of Earth Satellites by R. E. Fischell:
The maximum angle to which the satellite will swing is of great interest. If this angle is less than 90°, capture of the satellite into gravity gradient attitude stabilization will result. The angle can be calculated rather simply by equating the angular kinetic energy that the satellite must develop (to achieve an angular rate of 1.0 rpo) with the work
done by the gravity gradient torque as the satellite moves out to that maximum angle.
Calculation shows that if the satellite is vertical at the time the boom is deployed, then it will swing out to a peak angle of 35.36 0. If the satellite is initially at an angle greater than 54° off the vertical, it will swing out to 90°; this, therefore, is the limiting angle for capture.
Python script: https://pastebin.com/yVq6WuCu
$\theta$ and $\dot{\theta}$ in inertial frame for a thin rod in a 400 km circular orbit starting at various initial angles with respect to the nadir, with no initial rotation in the inertial frame (i.e. looking at some star):
