The simple answer
You are correct: it would only look big because you know it's big.
There are really three things, purely in terms of your visual system which together tell you how big something is:
- what angle it subtends in your vision;
- combined with focus & depth of field information;
- combined with comparing what your two eyes see, or how what you see changes if you move around.
For something like a planet the second and third of these don't give you any useful information: if you can see the planet as a whole then it's 'at infinity' from the point of view of the optical system of your eyes, and also you're not going to get any useful information from the differing input to your eyes, or by moving your head around.
So the things that tell you it's big are then just comparing it with other objects you know the size of. But, again, for a planet, there's not much useful to compare it with.
A more complicated answer
One thing I realised based on a comment is: if you wake up one day in a spacecraft orbiting some unknown planet (this sort of thing happens to all of us, I'm sure), what can you tell about the planet by how fast it seems to move under you, if you don't know your altitude over it? The answer is that you can tell some interesting things but not its mass.
So, let the mass of some planet be $M$ and its radius $R$. And let it be spherical (good enough approximation!), and not rotating (less good, but we can sort that out below).
Kepler's third law says that
$$T = 2\pi \sqrt{\frac{a^3}{GM}}$$
Where $a$ is the semimajor axis of the orbit. Let's just worry about circular orbits (see below), and define $r \doteq a$, the radius of the orbit.
So now the angular velocity of such an orbit is just $2\pi/T$:
$$\omega = \sqrt{\frac{GM}{r^3}}$$
OK, so a bit of trigonometry tells you two useful angles:
The angle subtended by the planet in your vision is $\theta = 2\sin^{-1}(R/r)$, and the angle through which you need to orbit for a feature on the planet's surface to move between the horizons as seen from orbit is $\phi = 2\cos^{-1}(R/r) = 2\cos^{-1}(\sin(\theta/2))$. Let's say it takes some time $t$ for this to happen.
So the things you can know, just by looking are:
- $\omega$, or equivalently $T$, by measuring the positions of the distant stars with your distant-star-o-meter (all spacecraft come with these);
- $\theta$, by looking at the planet (with a planet-o-meter, of course);
- $t$, the time it takes for features on the planet to move between horizons, with your trusty watch.
- $G$ which is engraved on a plaque on all spacecraft.
From $\theta$ you can work out $R/r$ and hence $\phi$, and in fact this and $t$ tells you $\omega$ so all that star-peering was not necessary. If you suspect the planet may be rotating, then you still want to measure $\omega$ using the star-o-meter because then you can use that value for it to compute the angular velocity of the planet by measuring how much $t$ differs from what you think it should be.
But you don't know $r$, or $R$: you just know $R/r$.
Well, looking at the Kepler expression for $\omega$ again: the mass of the planet is
$$M = \frac{4}{3}\pi R^3\rho$$
where $\rho$ is the average density. Plugging this into the Kepler expression you get
$$\omega = \sqrt{\frac{4\pi G\rho}{3}\left(\frac{R}{r}\right)^3}$$
or
$$\rho = \frac{3\omega^2}{4\pi G}\left(\frac{r}{R}\right)^3$$
So you can measure how dense the planet is, as well as its rotational velocity, by timing things and looking at its subtended angle, but you can't know how big it is.
Although I've shown this just for circular orbits, it's true for any orbit at all in Newtonian gravity: this is reasonably obvious from looking at Kepler's third law: because $a^3$ occurs in this, and because $M\sim R^3$, everything works out so that all you can know is $M/R^3$ if you only know $R/r$. A similar thing works for masses: if all you know is $m/M$ you can't use this to find either $m$ or $M$.