Scott Aaronson likes to motivate the factoring-to-period-finding algorithm used inside Shor's algorithm as follows.
Now, I want you to step back and think about what this means. It means that, if we can find the period of the sequence
$$x \bmod N,\quad x^2 \bmod N, \quad x^3 \bmod N, \quad x^4 \bmod N, \quad \dots $$
then we can learn something about the prime factors of $N$! In particular, we can learn a divisor of $(p-1)(q-1)$. Now, I’ll admit that’s not as good as learning $p$ and $q$ themselves, but grant me that it’s something. Indeed, it’s more than something: it turns out that if we could learn several random divisors of $(p-1)(q-1)$ (for example, by trying different random values of $x$), then with high probability we could put those divisors together to learn $(p-1)(q-1)$ itself. And once we knew $(p-1)(q-1)$, we could then use some more little tricks to recover $p$ and $q$, the prime factors we wanted.
See also Section 19.2 (pp 156) of his Lecture notes where he expounds upon on the same idea. However, in both expositions he ends with a dismissal of the idea:
Unfortunately, this doesn’t quite work with Shor’s algorithm, because the period of $f \colon r \mapsto x^r \bmod N$ might not equal $\phi(N)$, the most we can say is that the period divides $\phi(N)$.
But why doesn't it work?
As Aaronson himself noted, by repeating the algorithm with different values of $x$ wouldn't there be a chance of finding all the factors of $\phi(N)$ (or at least sufficiently many to figure out the remaining factors somehow)?