Why these specific number of rounds only?
Because AES is a standard; AES is an acronym for "Advanced Encryption Standard".
The standard specifies these specific number of rounds to ensure that different implementations are interoperable.
Why not more or less?
The reason these specific numbers of rounds were chosen was a choice of the designers. They did a lot of math to determine that these were the sweet spot between sufficient security and optimal performance.
Less might be insecure, and more might be slower with no benefit.
To quote the above book (from Section 3.5 The Number of Rounds):
For Rijndael versions with a longer key, the number of rounds was raised by one for every additional 32 bits in the cipher key. This was done for the following reasons:
One of the main objectives is the absence of shortcut attacks, i.e. attacks that are more efficient than an exhaustive key search. Since the workload of an exhaustive key search grows with the key length, shortcut attacks can afford to be less efficient for longer keys.
(Partially) known-key and related-key attacks exploit the knowledge of cipher key bits or the ability to apply different cipher keys. If the cipher key grows, the range of possibilities available to the cryptanalyst increases.