Extremely sensitive. In fact, there are only three other positive values that would not violate various security guarantees, and the example you provided was not among them. And even of those three, only the one chosen protects from all edge cases. Now, in a little more detail...
Curve25519 is a Montgomery curve in the form of $y^2 = x^3 + Ax^2 + x$ for efficiency reasons. For the curve to be secure, an adequate $A$ must be chosen. The three possible choices for $A$ that would be secure under the definitions set forth in the paper introducing the curve are $358990$, $464586$, and $486662$. The first two have prime factors smaller than $2^{252}$, which is an issue in the very unlikely situation that a given secret key matches the prime. Rather than amend the security definitions of the curve and place limits on the choice of key, Bernstein went with $A = 486662$ for simplicity.
If you go with a completely different $A$, then all sorts of unpredictable things can happen. Various security guarantees may be broken, and a number of vital performance features will go down the drain. The choices of $A$ must have curve and twist orders of $\{4 \cdot \mathrm{prime}, 8 \cdot \mathrm{prime} \}$ to avoid a number of attacks specified in section 3 of the Curve25519 paper. The specific change you proposed does not match these criteria, so any number of attacks may apply. This is really the crux of the answer. Anything can happen once the security constraints are violated.