Going by some information in the comments on SEJPM's answer:
By other I meant quadratic or factorial times
Let's list some of these other time complexities:
These are used to refer to how long it takes to solve a given problem. The lower down on the list, the longer it takes to solve a problem of a given size (assuming that the size is large).
From working with symmetric cryptography, we know that if a problem takes $O(2^{128})$ time to solve (or better, $O(2^{256})$), then we can rest assured that no adversary will ever be able to solve that problem.
Why not use easier time problems?
Could we use a problem that has a say quadratic time solution and make $n$ large enough such that it takes time equivalent to $O(2^{256})$? Sure, make $n = 2^{128}$ and then solving your problem will cost $2^{256}$ time. But then $n$ will be so very large that actually using such a system will be impossible.
We like problems that only have exponential-time algorithms because it implies that we can keep $n$ very small and have an efficient/practical implementation of the scheme.
Cryptography would be easy if you didn't have to worry about efficiency; the magic/challenge of cryptography is to have problems that are (relatively) easy to evaluate but hard to solve. If it is equally hard to evaluate as it is to solve, then it is not useful for cryptography.
So why care about polynomial time specifically?
In general, a polynomial time algorithm is considered "efficient". It will not take until the end of the universe to solve even a large such problem.
There are some conceivable exceptions: The exponent could be so large that even reasonably small values of $n$ would lead to problems that are intractable to solve in a practical amount of time.
This question on cs.stackexchange has a table that is relevant to this topic. I will re-post it here for the discussion:

And a quote from the answer:
... you'd need to give an algorithm of logarithmic complexity input of size on the order of 2 to the something enormous in order to get it to run for a century, but for an algorithm of complexity $O(n!)$ you only input of size ~10 to make it run for that long.
If an there is an algorithm that provides a solution to the problem in polynomial time, in general that means the problem is solved too easily to be of use for cryptography.
Why don't we use problems with $O(n!)$ time solutions?
If you know of a problem where you can prove that the adversaries best solution is $O(n!)$, I'm sure the world would love to hear about it. Remember also, that the legitimate users (not adversaries) need to be able to compute whatever it is efficiently.