It could be argued that there isn't really a general "optimal key length" that is optimal irrespective of a cryptosystem. Most cryptographers and e.g. NIST do expect a specific, related key strength of 128 bits (efficient but secure) to 256 bits (for higher confidence in the security, usually at the expense of performance). NIST usually also identifies 192 as an in-between security level, but that generally receives less attention.

However, most symmetric cryptosystems - and you seem to be proposing a symmetric cipher - manage to have a key size that is close to this security level. So AES-128 provides near 128-bit security, using 128 bit keys as the name suggests. This could be considered the optimal key length for the simple reason that it is the smallest key size that delivers the required level of security.

Quantum computing muddles the water though. Not all algorithms are quantum-safe. Hoever, most symmetric ciphers and derived constructions such as secure hash-algorithms are relatively safe. They can still be attacked using Grover's algorithm, and possibly by hybrid classical / quantum computer based attacks. Grover requires a lot of qubits though and a pretty stable quantum computer. In that case the strength against quantum computers is halved. It seems that your algorithm was designed to take this in mind.

So assuming that a shorter key size is considered beneficial then a 256 bit key size should be plenty to achieve 128 bit security in the theoretical sense. According to most cryptographers here the practical security may be higher than that due to the requirements asked of the quantum computer required to perform the attacks. Anything higher than this will make the use of the algorithm inefficient, and key sizes of 10k or higher (such as proposed in the question) will be especially problematic in many situations.

It could be argued that there isn't really a general "optimal key length" that is optimal irrespective of a cryptosystem. Most cryptographers and e.g. NIST do expect a specific, related key strength of 128 bits (efficient but secure) to 256 bits (for higher confidence in the security, usually at the expense of performance). NIST usually also identifies 192 as an in-between security level, but that generally receives less attention - at least for symmetric operations where the overhead of 256 bit security isn't that large.

However, most symmetric cryptosystems - and you seem to be proposing a symmetric cipher - manage to have a key size that is close to this security level. So AES-128 provides near 128-bit security, using 128 bit keys as the name suggests. This could be considered the optimal key length for the simple reason that it is the smallest key size that delivers the required level of security.

Quantum computing muddles the water though. Not all algorithms are quantum-safe. Hoever, most symmetric ciphers and derived constructions such as secure hash-algorithms are relatively safe. They can still be attacked using Grover's algorithm, and possibly by hybrid classical / quantum computer based attacks. Grover requires a lot of qubits though and a pretty stable quantum computer. In that case the strength against quantum computers is halved. It seems that your algorithm was designed to take this in mind.

So assuming that a shorter key size is considered beneficial then a 256 bit key size should be plenty to achieve 128 bit security in the theoretical sense. According to most cryptographers here the practical security may be higher than that due to the requirements asked of the quantum computer required to perform the attacks. Anything higher than this will make the use of the algorithm inefficient, and key sizes of 10k or higher (such as proposed in the question) will be especially problematic in many situations.