Is an attacker of an cryptographic algorithm allowed to change order of sub-keys but not changing their number in attack? Would it be called an attack on broken algorithm of cipher?
The attack you describe (at least insofar as I understand it) is essentially a type of related key cryptanalysis, which is a valid but very strong attack model that is not always accepted by cryptographers as showing serious weaknesses in a cipher. In terms of the plausibility of such an attack, the general idea is that if the expanded key (consisting of all the round keys) is stored or transmitted insecurely then it might be possible for an attacker to alter them in some particular way, even if she cannot simply read them directly. The alterations to the rounds keys would induce the cipher to behave differently in a way that enables the attacker to deduce bits of the key.
This scenario might seem to stretch credulity a little too much, which is why not all cryptographers are concerned about related key attacks. However, weaknesses in a cipher under related key attacks can often inspire breaks under the more conventional one-key model (e.g. Biham's 'key schedule' related-key cryptanalysis of LOKI inspired slide attacks, and differential related-key cryptanalysis techniques aided the development of biclique cryptanalysis). We also may be interested in the strength of a cipher when it is being used as the core of a hash function, where the attacker can directly control the 'key' input.
You do have to be a little careful about with the type of "key alteration" you propose (i.e. transposition of round keys). So long as the attacker is restricted to just transposing round keys that should be fine. But if the attacker is given more fine-grained control, to the point where she can transpose individual bits of the round key, then that is much too strong of a capability to grant her, because not even an Ideal Cipher can resist an attacker with that ability. Related key attacks are only really "interesting" (in the sense of revealing a structural weakness in a particular cipher) if an Ideal Cipher can in principle be secure against them.
|show 1 more comment|
Typically, in a cryptographical attack, the attacker is assumed to control the inputs to the cryptographical algorithm, but not the algorithm itself.
For example, for encryption algorithms, the attacker is typically assumed to be able to specify the plaintxt (chosen plaintext attack) and the ciphertext (chosen ciphertext attack). In addition, the attacker may be assumed to have some control over the key (in a "related key attack"); we may allow the attacker to ask for a specific plaintext to be encrypted with certain bits of the key flipped (although, even there, related key attacks are often assumed to be of academic interest, as attackers generally can't tweak the key). However, we usually assume that the nonkeyed part of the algorithm cannot be modified; for example, the attacker can't ask an AES encryption using a modified sbox.
Unless the order of subkeys are something that is potentially controlled by the attacker (e.g. it's controlled by some keying bits that can potentially be controlled in a related key attack), any such attack would not be considered a real attack against the cipher.
Now, just because it's not a real attack doesn't mean it's not of potential interest. Depending on how the attack works, it might be considered an attack against a weakened version of the cipher. That is, when considering real ciphers, we often don't attack the full version directly; instead, we consider reduced versions of them (e.g. with fewer rounds). As we develop these attacks, we then consider how they may be extended to be of use against the full version. An attack against a version of the cipher with reordered subkeys may be similar; perhaps with such an attack, someone might propose how to extend it so that it is of use against the original cipher.