I was examining how AES provides diffusion and noticed that indices of the shiftRows transposition step can be modeled as inputs/outputs of a 4x4 table:
`0123 4567 89AB CDEF -> 0123 5674 AB89 4567`
Where the input is the index of each byte of the 4x4 state, and the output is the destination index that the byte will be transposed to.
I was curious about the effect of changing the above mapping to one with strong cryptographic properties, like those found in this paper.
There is the requirement that each column should be spread evenly across the state, so as to ensure diffusion. I found that only certain S-Boxes in the above paper fulfill this property.
The first S-Box I found that does fit that criteria was the third S-Box from the second variation of the Hummingbird cipher (found in the appendix of the paper above). It looks like this:
0123 4567 89AB CDEF -> F458 9721 A30E 6CDB
If I understood the paper correctly, this mapping is among those that offer ideal differential, linear, and branch count properties. We can refer to this operation as shuffleBytes, as it's a generalization of shifting rows.
How would I go about calculating the branch count and active S-Boxes with AES using the proposed shuffleBytes operation in place of the shiftRows operation, with all the operations otherwise the same?
Is there some (obvious) reason this would be bad, assuming the transposition + mixColumns operations still ensures diffusion at least the same rate?
After further testing, the complete diffusion over two rounds is only realized with certain configurations of bytes, and not in all cases. Maybe stacking a combination of multiple ideal S-Boxes over multiple rounds might be applied. But with just the one S-Box, diffusion does take longer then two rounds in some cases, so it probably won't be considered an improvement.