Analyzing security flaws of cipher

Question

I know using homebrew encryption can be very dangerous as it is very likely to have many flaws in its design. The following concept is just for learning purpose in case anybody is getting red flags reading this. Hopefully this is the right place to ask.

I have made up a block cipher and would like to know how I can possibly find flaws or might even break it. As it is self-made and I have no prior in-depth knowledge in cryptography I assume it is likely to find ways to attack this encryption. As there is no general breaking tool for ciphers (at least not from what I understand) I would be very interested if some of you might share some ideas.

• The cipher is used in CBC mode has a block size of 512 bit and a key size of 1480 bit.

• The 512 bit input is filled into 64 cubes (or matrices) consisting of 2x2x2 bit. Each cube is filled with 8 bit of consecutive data from the input. Three times for each cube we are reading three bit from the key to determine operations to be done. These operations are the rotations of a single layer of the cube selectively in X,Y,Z direction and a circular shift of the bit (probability of circular shift is higher 2/8)

• The idea of using three operations is, that it cannot arrive at the initial position.

• Then for each cube 4 bit from the key are used to xor the cube from top to bottom. This avoids same outputs for cube filled just with zeros (or ones) and in generally should increase security as it adds confusion to the input.

• The 64 cube get shuffled, by using 64 times 6 Bit to determine a new position.

• Then 8 cubes each are used to create a 4x4x4 cube. We apply four operations on each cube. Three of them are again layer rotation, circular shifting and new rotation of the entire cube identified from 4 bit of the key (all operations same probability). Another 2 bit are used to either mirror on an axis or invert.

• Again the cubes get xored this time it needs 16 bit for each cube.

• In the final step the cubes get shuffled and put into an 8x8x8 cube. We read the output of the cube as the output of the cipher.

What would be the first approach to possibly break it?

My thoughts so far were, that it should not be likely to have key collision as each bit of the key would cause a different behavior of the algorithm. The cipher strongly depends on the key, so small portions of the key leaking would greatly compromise the security. Slightly different input with the same key will produce very similar output. Is it a security flaw that it cannot produce the same output as input as this could be case for the impossible differential cryptanalsis?

In case anyone is interested which I would greatly appreciate, there is an implementation that I could upload for further inspection.

Answer 1

When seeing clearly through all the cube "magic", one recognizes the following: All the cube operations are just key-dependent bit permutations. Therefore, the whole cipher is a sequence of key-dependent permutations and XORs with key bits.

This admits an algebraic description: For all keys, there is a permutation matrix $A\in\mathbb F_2^{512\times512}$ and a vector $b\in\mathbb F_2^{512}$ such that $$\mathit{cipher} = A\cdot \mathit{plain}+b\text,$$ where $\mathit{plain}$ and $\mathit{cipher}$ are interpreted as vectors over $\mathbb F_2$ of length $512$. Therefore, the cipher is an affine map over $\mathbb F_2$. This is about the most undesirable property a block cipher can exhibit. An attacker has of course no direct access to $A$ and $b$, but there are lots of weaknesses: For example, in a chosen-plaintext scenario, an attacker can

• encrypt $\mathit{plain}=0$, thus obtaining $\mathit{cipher}=b$; and subsequently
• encrypt $\mathit{plain}=\mathrm e_i$, the $i$th unit vector, for all $i$. He can then subtract $b$ from the resulting ciphertext, yielding $A\cdot\mathit{plain}$, and check which entry of this vector is $1$ — this is the target of the $i$th entry of $\mathit{plain}$ under the permutation $A$.

A chosen-plaintext attacker is therefore capable of obtaining (information equivalent to) the encryption key using a very simple procedure, hence making the cipher, to put it mildly, rather unsatisfactory (which is a synonym for "absolutely broken beyond any hope" in cryptography).

Answer 2

You say are asking as a learning exercise, to learn how to invent ciphers. The way to learn that is not to try to invent some block cipher and then ask others to break it. The way to learn is to learn cryptanalysis, by breaking other ciphers. See Schneier's self-study course on cryptanalysis for one good resource.