I know using homebrew encryption can be very dangerous as it is very likely to have many flaws in its design. The following concept is just for learning purpose in case anybody is getting red flags reading this. Hopefully this is the right place to ask.
I have made up a block cipher and would like to know how I can possibly find flaws or might even break it. As it is self-made and I have no prior in-depth knowledge in cryptography I assume it is likely to find ways to attack this encryption. As there is no general breaking tool for ciphers (at least not from what I understand) I would be very interested if some of you might share some ideas.
The cipher is used in CBC mode has a block size of 512 bit and a key size of 1480 bit.
The 512 bit input is filled into 64 cubes (or matrices) consisting of 2x2x2 bit. Each cube is filled with 8 bit of consecutive data from the input. Three times for each cube we are reading three bit from the key to determine operations to be done. These operations are the rotations of a single layer of the cube selectively in X,Y,Z direction and a circular shift of the bit (probability of circular shift is higher 2/8)
The idea of using three operations is, that it cannot arrive at the initial position.
Then for each cube 4 bit from the key are used to xor the cube from top to bottom. This avoids same outputs for cube filled just with zeros (or ones) and in generally should increase security as it adds confusion to the input.
The 64 cube get shuffled, by using 64 times 6 Bit to determine a new position.
Then 8 cubes each are used to create a 4x4x4 cube. We apply four operations on each cube. Three of them are again layer rotation, circular shifting and new rotation of the entire cube identified from 4 bit of the key (all operations same probability). Another 2 bit are used to either mirror on an axis or invert.
Again the cubes get xored this time it needs 16 bit for each cube.
In the final step the cubes get shuffled and put into an 8x8x8 cube. We read the output of the cube as the output of the cipher.
What would be the first approach to possibly break it?
My thoughts so far were, that it should not be likely to have key collision as each bit of the key would cause a different behavior of the algorithm. The cipher strongly depends on the key, so small portions of the key leaking would greatly compromise the security. Slightly different input with the same key will produce very similar output. Is it a security flaw that it cannot produce the same output as input as this could be case for the impossible differential cryptanalsis?
In case anyone is interested which I would greatly appreciate, there is an implementation that I could upload for further inspection.