Is this hand cipher any more secure than the Vigenère cipher?

Question

I know that inventing one's one crypto always sucks, but the problem is that hand ciphers are usually

1. very insecure
2. very slow.

This is an attempt to make a relatively secure, keyable, and modular multi-round cipher by hand.

In each round of encryption the following two steps are done:

1. The text is permuted by concatenating the reverse of the even-numbered characters to the odd-numbered characters. This is equivalent to writing the first char in the first place, the second in the last place, the third in the second place, the fourth in the second-to-last place, etc and is very fast to do by hand, and can be mentally reversed simply by "zig-zagging" the eyes when reading. (12345 -> 13542)

      T H I S X I S X S E C R E T
   => T I X S S C E T R E X I S H
   

2. The text is then added$\mod 26$ by repeating a series of numbers, i.e. the key. The numbers are from 0 to 10 so that they are easy to write underneath the letters and also because encryption can be done in head without memorizing, say, X is the 24th number of the alphabet. I'm especially concerned whether limiting this to numbers from 0 to 10 rather than from 0 to 25 seriously cripples security. (Obviously the digits of $\pi$ is a bad choice for a key but this is just an example)

      T I X S S C E T R E X I S H
      3 1 4 1 5 9 2 6 5 3 5 8 9 7
   => W J A T X J G Z W H C Q B O
   

This encryption process is continued for 5 to 10 rounds. The reason for the permutation phase is to break patterns that weaken the Vignère cipher.

I have observed the weakness that the first letter is never permuted to another place; thus known-plaintext attacks can easily trivially find the first number of the keystream given the number of rounds. Other than that I haven't found any very serious flaws.

Does this cipher system have any mathematical advantage over older (but slower to compute!) ones such as Vigenère or Playfair?

Answer 1

Let's assume a ciphertext only attack, and compare it to Vigenere. Your scheme is on a quite similar level of security: It might be hard to break it without a computer, but with one it is probably done in seconds. Here's how:

First, your permutation is static and only depends on the number of rounds and the length of the text, therefore you can just reorder straight away (e.g. 1st stays at 1st, 3rd becomes 2nd after the first around and then the second to last in the next round, etc.)

This leaves us with a substitution cipher (modular on the alphabet), where you add a combination of the key digits on each character. If the key is chosen uniform and has the same length of the alphabet, this is really hard to solve, but that's the same for Vigenere: If the key length has the same length as the text, and each digit is chosen from a uniform distribution, then Vigenere is just a OTP. If you just choose numbers 0-9 for Vigenere but without repetition, frequency analysis might still help (at least it can exclude certain messages).

Here's a problem tho: Your key looks like $k_1k_2...k_n$ and we know where each character ends up. In each round, one of the key digits is added to the original letter. But since your permutation each round is fixed, everyone knows which one of the key digits is added.

A simple example for the first letter: If you have 5 rounds, then the first letter will end up as $m_1+5k_1$. Is this a problem since only the first one is static? Yes it is. Here's a really short example, assuming 2 rounds:

• Message $(m_1,m_2,m_3,m_4)$ , Key $(k_1,k_2,k_3,k_4)$
• After the first permutation: $(m_1,m_3,m_4,m_2)$
• After the first round: $(m_1+k_1,m_3+k_2,m_4+k_3,m_2+k_4)$
• After the 2nd permutation: $(m_1+k_1,m_4+k_3,m_2+k_4,m_3+k_2)$
• After the second round: $(m_1+2k_1,m_4+k_2+k_3, m_2+k_3+k_4, m3+k_2+k_4)$

So... the problem here is that the remaining letters just are a sum of the original letter + a linear combination of the key digits and this will continue over any number of rounds. At this point a combination of frequency analysis and solving a linear system will give the attacker an advantage.

But coming back to Vigenere: The weakness there lies mostly in the fact that the codeword was shorter than the text and the repetition is leaking. If you happen to repeat a single code word, above mentioned linear system will collapse as well. The math involved is more complicated than calculating the auto correlation for Vigenere, but it is there.

edit: Further details:

As requested, I'll explain why short keys and repetition in this scenario are bad. Let's use the formula from the example above with 2 rounds, with the assumption

• $k_1 = k_3$ and $k_2=k_4$
• This leaves us with this ciphertext: $(m_1+2k_1,m_4+k_2+k_1, m_2+k_1+k_2, m3+2 k_2)$

What can we get from this? Let's assume we have a ciphertext "ABCD". At first, we look at the first letter: Since it is constructed from $m_1+2k_1$, we know that $2k_1$ is clearly an even number. Let's see which original letter would be the most likely, such that adding an even number (below 20) would result in the letter "A". If we have a correct guess, this gives us $k_1$. Let's look at the other letters: The second and third letter of the ciphertext both add $k_1+k_2$, so there we can have a guess for the sum of those letters and find a most likely combination. The last letter works just as the first, since it adds $2k_2$. Even if we can not identify the exact letters when looking at them individually, we can work with exclusion: Assuming that $k_1=1$ would mean that $m_1=Y$. Assuming english language, we can pretty much exclude $k_2=3$ from the 3nd letter ($m_2+...$) already, since there are no words which start with "YY" (It was a toy example, no idea if there is a likely plaintext).

One more thing: The distribution of a construction like $m_x+k_1+k_2+k_3+...$ is quite bad, because it is far from uniform. The problem is, that the sum of independent random variables is not uniform distributed: If you throw 2 dice, the chance of getting a sum of 7 is much more likely than a 2 or a 12. (See central limit theorem for the asymptotic behavior: it approximates the normal distribution).

Calculating the exact probabilities is quite difficult, but it is not uniform. Additionally, the modulus 26 makes it even more complex, but it can't even out all the problems.

Answer 2

Given that the permutation is fixed and the key step is independent of the permutation you can reduce this to an ordinary text-substitution cipher.

If the key is as long as the input you have a weak one-time pad, because the per letter change is limited to 10 instead of 26.

However if the key is short then you have a Vigenère cipher (if you "decode" with $key=0$ then it is exactly a Vigenère cipher).

In essence you use 2 transformations: a character scramble ($P$ $P'$) and a "xor" ($X+$ and $X-$).

Your algorithm is $text1=X+(P(text_0),key)$for each phase.

However this is equivalent to $text1= P(X+(text_0,P'(key)))$.

Then the next fase is $text_2= P(X+(P(X+(text_0,P'(key))),P'(key)))$ which can be rewritten to $text2 = P(P(X+(X+(text0,P'(key) ), P'(P'(key))))$ and so on.

Given that P is easily reversible you can extract $X+(text, X+(P'(key), P'(P'(key)) )$.