Reminder: Analyzing schemes is off topic on crypto.SE. This question will therefore be likely to be closed soon.
Your scheme can be summarized by:
pick a S-box $\sigma$ and two keys $(k_1, k_2)$ and do:
for i in 0 .. len(message) do
c[i] = σ(m[i])
if i & 1:
σ >>> k2
else
σ >>> k1
return c
This is broken by frequency analysis with a sufficiently long message:
- Find $n$ such that $n \times (k_1 + k_2) \equiv 0 \pmod{26}$.
- Remark that every $n \times (k_1 + k_2)$, a character you encrypted with the same key, thus frequency analysis will work.
Worth to be noted:
- this is equivalent to a alphabetic Vigenère Cipher with a long key.
- modern cryptanalysis (CPA & CCA) breaks this with a 26 chosen plaintext and 1 chosen ciphertext. See bellow.
Query messages:
AA B C D E F .. Z
$\alpha\beta$
These will give the $\sigma$ table.
Assuming that AA
is encrypted by αβ
query a decryption for the ciphertext: $xαβ$ (prepend a random character). Drop the first character decrypted, the following letters will correspond to the shift distances $k_1$ and $k_2$.
In your example, the query would be:
E(AA) = NT
and
D(XNT) = YDH
leading to
D $\implies k_1 = 3$,
H $\implies k_2 = 7$.