Kasiki's test and the index of coincidence are used to attack a Vigenère cipher - they both try to get the length of the keyword.

Kasiki's test gets probable prime factors of the keyword length, while the coincidence index test gets us an estimation of the absolute length of the keyword.

The original Kasiki test

This is what was done with CHR in your example: We look at repeated sequences of three or more characters, and at which distances they occur. We collect all these distances, and look at the prime factors of these.

The idea is that probably such a repeated sequence comes from the same plain text sequence, which then randomly hit the same keyword position. They will only hit the same position if their distance is a multiple of the keyword length.

Of course, we also can get random hits here (where different plaintext sequences got encrypted by different key sequences to get the same ciphertext sequence) - so be careful which prime factors to select. Either combine this method with Friedman's test mentioned below, or try different keyword length's for further analysis.

The coincidence index

The coincidence index of a text is defined as

Here n is the size of the alphabet, m_l the number of occurrences for the character l, k the total size of the text. (δ(x_i, x_i) = 1 if the characters at position i and j are equals, else 0. But for calculation the second sum is more convenient.)

The coincidence index of a totally random text would be $1/k$ (and this is also the total minimum), while for natural language texts it is higher (0.067 for english, a bit higher for German). For a ciphertext encrypted by a monoalphabetic cipher it is still the same as for the original plaintext, for polyalphabetic ciphers (like Vigenère) it is between those.

(Actually, there are different definitions of coincidence index, but the values you have seem to use this one.)

Column-wise coincidence index to get the keyword length

You now take guesses at the key length h, and writes the cipher text in h columns. Then (if we guessed right) effectively the letters in each column are encrypted with the same key letter, e.g. monoalphabetically. We now calculate the coincidence index of each column individually. If we had a good guess at the key length, the coincidence index is near the coincidence index of the original plaintext column (which, if long enough, is a good approximation for the coincidence index of the used language).

In your example, for the lengths 1,2,3,4 the coincidence indices are only around 0.045, which is not much higher than 0.385, the expected value for a random 26-letter distribution. For length 5, you see a sudden jump: the indices are around the value for English (0.067). For 6, it will likely get lower again.

This will get you quite similar results as the Kasiki test, but is easier automated.

Friedman's method

Actually, Friedman derived from this a formula which spares us the calculation of each column's coincidence index - we just need the global coincidence index of the ciphertext (and the one of the natural language) to get an estimation of the keyword length.

While the English Wikipedia page does not mention this formula, the german language Wikipedia shows this:

Here E_S is the expected coincidence index for our natural language, Φ(T) the coincidence index of our text, n the alphabet size, k the text length.

Your example

Using your value for m = 1 (which is the global coincidence index) and 26 for n, we get approximately

h ≈ (0.067 - 0.0385)k / ( (k-1) 0.045 - k0.0385 + 0.067) = 0.0285k / (0.0065*k +0.022)

For example, for k = 1000 we would get h ≈ 4.36. This is (together with the previous check from the column-wise coincidence index) a good hint at a keyword size of 5.

When we have the keyword length, we can use other methods, like frequency analysis on each "column", to crack the whole cipher.