I'm doing the Matasano crypto challenges. After implementing a naive character frequency scoring (that worked), i was looking for something mathematically more precise and found this thread: Developing algorithm for detecting plain text via frequency analysis
While the Chi Squared analysis does work for me, I encountered a problem. My current code just sets the score to infinity if I encounter a nonprintable character. I want to avoid doing that, since a possible plaintext could contain a few ascii non printable characters. (for example it is actually unicode and contains a greek word) But when I ignore every non printable character for scoring, as I already do with chars like ".,!\n\t", I run into the problem that strings with a few high scoring characters and lots of junk get scored better than actual english plaintext.
My current code is as follows: (frequency is a dict that associates ord(character) with its frequency for a-z and space)
import frequency
import copy
from math import inf
from string import printable
def score_string(inp):
exp_dist = copy.deepcopy(frequency.frequency)
length = len(inp)
for key in exp_dist:
exp_dist[key] *= length
dist = dict.fromkeys(exp_dist.keys(), 0)
for char in inp.lower():
if chr(char) not in printable: # This is the part i like to avoid
return inf
if char in dist:
dist[char] += 1
score = 0
for key in exp_dist:
score += (dist[key]-exp_dist[key])**2/exp_dist[key]
return score
If I understand what is going on correctly, my issue is that for shorter strings (the example i worked with was 34 characters) the negative impact of even a single low scoring printable charater outweights the negative impact of a character that is ignored by the scoring by alot.
I'm looking for an elegant way of guaranteeing that nonprintables affect the score worse than any printable character but don't dismiss them completely.