# How secure is the Vigenère cipher in file encryption if you encrypt the password first?

I created a basic file encryption method, that seems pretty secure to me (based on an answer below, it seems to be an implementation of the Vigenère cipher) , but one of my colleagues claims that it can be cracked if you know the length of the password.

The way the encryption works, is that it reads the byte value of the file, puts it into an array, and then takes the password, encrypts it with MD5, and subtracts each Unicode value of the current character of the hash that it is looking at (it uses a loop with an index, so that in, for example, a 3 letter password, it subtracts the value of the first character from the first byte, the second character from the second byte, the third character from the third byte, and then back to the first character from the 4th byte, only with the hash instead of the password) then it takes the absolute value of that, and puts that into an array which it returns, and then is written to a new file. How crackable is this type of encryption? It seems like it would be difficult, being that it changes for different files and different passwords, but I'm new to encryption.

In theory, how easy/hard would it be to crack an encryption like this (would be best if you listed how hard it is for longer vs shorter passwords)?

The actual source code is bellow (it's written in java, and the method that returns the encrypted array is the encrypt() method) It may help to explain what I am trying to do a little better than how I explained it above.

public ArrayList<Integer> encrypt(String password, ArrayList<Integer> hash) {
ArrayList<Integer> encryptedHash = new ArrayList<Integer>();

for (Integer currentByte : hash) {
if (currentByteProduct < 0) currentByteProduct += 255;

}

return encryptedHash;
}

for (int i = 0; i < password.length(); i++) {
}
}

private String md5(String s) {
try {
MessageDigest m = MessageDigest.getInstance("MD5");
m.update(s.getBytes(), 0, s.length());
BigInteger i = new BigInteger(1,m.digest());
return String.format("%1$032x", i); } catch (NoSuchAlgorithmException e) { e.printStackTrace(); } return null; }  Edit: I was told that in order to make it a decent encryption, I needed to make the password as long as the document. Since I am using the password's hash to encrypt the file, is there a way for me to encrypt the password in such a way, that it will dynamicly create a hash that is as long as the document? - If you decide to downvote this, can you please leave an explanation as to why? – Ephraim Apr 22 '12 at 20:04 I've taken your "please explain downvoting" out of the question as it kinda gets in the way of the question - and the comment should do it. Unfortunately, there's no requirement on anyone to explain their voting - it's totally anonymous, even from moderators. There have been proposals to require comments and they've been declined - have a look here. – Ninefingers Apr 23 '12 at 8:58 add comment ## 3 Answers This is essentially a Vigenère cipher; it's been known for centuries. As for how secure it is, well, it is actually fairly easy to break (unless the key is both as long as the ciphertext, and randomly chosen; however, at that point, if you could remember the key, you could have well just remembered the plaintext). As for your colleague, he's right, and it's also fairly easy to recover the key length as well. Here's how to attack it (assuming that the key is shorter than the plaintext): First thing an attacker needs to do is guess the key length. This can be retrieved by looking at the statistics of every Nth character; the N that corresponds to the key length will stand out, because every Nth character will be encrypted in the same way (for example, the ciphercharacter corresponding to a ' ' will be quite common at that step, and will be unlikely to occur at other steps). Once he has that, then the easiest way to proceed is to do "crib" dragging; that is, guess that a common phrase (the classical one for English is " the ") occurs at a particular point in the text; if that were true, certain bytes of the key would need to take on specific values, and that would then imply the decryption of other parts of the ciphertext; if those decrypted sections make sense, then the original guess of the phrase occuring right there is probably accurate, and now you've just recovered a large portion of text. - would it help if I encrypted the password using MD5 and use the hash to move the bytes around instead of the plain password? – Ephraim Apr 22 '12 at 6:56 @Ephraim no it won't help. This kind of cipher can be easily attacked as long as the length of the key is short compared to the ciphertext, and the plaintext has any recognizable structure. And it's trivially broken in a known plaintext scenario. – CodesInChaos Apr 22 '12 at 8:10 @CodeInChaos - is there a way for me to encrypt the password in such a way, that it will create a hash that is as long as the document? – Ephraim Apr 22 '12 at 21:17 @Ephraim If you want to go that way, you should simply use an existing stream cipher. – CodesInChaos Apr 22 '12 at 21:30 @Ephraim, I take it you want to implement the cipher for pedagogical reasons? If that's the case, have a look at the RC4 Stream cipher, and implement it yourself as an exercise. (It should take you less than 10 minutes.) If you're going for real world use, then dig through the javadocs for the javax.crypto module, and have a look at my answer on stackoverflow. – brice Apr 23 '12 at 10:58 show 1 more comment My friend (Jerome Kelman (he wanted me to quote him as the author when I posted this)), wrote a program to crack it. (this is assuming the password is exactly 3 characters long though). Here is the code he wrote (it's in java): import java.util.Scanner; import java.io.*; import java.util.ArrayList; public class Test { /** * @param args * @throws FileNotFoundException */ // following basic latin unicode from http://www.ssec.wisc.edu/~tomw/java/unicode.html#x0000 // assuming user can input the chars for 0-127 (which they can't) but i will still use 255 as my num chars public static void main(String[] args) { Scanner fileScanner = null; ArrayList<Integer> fullText = new ArrayList<Integer>(); Integer[] set[] = new Integer[3][]; int[] numOfChar[] = new int[3][256]; Integer arrayText[] = new Integer[0]; try { //Step b: Assign the file's input stream to the //Scanner object. FileInputStream fStream = new FileInputStream(new File(args[0])); fileScanner = new Scanner(fStream); do { int currentByte = fStream.read(); if(currentByte == -1) { System.out.println("broke loop"); break; } fullText.add(currentByte); } while (true); } catch (FileNotFoundException e) { System.out.println("File not found " + e.getMessage()); System.exit(-1); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } // this was me testing what the codePointAt function did. // BTW your using ascii i think.... because A-Z is 65 - 90 and a-z is 97 - 122 /*String s = " !" + '"' + "#$%'()*+,-./0123456789:;<=>?@ABCDEFGHIGKLMNOPQRSTUVWXYZ[\\]^_abcdefghijklmnopqrstuvwxyz";
for(int i = 0; i < s.length(); i++)
{
System.out.println("\"" + s.charAt(i) + "\" = " + s.codePointAt(i));
}*/

arrayText = fullText.toArray(arrayText);

set[2] = new Integer[arrayText.length/3];

// assuming hardcoded value of password length 3
int lengthMod3 = arrayText.length%3;
switch(lengthMod3){
case 2:
{
set[1] = new Integer[arrayText.length/3 +1];
}
case 1:
{
set[0] = new Integer[arrayText.length/3 + 1];
break;
}
default:
{
set[0] = new Integer[arrayText.length/3];

}
}
set[1] = new Integer[arrayText.length/3];

for(int i = 0; i < arrayText.length; i++)
{
set[i%3][i/3] = arrayText[i];
}

for(int j = 0; j < set.length; j++)
{
for(int i = 0; i < set[j].length; i++)
{
numOfChar[j][set[j][i]]++;
}
}

int shift[] = new int[3];
final int valueOfSpace = 32;
final int maxCharValue = 255;
int mostFrequent, index = 0;
int spaces = 0;
for(int j = 0; j < numOfChar.length; j++)
{
mostFrequent = Integer.MIN_VALUE;
for(int i = 0; i < numOfChar[j].length; i++)
{
if(numOfChar[j][i] > mostFrequent)
{
mostFrequent = numOfChar[j][i];
index = i;
}
}

if(index == valueOfSpace)
{
shift[j] = valueOfSpace;
}
else if(index < valueOfSpace)
{
shift[j] = valueOfSpace - index;
}
else
{
shift[j] = valueOfSpace + maxCharValue - index;
}
spaces += numOfChar[j][index];
System.out.println(index + ": " + numOfChar[j][index]);
}

System.out.println("Number of spaces: " + spaces);
}

}

-
passwordHash = md5(passwordHash + password);
`