How secure is the Vigenère cipher in file encryption if you encrypt the password first?

Question

I created a basic file encryption method, that seems pretty secure to me (based on an answer below, it seems to be an implementation of the Vigenère cipher) , but one of my colleagues claims that it can be cracked if you know the length of the password.

The way the encryption works, is that it reads the byte value of the file, puts it into an array, and then takes the password, encrypts it with MD5, and subtracts each Unicode value of the current character of the hash that it is looking at (it uses a loop with an index, so that in, for example, a 3 letter password, it subtracts the value of the first character from the first byte, the second character from the second byte, the third character from the third byte, and then back to the first character from the 4th byte, only with the hash instead of the password) then it takes the absolute value of that, and puts that into an array which it returns, and then is written to a new file. How crackable is this type of encryption? It seems like it would be difficult, being that it changes for different files and different passwords, but I'm new to encryption.

In theory, how easy/hard would it be to crack an encryption like this (would be best if you listed how hard it is for longer vs shorter passwords)?

The actual source code is bellow (it's written in java, and the method that returns the encrypted array is the encrypt() method) It may help to explain what I am trying to do a little better than how I explained it above.

public ArrayList<Integer> encrypt(String password, ArrayList<Integer> hash) {
    ArrayList<Integer> encryptedHash = new ArrayList<Integer>();
    ArrayList<Integer> passwordCodeArray = getArrayList(md5(password)); 
    Integer passwordIndex = 0;

    for (Integer currentByte : hash) {      
        Integer currentByteProduct =  currentByte - passwordCodeArray.get(passwordIndex);
        if (currentByteProduct < 0) currentByteProduct += 255;
        encryptedHash.add(currentByteProduct);

        if (passwordIndex == (password.length() - 1)) {
            passwordIndex = 0;
        } else passwordIndex++;
    }

    return encryptedHash;
}  

private ArrayList<Integer> getArrayList(String password) {
    ArrayList<Integer> passwordCodeArray = new ArrayList<Integer>();

    for (int i = 0; i < password.length(); i++) {
        passwordCodeArray.add(password.codePointAt(i));
    }
    return passwordCodeArray;
}

private String md5(String s) {
    try {
        MessageDigest m = MessageDigest.getInstance("MD5");
        m.update(s.getBytes(), 0, s.length());
        BigInteger i = new BigInteger(1,m.digest());
        return String.format("%1$032x", i);         
    } catch (NoSuchAlgorithmException e) {
        e.printStackTrace();
    }
    return null;
}

Edit: I was told that in order to make it a decent encryption, I needed to make the password as long as the document. Since I am using the password's hash to encrypt the file, is there a way for me to encrypt the password in such a way, that it will dynamicly create a hash that is as long as the document?

Answer 1

This is essentially a Vigenère cipher; it's been known for centuries. As for how secure it is, well, it is actually fairly easy to break (unless the key is both as long as the ciphertext, and randomly chosen; however, at that point, if you could remember the key, you could have well just remembered the plaintext). As for your colleague, he's right, and it's also fairly easy to recover the key length as well.

Here's how to attack it (assuming that the key is shorter than the plaintext):

First thing an attacker needs to do is guess the key length. This can be retrieved by looking at the statistics of every Nth character; the N that corresponds to the key length will stand out, because every Nth character will be encrypted in the same way (for example, the ciphercharacter corresponding to a ' ' will be quite common at that step, and will be unlikely to occur at other steps).

Once he has that, then the easiest way to proceed is to do "crib" dragging; that is, guess that a common phrase (the classical one for English is " the ") occurs at a particular point in the text; if that were true, certain bytes of the key would need to take on specific values, and that would then imply the decryption of other parts of the ciphertext; if those decrypted sections make sense, then the original guess of the phrase occuring right there is probably accurate, and now you've just recovered a large portion of text.

Answer 2

My friend (Jerome Kelman (he wanted me to quote him as the author when I posted this)), wrote a program to crack it. (this is assuming the password is exactly 3 characters long though). Here is the code he wrote (it's in java):

import java.util.Scanner;
import java.io.*;
import java.util.ArrayList;

public class Test {

/**
 * @param args
 * @throws FileNotFoundException 
 */

// following basic latin unicode from http://www.ssec.wisc.edu/~tomw/java/unicode.html#x0000
// assuming user can input the chars for 0-127 (which they can't) but i will still use 255 as my num chars
public static void main(String[] args) 
{       
    Scanner fileScanner = null;
    ArrayList<Integer> fullText = new ArrayList<Integer>();
    Integer[] set[] = new Integer[3][];
    int[] numOfChar[] = new int[3][256];
    Integer arrayText[] = new Integer[0];

    try
    {       
        //Step b: Assign the file's input stream to the 
        //Scanner object. 

         FileInputStream fStream = new FileInputStream(new File(args[0]));
         fileScanner = new Scanner(fStream);
         do {
                int currentByte = fStream.read();
                if(currentByte == -1) {
                    System.out.println("broke loop");
                    break;
                }
                fullText.add(currentByte);
            } while (true);
    }
    catch (FileNotFoundException e)
    {
        System.out.println("File not found " + e.getMessage());
        System.exit(-1);
    }   
    catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    // this was me testing what the codePointAt function did.
    // BTW your using ascii i think.... because A-Z is 65 - 90 and a-z is 97 - 122
    /*String s = " !"  + '"' + "#$%'()*+,-./0123456789:;<=>?@ABCDEFGHIGKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz";
    for(int i = 0; i < s.length(); i++)
    {
        System.out.println("\"" + s.charAt(i) + "\" = " + s.codePointAt(i));
    }*/

    arrayText = fullText.toArray(arrayText);

    set[2] = new Integer[arrayText.length/3];

    // assuming hardcoded value of password length 3
    int lengthMod3 = arrayText.length%3;
    switch(lengthMod3){
        case 2:
        {
            set[1] = new Integer[arrayText.length/3 +1];
        }
        case 1:
        {
            set[0] = new Integer[arrayText.length/3 + 1];
            break;
        }
        default:
        {
            set[0] = new Integer[arrayText.length/3];

        }
    }
    set[1] = new Integer[arrayText.length/3];

    for(int i = 0; i < arrayText.length; i++)
    {
        set[i%3][i/3] = arrayText[i];
    }

    for(int j = 0; j < set.length; j++)
    {
        for(int i = 0; i < set[j].length; i++)
        {
            numOfChar[j][set[j][i]]++;
        }
    }

    int shift[] = new int[3];
    String password = "";
    final int valueOfSpace = 32;
    final int maxCharValue = 255;
    int mostFrequent, index = 0;
    int spaces = 0;
    for(int j = 0; j < numOfChar.length; j++)
    {
        mostFrequent = Integer.MIN_VALUE;
        for(int i = 0; i < numOfChar[j].length; i++)
        {
            if(numOfChar[j][i] > mostFrequent)
            {
                mostFrequent = numOfChar[j][i];
                index = i;
            }
        }

        if(index == valueOfSpace)
        {
            shift[j] = valueOfSpace;
        }
        else if(index < valueOfSpace)
        {
            shift[j] = valueOfSpace - index;
        }
        else
        {
            shift[j] = valueOfSpace + maxCharValue - index;
            password += (char)shift[j];
        }
        spaces += numOfChar[j][index];
        System.out.println(index + ": " + numOfChar[j][index]);
    }

    System.out.println('"' + password + '"');
    System.out.println("Number of spaces: " + spaces);
}

}

Answer 3

As to your question about dynamically creating a longer hash: yes, that's easy to do. Once you exhaust your password hash, instead of starting from the beginning, create a new hash by re-hashing your hash! In pseudo-code:

passwordHash = md5(passwordHash + password);

This is basically just an adaptation of the key stretching method.