So my question is how do I continue from here? How do I figure out what encryption method is being used?
It sounds like you, in essence, have black box access to something that allows you to give it a key and a file, and it returns an encryption of that file (after adding some file format around it) using the supplied key.
Your goal is to figure out how the encryption works. There may be legal issues you want to consider before actually doing this, but legal issues aside, you have many options.
Your best option would be to reverse engineer the code that is doing the encryption. Is this software running on your computer or some sort of hardware device? In either case, you should be able to get the code and reverse engineer it by putting it in a disassembler and doing static analysis, or putting it in a debugger and doing dynamic analysis.
Let's pretend, however, that you do not have access in any way to the code. Maybe this black box is running on a remote server that you do not have access to. Your goal then is to figure out what cipher are they using (e.g., AES, DES, 3DES, RC4, etc) and what mode of encryption is being used.
You can start to do this by asking a few questions and using your black box access to start answering the questions.
Is the encryption deterministic? Or in other words, given the same plaintext (or file) and key, does it always return the same result?
If I flip a single bit in the plaintext, how does the resulting ciphertext change?
What happens if you feed a ciphertext in as the plaintext, do you get the original plaintext back? In other words, is the encryption mode the inverse of itself?
By flipping bits at different places, can you figure out the block size? Depending on the mode used, where you flip bits can change what blocks actually change?
Something else to investigate is why does the ciphertext appear to always be double the size of the plaintext? When you pass in
123, which as ASCII would have 3 bytes, why do you get 6 bytes back? Is there some sort of encoding going on?
This is how I would suggest continuing from where you are.