I asked a question on security.stackexchange, but was told it would be a better fit here:
My question is: Assume I'm an attacker trying to break a weak-but-unknown encryption algorithm. i.e., someone not too bright made a home-brew encryption algorithm, and I'm trying to break it.
BIG FAT DISCLAIMER: Using home-brew security algorithms is very bad for many reasons. Everybody in the security community agrees about it. It's very hard to keep algorithms secret. Please don't answer this question explaining why it's a bad idea to use a home-brew algorithm. I know it's bad. Assume the algorithm is completely unknown.
Okay, now back to my question.
How would you approach a completely mysterious ciphertext when you have no idea what algorithm was used to make it? For me it would look like a meaningless stream of hex, but I'm very curious to know if there's a method to analyze such data and try to break the code.
Is there a way to break it using a single ciphertext? If that's very difficult, I'll be happy to learn how to do it from multiple ciphertexts. I can create more ciphertexts using the same home-brew algorithm, if someone here thinks he can use that to figure out the algorithm.