I don't think it's possible to give any specific and concrete answer without seeing the actual ciphertext (which in turn would make your question borderline off-topic here). Nonetheless, I'll try to give a general answer below, outlining how you might be able to approach the problem.
Modern secure encryption algorithms all but invariably produce output that looks like uniformly distributed random bytes. In particular, they do not natively output printable ASCII text.
Thus, if you see ciphertext that looks like readable ASCII, there are basically two possibilities:
after encryption, some binary-to-text encoding has been applied to it, or
it's not actually encrypted with a secure modern cipher.
In the first case, your first task will be identifying the encoding scheme. The most common one is probably Base64, but plenty of other choices exist too.
Ideally, you'd like to find a specification describing the scheme, or the code that encodes and/or decodes it. Failing that, the first thing you should check is the set of characters used in the encoded ciphertext. Often, this alone may be sufficient to identify the encoding scheme. Note that the beginning and/or the end of the data may sometimes contain extra padding or metadata tags that aren't actually encoded with the same scheme as the encrypted data itself, and which may use different characters.
You should also check how often each character occurs in the encoded ciphertext. Binary ciphertext (which looks like random bytes) encoded with a base-conversion scheme like Base64 will typically feature all of the actual data-encoding characters with roughly equal¹ frequencies. Characters that appear with significantly different frequencies may be padding or formatting characters, like =
and newlines in Base64 or the length markers at the beginning of each line (usually M
, except for the last line) in uuencoding, or they might only occur in metadata.
Alternatively, if some characters appear significantly more often than others, that may be a sign that you're actually dealing with a variable-length encoding like URL-encoding or MIME quoted-printable, or that the unencoded bytes are not uniformly distributed (which may indicate that they're not really encrypted after all).
Note that, if the (supposed) encryption is any good, then you might still be out of luck even if you do manage to decode the ASCII text back into binary. Modern encryption schemes are generally designed according to Kerckhoff's principle, which says that a cryptosystem should remain secure even if the attacker knows everything but the key.
Even if the encryption scheme used is broken and vulnerable to attacks, with just a bunch of captured messages you're basically restricted to ciphertext-only attacks, which are the weakest kind of attacks possible — and that's assuming that you even know how the encryption scheme works, which may not be the case here.
It's also possible that the data you're trying to analyze is not properly encrypted at all, at least not in the modern sense. It might be text encrypted with some classical cipher (i.e. something from the era before byte-based computerized ciphers like DES became commonplace) or with some weird custom home-brew encryption scheme that might be more or less (usually less) deserving of the name "encryption".
It's also possible that the data is not really encrypted at all, but merely obfuscated to make it harder to read. The difference between obfuscation and encryption is basically Kerckhoff's principle, as mentioned above: actual encryption is designed to remain secure even if the attacker knows everything about the system except the key, whereas an obfuscation scheme may not even have a key, and simply relies on the attacker not being smart enough to figure out how it works.
One common place where obfuscation is often found is in computer games, which may "encrypt" (more properly speaking, obfuscate) their data and save files to deter players from modifying them in order to cheat. Sometimes, such obfuscation schemes may be designed to keep the obfuscated text as printable ASCII, e.g. to reduce the chance of accidental data corruption when editing partially obfuscated files.
Of course, in practice, these categories can blend together: many classical encryption schemes that were considered reasonably secure in their time are nowadays little more than puzzles for amateur cryptographers armed with computers and modern codebreaking techniques, and similarly, many home-brew encryption schemes developed by amateurs with little if any knowledge of modern cryptography also amount to little more than mere obfuscation. In any case, the techniques to attack all such schemes are similar:
- if possible, try to analyze the code that encodes or decodes the data;
- failing that, try to obtain and/or generate ciphertexts corresponding to known plaintexts with small differences, and see if the differences between the ciphertexts reveal a pattern;
- also, if possible, try to submit slightly modified ciphertexts for decryption and see if there's any pattern in how changes to the ciphertext affect the plaintext;
- if none of the above are possible, try to simply guess what the scheme might look like, based on resemblance to other common and/or standard encoding or obfuscation schemes.
You might also find the answers to these related questions useful:
(Yes, the top answers to both are mine, because I was lazy and only looked through my own contributions. If you can think of other related earlier questions with good answer, feel free to suggest them in the comments.)
Also, here's a practical example of ciphertext-only reverse engineering of (double) binary-to-ASCII encoded data that I posted to SO a while ago.
1) As a rule of thumb, two character counts may be considered "roughly equal" if their difference is no more than a couple of times the square root of their average. This is because, for a string of independently chosen random characters, the number of times each character occurs in the string approximately follows a Poisson distribution, which has a standard deviation equal to the square root of its mean.
Substitutional cipher
works out for the above encryption scheme. If you know the key you can do the reverse substitution to find out the plaintext. <br/> $\endgroup$