Cryptography Stack Exchange is a question and answer site for software developers, mathematicians and others interested in cryptography. It only takes a minute to sign up.
Sign up to join this community
They are asking me to deliver encrypted text in ISO/IEC 8859-1. I have always encoded it to hexadecimal or base64. This text will be sent in a file with plaintext and encrypted text. In my opinion it won't work because of the non-visible characters.
I also wonder if there are characters that exist in hexadecimal and not in ISO/IEC 8815-1 (Western-Latin encoding). Am I right in assuming so?
Binary ciphertext may contain any byte value. However, character encoding's such as ISO/IEC 8859-1 don't allow all bytes to be used. Furthermore, some of the characters defined (in the range 00 to 1F) are so called control characters that are not "printable", i.e. they don't show up as characters in text. Because of this you need to encode ciphertext if you need to store or transmit it as text.
Hexadecimals uses the characters 0-9 and a-f and / or A-F. As such they can be used in any text document. Note that hexadecimals are not binary themselves, although they can be used to represent a byte with any value using precisely two hexadecimal digits 00 to FF. I think the main issue you are having is not understanding the difference between binary and the representation of the binary (bytes) using hexadecimals.
Base 64 uses the characters A-Z, a-z, 0-9 as well as / and + Finally it uses the padding character =. It also always fits in any text, UTF-9 or ISO/IEC 8859-1. It's no different than hexadecimals in that respect, although it is of course more space efficient but less "readable". It is less readable because it is harder to calculate the size and determine which bits are set from base 64 encoding compared to hexadecimals).
So the idea is to:
During decryption this is performed in reverse: hex / base 64 decoding, decryption and then possibly character decoding to get back the original text.
In my opinions, encoding of ciphertext is performed much too often. Generally it is better left as binary. For instance, you can store it as file, VARBINARY in SQL databases or send it as binary over HTTP without major issues.
Both hex and base 64 encoded ciphertext are not very well suited to be directly embedded in human readable text Some kind of way is required to distinguish it from "normal" text after all. PEM encoding uses header and footer lines for that reason.
There are even more formal ways of doing this: XML and - to a less extend - JSON. With XML it is possible to use XML-enc to encrypt specific parts of an XML document, which is then usually stored as UTF-8. For JSON the JWE standard can be used to accomplish the same thing.
When using standard encryption algorithms like AES-GCM, ciphertext is bytes that are approximately uniformly distributed among the 256 possible byte values. Since ISO/IEC 8859-1 allows like 191-byte values out of 256, it's extremely unlikely that any practical ciphertext will entirely consist of these 191 bytes, as would be the case if some appropriate Binary-to-text encoding was used, including Base64. In the rest, I'll assume it is not used such encoding.
The curious case is that within that file there are two lines in plain text (header and footer) and the rest encrypted.
That does not make it impossible for decryption of such files to works. Many modern file transfer methods, including USB thumb drives, attachments to email or messaging, network shared volumes, cloud storage, UNIX pipes, windows pipes opened in binary mode… will happily transfer any (contiguous and single fork/stream) file with all its data (if not metadata like file name, date and security attributes) unchanged. Thus the problem of allowing decryption boils down to locating the ciphertext in the file.
That can be dealt with with conventions. One possibility is that ciphertext starts after a header of fixed size in bytes that the decryption skips. Another is that it starts after the first occurrence of a fixed text sequence, e.g. <-- ciphertext on next line --> followed by a single LF character. The end of ciphertext can be dealt with by embedding the ciphertext length at the start of ciphertext. That can work if the file is correctly prepared, e.g. by a correctly written program; or by manually concatenating a header file, the ciphertext, and the footer file using bash on a Unix system; or using copy /b of cmd.exe on Windows.
Things that go wrong when that's attempted include: the file is opened and mistakenly saved with a text editor (perhaps to remove the header), and that particular text editor ruins the ciphertext (many do); the sender used CR LF as end-of-line and the decryption is unaware of that; the file is piped to a console display emulating some glass terminal (e.g. a VT100 itself emulating an ASR-33), which emits silly noises, locks, or act strangely because it entered an obscure mode.
