1
$\begingroup$

In many applications, the MD5 algorithm is used which produces a 128-bit output which is represented as a sequence of 32 hexadecimal digits. This output is further encoded using a base62 or base64 scheme.

I wanted to understand what is the need for further encoding the hash value. Why could we not use the same hash value and store it in our databases?

Is it done to reduce space consumption?

Based on my understanding, the MD5 hash output is 16 bytes. When we encode it using the base64 scheme and produce 8 char long string, it will be 8 bytes.

$\endgroup$
  • 1
    $\begingroup$ "which produces a hexadecimal output of length 32." It does not. It produces an output of 128 bits. $\endgroup$ – Maeher Jan 9 at 7:05
  • $\begingroup$ @Maeher Both of us are correct. Please visit crypto.stackexchange.com/a/76891/75677 $\endgroup$ – Jainabhi Jan 9 at 7:08
  • $\begingroup$ No. Just no. The output of MD5 are 128 raw bits. Nothing else. Those may be encoded in hexadecimal if you so choose. But the MD5 algorithm does not do that. $\endgroup$ – Maeher Jan 9 at 7:10
  • $\begingroup$ Let me correct myself. The 128-bit (16-byte) MD5 hashes are typically represented as a sequence of 32 hexadecimal digits. $\endgroup$ – Jainabhi Jan 9 at 7:13
  • 1
    $\begingroup$ Either the the raw output bytes are represented as hex, or they are directly encoded from raw output into base64/58/62 etc, it doesn't go raw->hex->b64. $\endgroup$ – SamG101 Jan 9 at 10:43
5
$\begingroup$

What is the need for further encoding the hash value?

Representing the hash as a string of characters, without increasing the size too much. This is known as Binary-to-text encoding. It is commonly used for cryptographic data (hashes, ciphertexts..), because that can contain arbitrary sequences of bits (or arbitrary sequences of arbitrary bytes), and some data communication, storage or display means can only handle characters in some alphabet.
This is not to be confused with encoding characters into bytes, as needed to hash or encipher text.

I will make the explanations and examples with the MD5 hash of a 26-ASCII-character message given in example in RFC 1321 appendix A.5:

MD5 ("abcdefghijklmnopqrstuvwxyz") = c3fcd3d76192e4007dfb496cca67e13b

The hash is not c3fcd3d76192e4007dfb496cca67e13b. It really is a 128-bit sequence (or bitstring) which representation with 0 (resp. 1) for a bit clear (resp. set) is:

11000011111111001101001111010111011000011001001011100100000000000111110111111011010010010110110011001010011001111110000100111011

The 32-character sequence c3fcd3d76192e4007dfb496cca67e13b is a representation of the hash per lowercase big-endian hexadecimal. That was obtained from a representation of that hash as 16 bytes (or equivalently 16 octets) per this code C code (from the RFC):

/* Prints a message digest in hexadecimal.
 */
static void MDPrint (digest)
unsigned char digest[16];
{
  unsigned int i;
  for (i = 0; i < 16; i++)
    printf ("%02x", digest[i]);
}

Note: In the above, unsigned char is a byte representing an integer in range [0…255] that encodes 8 bits of the hash. Conversion to characters is in the printf function, with "%02x" specifying at least 2 lowercase hexadecimal characters.
Note: Each group of 4 bytes is the little-endian representation of a computer word, but that's a consideration internal to MD5.

Beside the aforementioned representations as 16 bytes, and 32 lowercase (big-endian) hexadecimal characters, there are other common representations, including

  • C3FCD3D76192E4007DFB496CCA67E13B : 32-characters, uppercase (big-endian) hexadecimal
  • w/zT12GS5AB9+0lsymfhOw== : 24 characters, Base64 per RFC 4648
  • w_zT12GS5AB9-0lsymfhOw== : 24 characters, Base64 with Filename Safe Alphabet
  • w/zT12GS5AB9+0lsymfhOw : 22 characters, Base64 with pad suppressed

If is possible to convert from a text (equivalently, character) encoding back to the hash. The same hexadecimal decoding method can handle uppercase and lowercase. The same Base64 decoding method can handle common alphabets and make pad optional.

There are other, more seldom-used, sometime more compact Binary-to-text encodings. See this for why Base64 is most popular.

When we encode (a 16-byte MD5 hash) using the base64 scheme and produce 8 char long string, it will be 8 bytes.

No. Each Base64 character encodes 6 bits (except the last non-pad character which can encode 2, 4 or 6 bits; and final pad characters, if any). Therefore, per Base64 encoding, a 128-bit hash requires at least ⌈128/6⌉ = 22 characters, plus pad if any.

Depending on computer language and options, each character may in turn use several bytes. For example, each character in a C string is represented by 1 byte, in a Java String 2 bytes. Also, there is size overhead for each object, which may not be negligible.
In a Python string, starting with version 3.3 or so of Python, each character in the above encodings by default uses 1 byte. In previous versions it used 2 or 4, depending on Python environment; refer to PEP 393.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.