What is the use of encoding a hash output?

Question

In many applications, the MD5 algorithm is used which produces a 128-bit output which is represented as a sequence of 32 hexadecimal digits. This output is further encoded using a base62 or base64 scheme.

I wanted to understand what is the need for further encoding the hash value. Why could we not use the same hash value and store it in our databases?

Is it done to reduce space consumption?

Based on my understanding, the MD5 hash output is 16 bytes. When we encode it using the base64 scheme and produce 8 char long string, it will be 8 bytes.

Answer 1

What is the need for further encoding the hash value?

Representing the hash as a string of characters, without increasing the size too much. This is known as Binary-to-text encoding. It is commonly used for cryptographic data (hashes, ciphertexts..), because that can contain arbitrary sequences of bits (or arbitrary sequences of arbitrary bytes), and some data communication, storage or display means can only handle characters in some alphabet.
_{This is not to be confused with encoding characters into bytes, as needed to hash or encipher text.}

I will make the explanations and examples with the MD5 hash of a 26-ASCII-character message given in example in RFC 1321 appendix A.5:

MD5 ("abcdefghijklmnopqrstuvwxyz") = c3fcd3d76192e4007dfb496cca67e13b

The hash is not the 32-character sequence c3fcd3d76192e4007dfb496cca67e13b. Depending on perspective it's either sixteen 8-bit bytes, or a 128-bit sequence (or bitstring) which representation with 0 (resp. 1) for a bit clear (resp. set) is (with translation of bytes to bits by big-endian convention¹):

11000011111111001101001111010111011000011001001011100100000000000111110111111011010010010110110011001010011001111110000100111011

The 32-character sequence c3fcd3d76192e4007dfb496cca67e13b is a representation of the hash per lowercase big-endian hexadecimal. That was obtained from a representation of that hash as 16 bytes (or equivalently 16 octets) per this code C code (from the RFC):

/* Prints a message digest in hexadecimal.
 */
static void MDPrint (digest)
unsigned char digest[16];
{
  unsigned int i;
  for (i = 0; i < 16; i++)
    printf ("%02x", digest[i]);
}

^{Note: In the above, unsigned char is a byte representing an integer in range [0…255] that encodes 8 bits of the hash. Conversion to characters is in the printf function, with "%02x" specifying at least 2 lowercase hexadecimal characters.}
_{Note: Each group of 4 bytes is the little-endian representation of a computer word, but that's a consideration internal to MD5.}

Beside the aforementioned representations as 16 bytes, and 32 lowercase (big-endian) hexadecimal characters, there are other common representations, including

• C3FCD3D76192E4007DFB496CCA67E13B : 32 characters, uppercase (big-endian) hexadecimal
• w/zT12GS5AB9+0lsymfhOw== : 24 characters, Base64 per RFC 4648
• w_zT12GS5AB9-0lsymfhOw== : 24 characters, Base64 with Filename Safe Alphabet
• w/zT12GS5AB9+0lsymfhOw : 22 characters, Base64 with pad suppressed

If is possible to convert from a text (equivalently, character) encoding back to the hash. The same hexadecimal decoding method can handle uppercase and lowercase. The same Base64 decoding method can handle common alphabets and make pad optional.

There are other, more seldom-used, sometime more compact Binary-to-text encodings. See this for why Base64 is most popular.

When we encode (a 16-byte MD5 hash) using the base64 scheme and produce 8 char long string, it will be 8 bytes.

No. Each Base64 character encodes 6 bits (except the last non-pad character which can encode 2, 4 or 6 bits; and final pad characters, if any). Therefore, per Base64 encoding, a 128-bit hash requires at least ⌈128/6⌉ = 22 characters, plus pad if any.

Depending on computer language and options, each character in the above may in turn use several bytes. For example, each such character in a C string is represented by 1 byte, in a Java 8 String 2 bytes (see comment for later editions). Also, there is size overhead for each object, which may not be negligible.
_{In a Python string, starting with version 3.3 or so of Python, each character in the above encodings by default uses 1 byte. In previous versions it used 2 or 4, depending on Python environment; refer to PEP 393.}

---

¹ ^{MD5 uses little endian convention for translation from input bytes to 32-bit words, and back to bytes on output, but big-endian convention for translation between input bits and bytes. Thus by symmetry I assume big-endian convention for translation between output bytes and bits.}