Why initialize SHA1 with specific buffer?

Question

SHA-1 is initialize with a specific buffer:

h0 = 0x67452301
h1 = 0xEFCDAB89
h2 = 0x98BADCFE
h3 = 0x10325476
h4 = 0xC3D2E1F0?

Why?

Answer 1

The initial values to a Merkle–Damgård type hash function are essentially the plaintext to a block cipher, with the input to the hash function becoming the key. The maximum length of the hash is determined by the amount of bits of initial value. Five 32-bit words gives SHA1 a state size and maximum output of 160 bits. In order for an MD type hash function to output a consistent value, the initial values must be constant.

The first 4 values are those used in MD4 and MD5. They are represented in big-endian format. If represented in little-endian, it can be seen it is a simple 4-bit counter starting at 0000, maxing out at 1111, then stepping back in reverse, in groups of 32 bits:

0111 0110 0101 0100 0011 0010 0001 0000 = LE 0x76543210 = BE 0x67452301
1111 1110 1101 1100 1011 1010 1001 1000 = LE 0xFEDCBA98 = BE 0xEFCDAB89
1000 1001 1010 1011 1100 1101 1110 1111 = LE 0x89ABCDEF = BE 0x98BADCFE
0000 0001 0010 0011 0100 0101 0110 0111 = LE 0x01234567 = BE 0x10325476

The final big-endian value is an incrementing counter from 1100 for odd 4-bit groups, and a decrementing counter from 0011 for even groups.

These are called Nothing Up My Sleeve numbers, and are chosen as a way of proving the constants are not specially chosen, such as to provide a backdoor to the algorithm. As per figlesquidge's comment as to why these are not simply 0 bits, the average hamming weight of the entire set of initial values is 0.5, half the bits are 1 and half are 0.

The round constants for SHA-1 were also chosen not to be "special", but are generated quite differently. They take the square root of a number in binary, then the first 32 bits, including those before the decimal place, become the round constant. This is the same method used to generate the two round constants for MD4.

0x5A827999 = $\sqrt2$
0x6ED9EBA1 = $\sqrt3$
0x8F1BBCDC = $\sqrt5$
0xCA62C1D6 = $\sqrt{10}$

The design criteria for the first 2 round constants was specified in the original MD4 paper as well as the RFC. The initial value choice was never defined nor explained, most likely because of the obviousness of the pattern. The choices of initial values and round constants were not explained in FIPS-180-1 and later revisions. It is interesting that they chose 10 to generate the final round constant rather than 7, as it is not prime.

One possible explanation for the choice of $\sqrt{10}$ instead of $\sqrt7$ might be that $\sqrt{10}$, along with the other three constants, are the only square roots given in the "Tables of Numerical Quantities", Appendix A Table 2 (ocatal representations), in Donald Knuth’s The Art of Computer Programming, Vol. 2: Seminumerical Algorithms, Second Edition (1981), p. 660, which is referenced in the original RFC 1320 to MD4 (on p. 4) as the source for the constants. In the Third Edition (1997) of the book, these are found on page 727.

SHA-2 (256 and 512 bit digests) in comparison uses the fractional component (after the decimal place only) of the square root of sequential primes 2 to 19 for initial values, and the fractional component of the cube roots of sequential primes 2 to 311 for round constants.

Answer 2

If the question is 'why are those variables initialized at all', well, that's because those values will be used as inputs to the initial SHA-1 compression function; they must be consistent values; otherwise, the resulting hash will be different (depending on what values were used).

If the question is 'why are those specific values used (rather than other values)', I don't believe NIST ever formally published that reasoning. However, those values in hexadecimal follow a pattern; this implies that they were unlikely to be chosen with some special property in mind.