Why did the NSA create SHA?

Question

New to cryptography, I'm trying to learn the program Veracrypt, and I'm now trying to understand the history of hash functions and how it applies to Veracrypt.

So this is what I understand so far, the NSA made SHA-2, which is one of the hash functions that Veracrypt can use for its "key derivation function," and the NSA made SHA-2 because they wanted an improvement on SHA-1, and they made SHA-1 because they wanted an improvement on SHA-0.

What I don't understand is, why did NSA make SHA-0 in the first place? And when did hash functions start being used for password storing and "key derivation functions"?

Did hash functions start with the need for network security for verifying packages? Did they start with the need for hashing passwords or both, or what was the deal?

Answer 1

This question is asking a bunch of things, I will try to sort things out. Hash functions were around long before SHA-0. The popular hash before SHA-1 was MD5 which shares a lot with SHA0–2 but there were many others.

The SHA family were designed as part of a project trying to supply a full set of standardized cryptographic tools providing 80-bit security: the Capstone project.

They published SHA (now named SHA-0) and immieiatly retracted it due to a serious flaw and published SHA-1 instead. As computers grew stronger and cryptanalysis better,we needed a stronger hash functions. 80 bits of security doesn't seem all that great, so without being too creative the SHA2 family came out. This is essentially a supersized SHA-1.

Fast forward a few years, SHA-1 is still the most popular hash function, serious weaknesses emerge and computers keep getting faster. The Merkle-Damgård construction also loses favor following attacks on the construction itself.

Following the success of the AES competion, the SHA-3 competition is launched, and eventually the SHA-3 algorithm is standardized using a very different sponge construction and is hoped to remain secure for many years.

Key derivation functions are not the same as hash functions, and password-based key derivation functions are not the same as general-purpose key derivation functions. Password-based key derivation functions are very similar to password-based hashing functions. Using something like PBKDF2 we can convert a general-purpose hash function to a password hashing function. One should NOT use a general-purpose hash function as is to hash passwords, for the simple reason general purpose hash functions (such as SHA*) are way too fast, and password-based KDF or hashing are required to be slow.

Early hash functions were used for message digest, not sure exactly what came when. MD2 is from 1989, I'm not sure what came before it. For password hashing, the earliest I know is Unix crypt based on the DES cipher from 1973. You can look at as a hash function, but it didn't do compression: it was 56 bits and 64 bits out.

Answer 2

The NSA created SHA (quickly replaced by SHA-1 to fix a security issue that remained unpublished for years) because they/the US government needed a 160-bit hash function, so that it has 80-bit security against collision-resistance, in particular towards use in the DSA signature system of FIPS 140.

In this usage, a hash functions condenses a possibly large piece of information $m$ (e.g. a 10MB file) into a smaller, easier to manipulate piece of information $h=H(m)$ (20 bytes for SHA-1), wich is what really gets signed. The hash must be collision-resistant, so that an adversary can't come with two distinct messages $m$ and $m'$ with $H(m)=H(m')$, which would allow the adversary to obtain a signature for $m$ from a legitimate signer, and pass it as the signature for $m'$.

SHA-1 uses principles similar to the earlier 128-bit MD5, but with a more regular design. Also it is big-endian, likely because that's customary (to the point of being unstated) in NIST crypto standards including DES/FIPS 43.

Answer 3

Note, this answer is short and a mix of historical information and opinion, covering only the first half of your question, as the 2nd half deserves its own.

SHA0 and the Secure Hash Standard was created because the NSA was asked to develop cryptographic standards meeting an 80-bit security level for government and the American public, which would be provided for free (in comparison to say, RSA Security (the company)), but require the government to be in charge of the keys.

SHA1 was a revision to SHA0 after it was determined there was a flaw in the compression function which reduced the security.

The project failed to take hold, but the hash and signature algorithm became very successful standards as they were essentially independent of the key management and symmetric encryption standards (like Skipjack) of the project.

Later, during the process leading up to the AES competition, it became clear that hash algorithms matching the security level of the new cipher would be needed, and the NSA was tasked with creating the new algorithms, which followed the design of SHA1 but were significantly stronger with a more complex design.