How can a hash function be suitable for hashing passwords, but not for deriving encryption keys, and vice versa?

Question

Not being a specialist in this field, I am nevertheless constantly trying to keep up with which methods are being considered secure for storing passwords and for encrypting data.

During my research and desperate tries to extract the key points which are important for practice :-), I am reading often that certain hashing functions are suitable for password hashing, but not for deriving encryption keys, and vice versa. Since I absolutely can't get this, but have read it too often, I have to seek help.

For the following, let's forget about all complicated things (rainbow tables, salting, side channel attacks and so on). Furthermore, I am over-simplifying things in the following sections. But the question is so basic that I think that the over-simplification does not hurt in this case.

My understanding problem is this:

An OS should not save a password in clear text for obvious reasons. So a hashing function (MD4 (yes, Windows still uses it), SHA-2, scrypt, ...) is applied to the password, and the hash is stored on disk instead of the password.

If an attacker wants to crack the password, he has to get the hash, and then has to take every possible input string, apply the hashing function to it, and see if the hash of the input string is the same as the hash of the password the OS had stored.

This means that the hashing function should be slow (in terms of cycles), should be hard to parallelize, and should be memory intensive to make brute force cracking (even with specialized hardware) as expensive and time consuming as possible.

On the other hand, as far as I have understood, when deriving an encryption key from a password, nearly the same method is applied. The difference is that the hash of the password is not stored on disk, but is used directly as encryption key for some symmetric encryption method (like AES) (over-simplified, but shows the general idea, right?).

Now, if an attacker would like to break the encryption, he also would have to test every possible password, apply the hashing function to it, and then decrypt the encrypted data with the hash and test if the result contains something meaningful. Thus, as with password hashing, the hashing function should be very hard to parallelize, memory and cycle intensive so that the attacker could test only a few encryption keys (= hashes) per second.

So the requirements to a hashing function for hashing passwords seem to be the same as the requirements to a hashing function whose output is used as encryption key. For the life of me, I don't get why some hashing functions are considered suitable for the one, but not the other, and vice versa.

Could somebody please explain this in simple words, taking into account that I have a mathematical background in general (if urgently needed :-)), but no background in cryptography's mathematics?

Answer 1

So the requirements to a hashing function for hashing passwords seem to be the same as the requirements to a hashing function whose output is used as encryption key

This is correct only when the encryption key is derived from a password.

The similarity you observe is due to the fact that a password is the input, and passwords tend to be relatively easy to guess.

Other circumstances where encryption keys are derived do not require key stretching, because the input is already an unguessable (psuedo)random value.

As an example, when using a key agreement scheme the resultant shared secret is typically hashed with a fast cryptographic hash function such as SHA256 to protect the private keys in the case that the shared secret key is compromised. Since the value is already random and large enough to be unguessable, there is no need to apply a slow hash to prevent the adversaries ability to guess it.

tl;dr

If the input is a password, then the hash needs to be slow to prevent an adversaries ability to perform exhaustive search on the entire password space.

If the input to the hash is already a large random value, then slow hashing is not required.

Answer 2

In fact, a hash function designed for derivation of encryption keys is typically plain unsuitable for directly hashing passwords; and sometime vice versa.

For example of the former, a standard hash function like SHA-256 is perfectly fine for deriving an encryption key in some contexts, e.g. from a larger but imperfect shared secret established by Diffie-Hellman key exchange. And HMAC-SHA-256 is perfectly suitable for deriving multiple encryption keys from any large master key and a key index. However, neither SHA-256 nor HMAC-SHA-256 are directly usable for hashing passwords, for the reason stated in the question: they are too fast. As an aside, SHA-256 has no explicit salt input, thus requires a modification.

In the other direction: a key derivation function should produce a result that is close to a uniformly distributed bitstring. That requirement does not exist for password hash functions (or only mildly, so as to not wildly enlarge the password file). Some passwords hashes accordingly intentionally have a low-entropy section in their output (like a version, iteration count, or a copy of the salt), or/and have their output formatted as ASCII text. Independently, their input could be restricted in some way (that's the case for bcrypt: most implementations have limited input length, or silently truncate large input; some sanitize the password input before processing). And quite a few password hashes consider that a zero byte in the password input marks its end, which would be very bad for a general-use key derivation function.

On the other hand, a good password hash function which cost can be lowered to negligible by a parameter, that process all input as arbitrary bits, and has a near uniformly distributed output (like PBKDF2), should be usable for key derivation.