Cryptography Stack Exchange is a question and answer site for software developers, mathematicians and others interested in cryptography. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

E.g., in Python's hashlib, update() basically merges in string buffer functionality into the object. Several other digest APIs I've seen have this pattern.

Considering this adds functionality that arguably doesn't belong here, why is this pattern popular?

share|improve this question
Why shouldn't it be there? What alternative do you propose? – CodesInChaos Jul 12 '13 at 21:06
@CodesInChaos The alternative would be to accept the entire input in digest(). There's nothing about the design of hash algorithms that requires feeding the input in piece by piece. This seems to run against the – Steve Clay Jul 13 '13 at 18:19
Which requires putting the full message into a continuous piece of address space. So it needs unnecessary copying in some cases and doesn't fit at all for large inputs. With an update() based API I can easily hash a multi gigabyte file on a modest computer. – CodesInChaos Jul 13 '13 at 19:02
@CodesInChaos Technically on 64-bit modern operating systems you can map an arbitrarily large file into the virtual address space and let the OS stream the data in and out but I agree it's much better to have this functionality built into the API itself, so it works everywhere the same (and works everywhere period). Plus it isn't that hard to implement.. – Thomas Jul 14 '13 at 20:11
There are some hash functions (e.g. SHA-512) that allow around 4.25*10^37 bytes of input. You could not map such input into the address space of modern OS. (Then again, hashing large input, such as 2^64 bytes takes long time even with HW acceleration and thus it is performed rarely.) – user4982 Jul 19 '13 at 7:16
up vote 11 down vote accepted

Being able to generate hashes incrementally (by presenting parts of the string being hashed) is there for basically two reasons:

  • It's commonly useful in practice. Quite often, we don't have the entire string in one contiguous segment; instead, we often have parts laying around separately. For one simple example of this, consider the HMAC function; the inner hash is some preprocessed keying material, followed by the message being MAC'ed. We could create a temporary buffer consisting of the HMAC inner key ("ipad") and the message, but why?

  • It's actually easy to do with our current hash function (including SHA-1, all the SHA-2 versions and SHA-3). You appear to think that the Python hashlib library copies the parts of the message being hashed into a buffer, and then runs the hash on the entire buffer. That's extremely unlikely to be how they implement it; instead, it is easy (with a small amount of side memory) to have the hash process the message in pieces.

So, the answer is: they provide this functionality because people find it useful, and its easy for the library to do.

share|improve this answer
It also makes the hash implementation run in constant memory, which I think is actually the real reason it's done this way - you can convert a constant-memory scheme into a more convenient but less flexible "all at once" scheme, via a wrapper implementation, but doing the opposite is harder. So it's a "lowest common denominator" kind of deal. – Thomas Jul 13 '13 at 7:52
Yep. hashlib uses update() in their HMAC implementation: – Steve Clay Jul 13 '13 at 18:29
I think the most likely explanation for the API is to support intermediate hashes ("I need separate hashes for i1, i1 + i2, and i1 + i2 + i3"):… – Steve Clay Jul 13 '13 at 18:42

To encourage the culture of avoiding dissemination of confidential data across multiple copies of "the same object" in runtime memory:

instead of:

string password = copyPasswordFromAnotherString(inputArg);  
string salt = generateRandomSalt();  
string tohash = password+salt;  


string salt = generateRandomSalt();  

Also (as probably meant by poncho), to help use cases involving hashes built from data obtained incrementally (i.e.: hashing a large confidential document without having copies of it in runtime memory)

share|improve this answer

I would argue that hash APIs wrap string buffer creation because leaving buffer creation up to the developer is a flaw waiting to happen. Consider hashing a set of words without inserting the word lengths in between - this construction could be exploited by an attacker to find collisions.

share|improve this answer
I don't get your point. What are you comparing update to? – CodesInChaos Jul 12 '13 at 21:22
Not having update() at all, and only offering a function hash() that takes an already-created buffer and computes the hash. – pg1989 Jul 12 '13 at 21:23
It depends on how the update() function is implemented. Maybe the library automatically prepends the length. My point is that an explicit update() function makes that possible. – pg1989 Jul 12 '13 at 21:37
I still don't get your point. An update function that prepended the length would be broken since now calling update twice isn't equivalent to calling it once on a combined buffer. – CodesInChaos Jul 12 '13 at 21:43
Hmm, you're right. – pg1989 Jul 12 '13 at 21:51

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.