6
$\begingroup$

E.g., in Python's hashlib, update() basically merges in string buffer functionality into the object. Several other digest APIs I've seen have this pattern.

Considering this adds functionality that arguably doesn't belong here, why is this pattern popular?

$\endgroup$
6
  • 2
    $\begingroup$ Why shouldn't it be there? What alternative do you propose? $\endgroup$ Jul 12, 2013 at 21:06
  • $\begingroup$ @CodesInChaos The alternative would be to accept the entire input in digest(). There's nothing about the design of hash algorithms that requires feeding the input in piece by piece. This seems to run against the en.wikipedia.org/wiki/Single_responsibility_principle $\endgroup$
    – Steve Clay
    Jul 13, 2013 at 18:19
  • 2
    $\begingroup$ Which requires putting the full message into a continuous piece of address space. So it needs unnecessary copying in some cases and doesn't fit at all for large inputs. With an update() based API I can easily hash a multi gigabyte file on a modest computer. $\endgroup$ Jul 13, 2013 at 19:02
  • $\begingroup$ @CodesInChaos Technically on 64-bit modern operating systems you can map an arbitrarily large file into the virtual address space and let the OS stream the data in and out but I agree it's much better to have this functionality built into the API itself, so it works everywhere the same (and works everywhere period). Plus it isn't that hard to implement.. $\endgroup$
    – Thomas
    Jul 14, 2013 at 20:11
  • $\begingroup$ There are some hash functions (e.g. SHA-512) that allow around 4.25*10^37 bytes of input. You could not map such input into the address space of modern OS. (Then again, hashing large input, such as 2^64 bytes takes long time even with HW acceleration and thus it is performed rarely.) $\endgroup$
    – user4982
    Jul 19, 2013 at 7:16

3 Answers 3

13
$\begingroup$

Being able to generate hashes incrementally (by presenting parts of the string being hashed) is there for basically two reasons:

  • It's commonly useful in practice. Quite often, we don't have the entire string in one contiguous segment; instead, we often have parts laying around separately. For one simple example of this, consider the HMAC function; the inner hash is some preprocessed keying material, followed by the message being MAC'ed. We could create a temporary buffer consisting of the HMAC inner key ("ipad") and the message, but why?

  • It's actually easy to do with our current hash function (including SHA-1, all the SHA-2 versions and SHA-3). You appear to think that the Python hashlib library copies the parts of the message being hashed into a buffer, and then runs the hash on the entire buffer. That's extremely unlikely to be how they implement it; instead, it is easy (with a small amount of side memory) to have the hash process the message in pieces.

So, the answer is: they provide this functionality because people find it useful, and its easy for the library to do.

$\endgroup$
3
  • 4
    $\begingroup$ It also makes the hash implementation run in constant memory, which I think is actually the real reason it's done this way - you can convert a constant-memory scheme into a more convenient but less flexible "all at once" scheme, via a wrapper implementation, but doing the opposite is harder. So it's a "lowest common denominator" kind of deal. $\endgroup$
    – Thomas
    Jul 13, 2013 at 7:52
  • $\begingroup$ Yep. hashlib uses update() in their HMAC implementation: hg.python.org/cpython/file/default/Lib/hmac.py $\endgroup$
    – Steve Clay
    Jul 13, 2013 at 18:29
  • $\begingroup$ I think the most likely explanation for the API is to support intermediate hashes ("I need separate hashes for i1, i1 + i2, and i1 + i2 + i3"): docs.oracle.com/javase/1.4.2/docs/guide/security/… $\endgroup$
    – Steve Clay
    Jul 13, 2013 at 18:42
3
$\begingroup$

To encourage the culture of avoiding dissemination of confidential data across multiple copies of "the same object" in runtime memory:

instead of:

string password = copyPasswordFromAnotherString(inputArg);  
string salt = generateRandomSalt();  
string tohash = password+salt;  
return(hash.digest());  

prefer:

hash.update(inputArg); 
string salt = generateRandomSalt();  
hash.update(salt);  
return(hash.digest());  

Also (as probably meant by poncho), to help use cases involving hashes built from data obtained incrementally (i.e.: hashing a large confidential document without having copies of it in runtime memory)

$\endgroup$
-2
$\begingroup$

I would argue that hash APIs wrap string buffer creation because leaving buffer creation up to the developer is a flaw waiting to happen. Consider hashing a set of words without inserting the word lengths in between - this construction could be exploited by an attacker to find collisions.

$\endgroup$
5
  • $\begingroup$ I don't get your point. What are you comparing update to? $\endgroup$ Jul 12, 2013 at 21:22
  • $\begingroup$ Not having update() at all, and only offering a function hash() that takes an already-created buffer and computes the hash. $\endgroup$
    – pg1989
    Jul 12, 2013 at 21:23
  • $\begingroup$ It depends on how the update() function is implemented. Maybe the library automatically prepends the length. My point is that an explicit update() function makes that possible. $\endgroup$
    – pg1989
    Jul 12, 2013 at 21:37
  • 3
    $\begingroup$ I still don't get your point. An update function that prepended the length would be broken since now calling update twice isn't equivalent to calling it once on a combined buffer. $\endgroup$ Jul 12, 2013 at 21:43
  • 1
    $\begingroup$ Hmm, you're right. $\endgroup$
    – pg1989
    Jul 12, 2013 at 21:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.