I could not find any mention on the Internet of a proven/known cryptographically secure keyed rolling hash function (ie rolling MAC). Has the question been studied, is it possible to build one ?

By cryptographically secure I mean properties equivalent to HMAC with a cryptographic hash function :

  • without knowing the key, knowing the hash value does not leak information about the data,
  • knowing or chosing some data does not allow to recover the key more easily than brute force.

By rolling hash, I mean that it can be efficiently iteratively computed on a window sliding byte by byte, ie that there exist an update operation $f$ in the form $H(c_{n-k-1}...c_{n}) = f\left[c_n, c_{n-k-2}, H(c_{n-k-2}...c_{n-1})\right]$, where the $c_i$ are the data characters/bytes, and $k$ is the size of the sliding window.

To be more specific about the application, we want to build a secure chunking algorithm for data deduplication, which produces variable size chunks constructed by splitting data at content-dependent points, and should avoid that the cut points leak information about the data. Existing programs are using solutions based on common rolling hash functions with custom obfuscation, which don't seem to have been well analyzed, for instance :

  • rsyncrypto is using the same function than gzip --rsyncable, which is simply the sum of the bytes (obviously weak).
  • Attic is using cyclic polynomial (Buzhash) with random secret substitution of input data bytes (a substitution from bytes to 32 bits integers, currently obtained from a public table xored with a secret 32 bits seed, but which is planned by the author to be obtained instead with AES-CTR encryption of a zeroed table ; this substitution corresponds to the function $h$ in the wikipedia page).
  • Tarsnap is using Rabbin-Karp hash with random secret substitution of input data bytes and random secret parameters (a substitution from bytes to 32 bits integers using a table of which values are HMAC of indexes, and parameters $a$ and $n$ of the wikipedia page are also depending on HMAC of some fixed data ; in addition the window size of the rolling hash is also only known within a range and is variable).

Are there reasons to believe that some would be more secure than others, in particular Rabbin-Karp vs Buzhash ? Are they known to be preimage resistant ?

Note: The chunking application has the properties that little ciphered data is disclosed (only one hash everytime the cut decision is taken), and that only part of the ciphered data is disclosed (the cut decision is usually taken if the last bits of the hash are null or equal some value, so only these last bits are disclosed).

Performance requirements

For the deduplication application, we would like that in the worst case the processing speed of the hash on an average machine be at least equal to disk read throughput (say around 60MB/s), so that it does not become the bottleneck. Ideally it should be as fast as possible while preserving sufficient security guarantees.

[AES] Implementing D.W.'s answer with buzhash and AES-128 shows that on a high-end modern CPU (Intel(R) Core(TM) i7-4800MQ CPU @ 2.70GHz) the secure rolling hash runs at 9MB/s (corresponding to raw AES at 140MB/s). If enabling AES-NI using the EVP API of openssl, performance peaks to 56MB/s (corresponding to raw AES at 900MB/s), or even to 150MB/s if computed by batch in ECB mode (computations seem to be parallelized). However we cannot assume yet that an average machine has such instructions : for instance with an older CPU (Intel(R) Core(TM)2 Duo CPU P8400 @ 2.26GHz), performance is limited to 4.4MB/s, and with a low end modern CPU (Intel(R) Atom(TM) CPU N2800 @ 1.86GHz) to 1.5MB/s, which is not acceptable.

[SipHash] According to D.W.'s answer, the requirement for $E$ is that it is a PRF or a PRP with large domain, so SipHash for instance which is claimed to be a PRF should be ok if I'm not mistaken. The secure rolling hash built with it runs at 57MB/s, without using any particular instruction set, or even 97MB/s if specialized for 4 bytes inputs. However this is still way below the 420MB/s obtained with buzhash alone, and on the older CPU mentioned before performance drops to 12MB/s.

[CBC-MAC-AES] Now assume the length of the unsecure rolling hash $R$ is $n$ bits (eg $n=32$), and we only need a secure rolling hash of length $m<n$ (eg $m=16$), because we only want to test if the $m$ last bits are null. AES encryption of $m$ bits blocks (padded to 128 bits) is a PRF, which can be precomputed in a table if $m$ is small enough, and whose domain can be extended to $n$ bits using CBC-MAC to generate a PRF (see introduction of this paper).

Unfortunately after more investigation, the dictionary of the $m$ bits blocks cipher can be cracked in $O(2^m)$ under a CPA with CBC-MAC (finding 3 particular collisions). In our particular context where only collisions of value 0 are revealed, this is not possible, and the best distinguisher requires $O(2^m)$ operations, equivalent to the birthday bound for an $n$ bits blocks cipher, so this scheme would be as secure as an $n$ bits blocks cipher. However $n=32$ is also too small and is not secure, and if increasing $n$ then the best distinguisher still requires only $O(2^m)$ operations (basically because if testing all the values of one $m$ bits word, this scheme will generate exactly one collision with 0, whereas a random function could generate 0,1,2...), so this scheme with CBC-MAC is not IND-CPA with $m=16$ (it would with $m\geq64$ or $128$).

  • $\begingroup$ Also, if you want us to comment on the security of Tarsnap or Attic, please define more precisely what you mean by "random secret substitution". $\endgroup$
    – D.W.
    Commented May 9, 2014 at 5:01
  • $\begingroup$ I edited the question to add the requested information, thanks. $\endgroup$
    – cyril42e
    Commented May 9, 2014 at 6:32
  • $\begingroup$ Thanks for the update. I'm still not clear on how Attic and Tarsnap work. Is the substitution applied to the input data, or to the output of the Rabin-Karp/cyclic hash? I don't know what you mean by "HMAC of indexes". Would you like to try expressing it in mathematics? $\endgroup$
    – D.W.
    Commented May 9, 2014 at 15:09
  • $\begingroup$ Sorry, I'm still not clear on how Tarsnap works. When is the substitution done? Is it applied to the input data, or to the output? If it substitutes from bytes to 32-bit values, does it expand the size of the input by 4x before hashing (or expand the size of the output by 4x after hashing)? $\endgroup$
    – D.W.
    Commented May 10, 2014 at 1:08
  • $\begingroup$ You got 50 or 56 MB/s, and your goal was about 60 MB/s: sounds like you're basically done. Intel processors are widely deployed on servers, and modern Intel processors do support AES-NI. If in your application domain the processors typically don't support AES-NI, then you might want to measure what processors are used by your users (hint: make sure to measure), then do some research on other efficient block ciphers/PRFs (there are other candidates). SipHash is probably fine, if it meets your performance requirements, but I haven't studied it in detail. 128-bit keys are plenty. $\endgroup$
    – D.W.
    Commented May 12, 2014 at 22:00

1 Answer 1


One simple cryptographically secure rolling hash function is the following:

$$F_{k1,k2}(x) = E_{k1}(R_{k2}(x))$$

where $R_{k2}(\cdot)$ is a non-cryptographic rolling hash function (e.g., Rabin-Karp), and $E_{k1}$ represents encryption with a block cipher (e.g., AES). By $R_{k2}(\cdot)$, I mean that the parameters of the rolling hash should be derived from $k2$. Also, I require that you choose $R_{k2}(\cdot)$ so that it is a universal hash function ($\epsilon$-almost universal is also adequate). With those choices, this will be cryptographically secure, and it will be a rolling hash.

For instance, Rabin-Karp is $\epsilon$-almost universal if you choose all of the parameters to be secret and derived from $k2$, so it is a fine choice for this. The same is true for cyclic hashing (Buzhash), so it too can be used in this way.

Note that when I write $E_{k1}(x)$, I mean the AES encryption of the 128-bit block $x$. There is no mode of operation; this is just the raw block cipher. (You might be tempted to think that this is using ECB mode so is problematic, but that's not correct. This is not encryption and the issues with ECB mode encryption don't apply here. In fact, it is possible to prove that my construction is secure; see below for a sketch of the proof.)

Why is this a rolling hash, you ask? Well, given $y=F_k(x)$, if you know the key $k$, you can compute $z=E_k^{-1}(y)$, which is an output of the rolling hash $R(\cdot)$; then update $z$ using whatever update algorithm is used with $R(\cdot)$ when updating input $x$ to $x'$, to get the updated rolling hash $z'$; then encrypt $z'$ to get $y'=E_k(z')$. Now $y'$ is the secure rolling hash of $x'$, i.e., $y' = F_k(x')$. Therefore, this provides an update procedure for $F_k(\cdot)$, based upon the update function of the underlying rolling hash.

Why is this cryptographically secure? That follows from two standard theorems in cryptography: (1) if $R$ is universal and $E$ is a PRF, then $E \circ R$ is a PRF; (2) if $E$ is a PRP with large domain, then $E$ is a PRF. Of course, the definition of security for a block cipher is that it should be a PRP. Consequently, if $E$ is a secure block cipher, then it is a secure PRF, and so too will $F$ be. A PRF (pseudorandom function) provides exactly the security property you want; it is the right way to formalize what you mean by "cryptographic security" for a keyed hash.

Performance: Because block ciphers are very fast, this will be almost as fast as the underlying rolling hash -- i.e., very fast.

As far as other schemes that you asked about:

Rabin-Karp with secret parameters is not secure. Given some known plaintext, you can easily recover the secret parameters using linear algebra (just solve some simultaneous linear equations). I'm not clear on what you mean by "with random secret substitution", so I couldn't comment on Attic or Tarsnap.

If you mean Buzhash as defined in Wikipedia, where each byte of the input is mapped to a 32-bit value using a table lookup (secret random substitution) and then a cyclic hash is applied, then that is not cryptographically secure. Everything is linear, and it can be broken by linear algebra. Introduce 256 unknowns $T_0,T_1,T_2,\dots,T_{255}$ to represent the 256 table elements; in other words, the byte $b$ in the input gets mapped to $T_b$. Then if you know the input to the hash function and the corresponding output, you get a linear equation on the $T$'s: for instance, if the input is known to be 0x07 0x13, then the output is $s(T_7) \oplus T_{19}$, which is a linear function of the $T$'s and gives you one linear equation on them. Therefore, if you have more than 256 known input-output pairs, you can use linear algebra to recover all of the $T$ values, learn the secret substitution, and then completely break/predict the hash function in the future. So, if you need cryptographic security, don't use Buzhash alone.

  • $\begingroup$ That looks convincing! However I'm concerned about performance now. Blocks ciphers are very fast, that is true, but this scheme encrypts a 128 bits block for processing each byte of input, so it would be actually 16 times slower than the underlying raw block cipher. Are there secure block ciphers that work with 32 bits blocks, and available in standard crypto libraries, to at least avoid mostly encrypting padding bytes? $\endgroup$
    – cyril42e
    Commented May 10, 2014 at 0:28
  • $\begingroup$ @cyril42e, have you benchmarked it? The AES-NI instructions are surprisingly fast. So you might want to implement, benchmark, and see if it meets your performance requirements. If it doesn't, I suggest editing your question to describe your performance requirements and how close this scheme gets and what you tried, to improve performance. Selecting the fastest block cipher for your platform is beyond the scope of this question, but you can find lots of other questions here that talk about that, or you can ask a new question. $\endgroup$
    – D.W.
    Commented May 10, 2014 at 1:06
  • $\begingroup$ @D.W. $\;\;\;$ "If $E$ is a PRP" with large domain "then $E$ is a PRF". $\:$ Also, does (almost-)2-universality help more than (almost-)universality for this use? $\;\;\;\;\;\;$ $\endgroup$
    – user991
    Commented May 10, 2014 at 17:50
  • 1
    $\begingroup$ You should understand "secure" encryption as being with a PRP, and you should understand "large domain" as meaning that for all integers $q$, if $\:1<q\:$ and the adversary is limited to learning at most $q$ hash values then the block cipher's contribution to the adversary's advantage is less than $\hspace{1.177 in} 1\hspace{-0.05 in}-\hspace{.02 in}$[$\exp$](en.wikipedia.org/wiki/Exponential_function)$(-(q\cdot \hspace{-0.03 in}(q\hspace{-0.04 in}-\hspace{-0.05 in}1))/(2\text{^}($[$n$](en.wikipedia.org/wiki/Block_cipher#Definition)$+\hspace{-0.05 in}1)))\;$. $\;\;\;$ $\endgroup$
    – user991
    Commented May 12, 2014 at 20:45
  • 2
    $\begingroup$ @D.W., just to be clear, you are saying that for perfect security in the scheme you proposed in your answer, the non secure rolling hash $R_k$ should be at least 128 bits ? $\endgroup$
    – cyril42e
    Commented May 16, 2014 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.