Having a little trouble getting Nessie vectors working with the Bouncy Castle SerpentEngine. Do the Nessie vectors require a mode or some pre-processing? Nessie Serpent 128, set 1, vector 0 reads:





Bouncy Castle SerpentTest.js, vector 10 reads:





Why the different expected output?

  • $\begingroup$ Serpent-0 and Serpent-1? $\endgroup$ – Maarten Bodewes Nov 18 '14 at 1:15
  • $\begingroup$ I thought it might be versioning, maybe changes made during AES.. but these are the vectors posted on the serpent homepage. I'll investigate tomorrow, compare to final submission. $\endgroup$ – JGU Nov 18 '14 at 1:33
  • $\begingroup$ @owlstead Do you know where the Bouncy Castle vectors came from? Looks like the discrepancy may be due to a difference in s-box implementations. $\endgroup$ – JGU Nov 18 '14 at 1:55
  • 1
    $\begingroup$ it could be that bouncy castle is using the AES submission bit ordering. the only reason the s-boxes would be different is if it is serpent-0, which uses DES s-boxes $\endgroup$ – Richie Frame Nov 18 '14 at 5:08
  • 1
    $\begingroup$ @John I couldn't find any other test vectors either, but I guess it is solved now... $\endgroup$ – Maarten Bodewes Nov 18 '14 at 8:41

Bouncy Castle seems to be using a reversed byte order for inputs and outputs when compared to NESSIE test vectors.

In order to replicate the NESSIE vector in Bouncy Castle, the order of all inputs and outputs needs to be flipped at the byte level, therefore the following results are given from a NESSIE compliant implementation (Set 1, Vector 120):

key =    00000000000000000000000000000080
plain =  00000000000000000000000000000000
cipher = DDD26B98A5FFD82C05345A9DADBFAF49

Note the first key byte is now in the final position, and the ciphertext when reversed matches BC vector 10. Additionally if we perform a Monte Carlo test, the results should be similar:

Serpent Encryption, ECB Mode, 128-bit key

COUNT =      0
KEY =        d692af3ea36f5ef30de8c5eb79a8fb31
PLAINTEXT =  14f63ff537a39296008945f7125ea070
CIPHERTEXT = 2d74412f3d89016083c93b4d0fa553fc

These results match vector 14 in reverse byte order, using a 100 block loop test, the BC test does not seem to perform an outer key loop like other Monte Carlo test implementations, such as NIST CAVP. The CAVP MCT performs 100 outer key loops with 1000 inner block loops for testing AES implementations. If continued, next iteration in the key loop would be as follows:

COUNT =      1
KEY =        fbe6ee119ee65f938e21fea6760da8cd
PLAINTEXT =  2d74412f3d89016083c93b4d0fa553fc
CIPHERTEXT = 4597017beb21a4c6c75acb85ac4645f7
  • 1
    $\begingroup$ Send a mail to one of the BC maintainers... $\endgroup$ – Maarten Bodewes Nov 18 '14 at 8:40
  • 1
    $\begingroup$ as per user13484's answer, it want matter. however, if you look at their code, their counters go in reverse, and they use word to byte conversions which reverse the byte order. NESSIE implementations count foreward, and can treat byte arrays as word arrays, leading to better performance and less potential leakage. $\endgroup$ – Richie Frame Nov 18 '14 at 10:42
  • $\begingroup$ I added the Array.Reverse step to key, plain and ciphertext, and now results are aligning.. thanks for the insight $\endgroup$ – JGU Nov 18 '14 at 17:55
  • $\begingroup$ how can I reverse using CBC encoding? $\endgroup$ – drizzt Aug 14 '15 at 23:32

Important Change, 23rd November 2015

Please see:


Okay, originally I answered saying we were doing the right thing, however, when the BC project approached the Serpent authors in 2009 it appears there was a breakdown in communication. We have just been told that the NESSIE vectors are in fact the correct ones. The previous incarnation, based on a direct interpretation of floppy 4 in the Serpent AES submission, actually had reversed vectors and keys in it.

So from BC Java 1.54, and already in BC C# 1.8, Serpent now conforms to the NESSIE vectors and in line with common practice (yes, this wasn't a rare mistake...) the previous version of Serpent which incorporates the reversal is now called Tnepres.

Note also: as we're trying to spread the word on this, there are also some versions of Serpent where, while the implementers were evidently aware of the reversal issue in the inputs/outputs, they clearly didn't realise that the keys were also reversed. If you run into one of these, please alert the implementer if possible. Quite a few of the NESSIE vectors will pass if this mistake is made, it requires testing with a broad range of key values to show it up.

  • 2
    $\begingroup$ I prefer the NESSIE order as it allows higher performance software implementations (maybe), and I assume that is why it was used. The standard Serpent order was optimized for hardware implementation (supposedly) $\endgroup$ – Richie Frame Nov 18 '14 at 10:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.