I have recently written an implementation of Serpent and was testing it against known vectors to no avail. Using 256-bit key, I compared my encryption to the test vectors located here:
http://www.cs.technion.ac.il/~biham/Reports/Serpent/Serpent-256-128.verified.test-vectors
For some reason, I match none of these test vectors, and I do not know why. I downloaded the Python implementation found here:
https://www.cl.cam.ac.uk/~fms27/serpent/serpent.py.html
The Python implementation agreed with me and disagreed with the published test vectors. I did, however, find two files in folder Floppy 4 (ecb_tbl.txt) and (ecb_iv.txt) from the full submission package (http://www.cl.cam.ac.uk/~rja14/Papers/serpent.tar.gz) which I completely agree with. If I put in the given 256-bit key and the given plaintexts, I do achieve the correct encrypted ciphertexts.
Are the test vectors published on biham's website incorrect, or is there perhaps something I am missing? If they are wrong, why are they still up and way easier to find than the ones I matched with?
As an example, on Biham's site,
key=8000000000000000000000000000000000000000000000000000000000000000
plain=00000000000000000000000000000000
cipher=A223AA1288463C0E2BE38EBD825616C0
But both my implementation and the Python reference implementation give:
cipher=abed96e766bf28cbc0ebd21a82ef0819
I think it has something to do with this NESSIE format, but so far, I have been unable to determine if that is true and what exactly that means.