cryptographically signed streaming data

Question

On a high level the question is: how does one cryptographically sign streaming data?

Here is the context:

• a number of people are continuously streaming data to me
• the data is public and does not necessarily need to be encrypted
• I would like to confirm that it is indeed them sending the data
• I would like to compensate them (perhaps a payment on a blockchain to a public address)

What is the general strategy (or is there a standard?) to achieve this? Here are some rough thoughts:

Alice signs contiguous chunks of data with her private key, I confirm it comes from her with her public key and can encode some rules to pay her (as identified by her public key).

Are there papers or further reading or I'm I on the wrong track?

Answer 1

This is very similar to establishing transport security with entity authentication. In other words, this would be much like TLS, except that in this case the client seems to require authentication rather than the server.

Still, using TLS with client authentication would probably solve your needs. Sure, it would also encrypt the streams, but that should not necessarily pose a problem. Otherwise you could replace the encryption and authentication with just HMAC and skip encryption.

Otherwise it would be a very good idea to study the TLS handshake and work out how the entity authentication works and how the shared session keys are established. Once the session keys are established they can be used to authenticate the records / messages in the stream. These are just blocks of data that together form the stream.

Beware that designing / implementing transport security is fraught with danger and should not be attempted without a very good understanding of the subject.

---

Note that SSH is usually more geared towards client authentication rather than TLS, which requires server authentication. So SSH might be a better transport security protocol when it comes to fulfilling the requirements. I'm however less at home with the current security details of SSH.

---

Another option would be to sign the data before sending (application level security rather than transport security). This would let you verify that the stream was delivered by a certain entity even after transport. If this is a valuable property depends on your usage scenario; it might be a very dangerous one as well - others would also be able to verify that it was delivered by the entity that signed it after all.

This seems to be the current road you're on; however in this scenario you might as well sign the whole stream / file rather than blocks. Otherwise you would just be designing a pretty bad transport mode security without forward secrecy, a lot of overhead and a proprietary messaging layer.

Answer 2

If you can meaningfully act on a small chunk of data at a time independently, then, well, sign the small chunk of data! Indeed, you shouldn't have a system where anyone is tempted to sign a larger chunk of data than the receiver wants to be able to buffer at once—an adversary trying to deny service will send the largest possible forgery, which you can't reject until the signature is verified.

If you can't meaningfully act on a small chunk of data at a time independently, then more details of your protocol are needed. What can you act on? How much buffer space do you have? What are you trying to do with it? In what way are the chunks of data dependent?

Of course, if you are trying to ensure correct attribution of novel information, you have a problem:

1. Alice discovers the formula for the fountain of youth.
2. Alice signs it and nails it to the door of the church (which as we all know was the original blockchain, invented by Martin Luther Jan Hus).
3. Paul, the plagiarist, reads Alice's remarkable formula before anyone else gets there, removes the nails, rewrites it in his own hand and signs it himself, and nails it to the blockchain.
4. You, seeking to reward the genius inventor because you subscribe to and aim to amplify the western imperialist patriarchal great man theory dogma, shower Paul in gold, while Alice looks on with the dismay that she has become accustomed to in this world.

To avoid this gross misattribution, which is naturally the only possible failure of the great man theory, you will need more than just signatures! For example, you might want commitments and some kind of trusted timestamping service in a much more elaborate protocol…which still won't be able to prevent Stephen Wolfram from taking credit for his underlings' work no matter how much cryptography you throw at it.