0
$\begingroup$

On a high level the question is: how does one cryptographically sign streaming data?

Here is the context:

  • a number of people are continuously streaming data to me
  • the data is public and does not necessarily need to be encrypted
  • I would like to confirm that it is indeed them sending the data
  • I would like to compensate them (perhaps a payment on a blockchain to a public address)

What is the general strategy (or is there a standard?) to achieve this? Here are some rough thoughts:

Alice signs contiguous chunks of data with her private key, I confirm it comes from her with her public key and can encode some rules to pay her (as identified by her public key).

Are there papers or further reading or I'm I on the wrong track?

$\endgroup$
  • 1
    $\begingroup$ Generally, you sign the key exchange and use the shared secret to generate a MAC key (for e.g. HMAC). Is that the kind of thing you're looking for? $\endgroup$ – forest Apr 17 at 3:33
  • $\begingroup$ If I understand you correctly that would require the sender and receiver to share a private key. Then sender can hash the message+key and receiver can hash message+key and they can agree that both have the same message and same key. My problem however is different: Without knowing the secret key of the sender (but knowing her public key), can I verify that she is the one sending it? $\endgroup$ – Massagran Apr 17 at 3:49
  • $\begingroup$ You can sign with ECDSA or RSA digital environment. Could you specify what you have currently on your systems? $\endgroup$ – kelalaka Apr 17 at 6:21
  • $\begingroup$ @kelalaka There is no system yet. I am trying to flesh out an idea and what the requirements would be. More specifically, I want crowdsourced streaming data to train a neural network (data has to be streaming, it can't just be sent once as one or several files) and I also want to correctly attribute who contributes what data and reward them - the cryptography comes in to avoid freeloaders claiming ownership of data they have not contributed. $\endgroup$ – Massagran Apr 17 at 11:31
  • $\begingroup$ You cannot sign (or MAC) a data set of infinite size. You will have to split the stream in chunks and sign (or MAC) those. Those chunks can be of fixed size (every x MB) or delimited by a recurring token (like 0x000000000). The signatures will need to be inserted in the stream or accessible in a 2nd channel. $\endgroup$ – A. Hersean Apr 17 at 15:18
2
$\begingroup$

This is very similar to establishing transport security with entity authentication. In other words, this would be much like TLS, except that in this case the client seems to require authentication rather than the server.

Still, using TLS with client authentication would probably solve your needs. Sure, it would also encrypt the streams, but that should not necessarily pose a problem. Otherwise you could replace the encryption and authentication with just HMAC and skip encryption.

Otherwise it would be a very good idea to study the TLS handshake and work out how the entity authentication works and how the shared session keys are established. Once the session keys are established they can be used to authenticate the records / messages in the stream. These are just blocks of data that together form the stream.

Beware that designing / implementing transport security is fraught with danger and should not be attempted without a very good understanding of the subject.


Note that SSH is usually more geared towards client authentication rather than TLS, which requires server authentication. So SSH might be a better transport security protocol when it comes to fulfilling the requirements. I'm however less at home with the current security details of SSH.


Another option would be to sign the data before sending (application level security rather than transport security). This would let you verify that the stream was delivered by a certain entity even after transport. If this is a valuable property depends on your usage scenario; it might be a very dangerous one as well - others would also be able to verify that it was delivered by the entity that signed it after all.

This seems to be the current road you're on; however in this scenario you might as well sign the whole stream / file rather than blocks. Otherwise you would just be designing a pretty bad transport mode security without forward secrecy, a lot of overhead and a proprietary messaging layer.

$\endgroup$
1
$\begingroup$

If you can meaningfully act on a small chunk of data at a time independently, then, well, sign the small chunk of data! Indeed, you shouldn't have a system where anyone is tempted to sign a larger chunk of data than the receiver wants to be able to buffer at once—an adversary trying to deny service will send the largest possible forgery, which you can't reject until the signature is verified.

If you can't meaningfully act on a small chunk of data at a time independently, then more details of your protocol are needed. What can you act on? How much buffer space do you have? What are you trying to do with it? In what way are the chunks of data dependent?

Of course, if you are trying to ensure correct attribution of novel information, you have a problem:

  1. Alice discovers the formula for the fountain of youth.
  2. Alice signs it and nails it to the door of the church (which as we all know was the original blockchain, invented by Martin Luther Jan Hus).
  3. Paul, the plagiarist, reads Alice's remarkable formula before anyone else gets there, removes the nails, rewrites it in his own hand and signs it himself, and nails it to the blockchain.
  4. You, seeking to reward the genius inventor because you subscribe to and aim to amplify the western imperialist patriarchal great man theory dogma, shower Paul in gold, while Alice looks on with the dismay that she has become accustomed to in this world.

To avoid this gross misattribution, which is naturally the only possible failure of the great man theory, you will need more than just signatures! For example, you might want commitments and some kind of trusted timestamping service in a much more elaborate protocol…which still won't be able to prevent Stephen Wolfram from taking credit for his underlings' work no matter how much cryptography you throw at it.

$\endgroup$
  • $\begingroup$ Or possibly time stamping? $\endgroup$ – Maarten Bodewes Apr 17 at 23:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.