Can sleeper agents in a public network find each other and avoid infiltration?

Question

Suppose a bunch of software entities are participating in a wide-area broadcast network (the network carries a lot of traffic, and the software entities also participate in those conversations, so their transmissions are likely to be seen by friend and foe alike). How can they find each other efficiently and establish communications, without being infiltrated by enemy agents who (we will assume) can capture and inspect any particular entity?

For example, one entity could find another candidate entity by noticing a particular type of message error or checksum, and could send a decoy message that would contain a similar type of error. But an enemy agent could do the same thing, or an enemy monitor could watch for entities engaging in this protocol.

What are the search keywords for learning about protocols that are useful in this area?

Edit to add:

My specific use case is within the BitTorrent network / ecosystem. The main function of BT is to distribute data files among interested participants. The data files are identified by a 160-byte hash (the infohash). An auxiliary network, the Distribute Hash Table (DHT), helps clients find a node that is distributing the data file having a particular infohash.

DHT nodes are identified by a random 160-byte number, and their basic operation is to ask other nodes if they know a BT node that is distributing a particular infohash. DHT nodes self-organize by their ID numbers, and searching is done by matching the hash from MSB to LSB.

If a DHT node is asked to find infohash abcdefghijklmnopqrs, then it will ask nodes it has already encountered, whose ID numbers start with abcd.... (The longer the matching prefix, the more likely that node will know the answer we're looking for.) If those nodes don't know, they recursively ask the nodes they know about with longer matching prefixes. Eventually, a node knows the exact answer, or does not know any other node closer to the exact answer, and that node's address is returned to the original querier.

So DHT nodes are continually asking each other where to find 160-bit hashes of one thing or another.

I would like my DHT nodes to be able to recognize each other by asking for particularly-formatted infohashes. For example, infohashes where every prime bit is 0. (I understand that some ordinary inquries may match this pattern as well, but that's OK -- one of my nodes might respond, but the ordinary querier wouldn't proceed with the protocol so my node will know that it's not one of my special nodes.)

The problem is, an adversary can reverse-engineer my code and determine what my special infohashes are, and can implement a node that pretends to be one of my special nodes, but which does not play nice -- it might feed me bogus data, instead of the data my nodes are actually trying to collect.

So, I would like to devise a way that my nodes can identify each other, and can detect and exclude nodes that pretend to be "on my team," but are in fact enemy nodes. Obviously, I could hard-code the IP addresses of my nodes (or something similar), but I would like to be able to start an arbitrary number of nodes running at arbitrary places around the Internet, and have them find each other and start cooperating.

Answer 1

I'm looking for a way for one message recipient to distinguish steganographic (or other) messages from a friend client from similar messages from a foe client.

That's possible only if there's something friends know that foes don't. And if we talk crypto implemented by local-run software, that thing must be a key. If that key is not in the software itself, and that software is public, then it must be an input of the software available selectively to the "friend" instances. That does not seem to be the case here.

If we nevertheless assume such a key, we can think of using an authenticated encryption which cryptogram is indistinguishable from random for one without the key, which gives a method to hide the traffic in a context where there is legitimate reason to send arbitrary-looking data blobs like hashes. However:

• If the software is public, the fact that it has provision allowing this can't be hidden. Only the use of that provision can perhaps be hidden.
• That method rules out that one without the key can make a static analysis of a packet's content to tell if it contains a hidden payload (or which). But it can't rule out that such hidden payload gets indirectly detected by other means (like analysis of if there's any other reason a node could send a packet like this at that time; or the effect of that packet on the friend client software receiving it).

Answer 2

This is called steganography, or the techniques used to hide secret messages within other messages.

If we're communicating over the internet though, it's probably a better idea to just use encryption. For example, the Tor network is quite large, and everyone's traffic on Tor theoretically should be indistinguishable, whether you are using it to access Facebook or you are an undercover intelligence agent. If enough people use Tor, your Tor traffic won't stand out (which is why it was released to the public, so the US government could have their users blend in with the traffic of users from the general public).