Suppose a bunch of software entities are participating in a wide-area broadcast network (the network carries a lot of traffic, and the software entities also participate in those conversations, so their transmissions are likely to be seen by friend and foe alike). How can they find each other efficiently and establish communications, without being infiltrated by enemy agents who (we will assume) can capture and inspect any particular entity?
For example, one entity could find another candidate entity by noticing a particular type of message error or checksum, and could send a decoy message that would contain a similar type of error. But an enemy agent could do the same thing, or an enemy monitor could watch for entities engaging in this protocol.
What are the search keywords for learning about protocols that are useful in this area?
Edit to add:
My specific use case is within the BitTorrent network / ecosystem. The main function of BT is to distribute data files among interested participants. The data files are identified by a 160-byte hash (the infohash). An auxiliary network, the Distribute Hash Table (DHT), helps clients find a node that is distributing the data file having a particular infohash.
DHT nodes are identified by a random 160-byte number, and their basic operation is to ask other nodes if they know a BT node that is distributing a particular infohash. DHT nodes self-organize by their ID numbers, and searching is done by matching the hash from MSB to LSB.
If a DHT node is asked to find infohash abcdefghijklmnopqrs, then it will ask nodes it has already encountered, whose ID numbers start with abcd.... (The longer the matching prefix, the more likely that node will know the answer we're looking for.) If those nodes don't know, they recursively ask the nodes they know about with longer matching prefixes. Eventually, a node knows the exact answer, or does not know any other node closer to the exact answer, and that node's address is returned to the original querier.
So DHT nodes are continually asking each other where to find 160-bit hashes of one thing or another.
I would like my DHT nodes to be able to recognize each other by asking for particularly-formatted infohashes. For example, infohashes where every prime bit is 0. (I understand that some ordinary inquries may match this pattern as well, but that's OK -- one of my nodes might respond, but the ordinary querier wouldn't proceed with the protocol so my node will know that it's not one of my special nodes.)
The problem is, an adversary can reverse-engineer my code and determine what my special infohashes are, and can implement a node that pretends to be one of my special nodes, but which does not play nice -- it might feed me bogus data, instead of the data my nodes are actually trying to collect.
So, I would like to devise a way that my nodes can identify each other, and can detect and exclude nodes that pretend to be "on my team," but are in fact enemy nodes. Obviously, I could hard-code the IP addresses of my nodes (or something similar), but I would like to be able to start an arbitrary number of nodes running at arbitrary places around the Internet, and have them find each other and start cooperating.