Disclaimer: This may not be the most accepted academic conclusion since I'm talking (mostly) from my personal experience.
One big area where well defined security protocols are missing is distributed systems resistant to segmentation. For example, most NoSQL databases (mongo, couch) are such systems.
PKI (or even general purpose public key cryptography) works well for distributed systems but it requires connectivity to some kind of master node or a pool of master nodes. Yet, if a group of nodes become alienated from the master nodes (whilst being able to communicate between themselves), these nodes cannot perform any work.
In other words, using the current PKI methods the is a single point of failure. Or, in the best case, a point of failure where a group of nodes is more important than other nodes.
The research in the field of distributed systems (in general) has some US patents. And it was heavily studied a decade ago. The results of those studies are the systems we see today with master nodes responsible for the cryptography (that must not become segmented) and several worker nodes.
The distributed systems that aim for segmentation resistance simply ignored network security. To become segmentation resistant mongoDB and couchDB (for example) do not even have a user or node identification. In practice, the security of these systems is implemented by a firewall around a subnet containing all nodes and some kind of encrypted tunnel between node farms (if the traffic needs to go through the internet for example).
(I believe that) A protocol to implement PKI (or even a different public key infrastructure) over such systems is a researcher's dream. I need to admit that I try to get my head around this problem when insomnia hits me, and I'm probably not the only one.