Is it possible to authenticate a client with encryption on a simplex communication channel without the use of authoritative root certs?
Sure. For instance:
- Use PKI, register client public key (or self signed certificate) on the server and consider any registered public keys as trusted. You should understand also all drawbacks that it has, like dealing with validity period (how are you going to check if some PKs are still valid / trusted or not and should you remove them from your server PK store or not?), dealing with compromized private keys (instead of revocation list of CA), etc.
- Use shared secret. Again, be prepared to the drawbacks. For instance, in such case not only the user, but also you on the server side will be responsible for keeping user authentication material safe.
- One time passwords.