- is it a reliable thing to map a SHA512 of Rainbow-1 or GEMMS128 public key to such key, to prevent transfer of such huge keys with every message or file transferred?
Well, the verifier would need to get the entire public key in order to verify - the hash of that public key would not be sufficient.
On the other hand, if the verifier can use that hash to look-up the public key somehow (either to say "I have 3 preinstalled public keys that I trust; this hash will allow me to pick out which one it is"; or "I can use this hash to download the public key from a public server"), that can work.
In fact, one common suggestion to address these huge public keys is, in fact, to send a "URL+Hash" in place of the public key - the URL would identify where to download the public key from, and the hash would allow you to verify that what you downloaded was correct (and that it wasn't modified on the public server) [1].
- Or if transferred thing will be the random 16 byte stuff with signature done with these algorithms, is it ok to used such construct (
key_random_id||signature_of_id_by_key
as key for public keys lookup table-service?
By 'signature_of_id_by_key', do you mean that the signer signs the key_random_id, and sends that?
Well, I'm not so certain about that. I believe that (depending on the signature algorithm involved) it may be possible to modify the public key so that, for a specific message (key_random_id), the signature is unmodified, and so someone who can modify things on the table-service could replace the public key with something else.
What could they do with that (other than general mischief)? I'm not sure, however by using a collision-resistant hash of the public key, we know that would not be an issue.
[1]: We're sending this URL+Hash where we would normally send the public key. This transmission of the public key must be protected somehow by the protocol (otherwise we can't trust the public key), and this same protection would extend to the 'URL+Hash' (and hence we don't have to worry about someone replacing the hash with the hash of their own public key).