Does public-key + hashing create a security vulnerability?

Question

The title says it all really...let's say I have a packet of data that one of my applications needs to store encrypted with a public key (RSA, say, with the private key kept elsewhere). Let's say the application also then needs a way of determining if said packet has already been stored, so I combine it with a randomly generated salt and save the hash of packet+salt so I can check it later (e.g. MD5). Does storing the combination of these three pieces of data...i.e. encrypted packet, the salt and the hash....result in data that is significantly less secure than the encrypted packet alone?

I realize "significantly less secure" is an arbitrary metric. I'm not after anything military grade (for want of a better nonsensical term), I just want to make sure I'm not doing anything overtly silly.

Answer 1

The hashing does not compromise security in any way. Assuming you're calculating hash(ciphertext+salt), the hash provides no new information that an attacker with access to your ciphertexts doesn't already have.

A secure hash like SHA-3 is a one-way function, safe even on a salted plaintext, and the outputs of such are stored in authenticated encryption. You can actually strengthen your system by using a HMAC rather than just a hash and incorporating an authentication mechanism.

As for whether the whole system is secure, it depends on what you're storing.

Checking if the packet has been stored is likely to create an oracle for a side channel attack. If the attacker can send packets to encrypt and your response or its timing is in any way different for duplicates, they can verify if a certain file of interest or a certain password is being stored in your system.

Whether it's effective depends on the search space for the plaintext. If it's small, e.g. passwords, or files people don't want to be known to possess, that's a vulnerability. Deduplication is generally at odds with security, and creating secure deduplication is a current challenge.

N.B. I've addressed only security, not utility above. Keep in mind that you can only compare hashes with the same salt, so this scheme may have limited effectiveness for deduplication.