Yes, this is usually possible. You can use the RSA public key to:
- perform modular exponentiation using the public exponent and modulus (raw RSA, sometimes you can use raw RSA decrypt with the public key for this);
- interpret the result as unsigned big endian integer (usually this is already the case);
- perform the unpadding (usually this just means finding the hash within the padding).
Note that there are multiple RSA schemes possible, e.g. PKCS#1 v1.5 padding or PSS padding. It depends on the signature generation which one is used. For an insecure 512 bit key you would generally bet on PKCS#1 v1.5 padding.
You can find both schemes in the relatively readable PKCS#1 RFC which includes the full description of above steps. However, it ends with creating the padding from an existing hash instead and compare, you need to reverse the padding creation instead - this isn't hard for PKCS#1, but you would not be able to do it for PSS as the hashed (
mHash) value is hashed again (to get a hash value
For PKCS#1 v1.5 padding the hash value is just the
hLen bytes that are right before the last byte, where
hLen is the hash size in octets / bytes. Of course you'd still have to validate the rest of the padding as well to perform full signature verification.
If you are very lucky then decryption with the public key while choosing the PKCS#1 decryption scheme will directly give you the hash. But note that decryption generally requires the private key and a different unpadding scheme (also generally referred to as PKCS#1 padding, but the padding for encryption is still different than that for signature generation). However, it may just choose the signature unpadding instead for compatibility reasons.