I was recently solving a CTF problem and it involved a shadow and passwd file. I got to know the username easily but I am unable to find the password since it is SHA512 hashed. My doubt is that when I read the hashed value it is not exactly a random hexadecimal value. It contains many characters other that 1, 2, 3 ... F. How can I decrypt it because I can't even use tools online since they return a hexadecimal value.
Instead, when you are trying to “break“ a hash output like the hex string you are looking at, you will need to find the input that produces that hex string. You will have to use a brute force attack using the hints provided within the CTF itself.
As for any potential encoding/decoding issues you might be facing (since you mention something about characters outside of the HEX range), that’s more of a programming thing and not really related to cryptography.