When do we use a hash algorithm and when do we use an encryption algorithm (a cipher)?
Hash functions are typically used to map an arbitrary-length (usually long) input into a fixed-length (usually short) and random-looking value. E.g., SHA-256 always outputs a 256-bit random-looking string (or digest) for any given input.
A very common property of hash functions is collision-resistance, i.e., it is very hard to find two inputs that generate the same output. This is very useful when the input size is very long, which implies that we can compress a big file/string into a small one while still maintaining its "uniqueness". (Of course, theoretically, there are many many long inputs that generate the same short output due to Pigeonhole principle, but they are very hard to find given reasonable computing resources.) Based on this property, hash functions have many good applications. For instance, one can use the hash values of big files to efficiently find duplicates, and some websites often publish the hash values of their downloadable application installation packages to guarantee integrity. However, hash functions themselves are not directly related to data privacy or confidentiality.
The other property that hash functions (are expected to) have is that they output random-looking values. This can be used to generate random keys given non-random (but high-entropy) inputs, i.e., the so-called key derivation functions.
Encryption algorithms (more precisely, encryption schemes) are designed to "hide" the input/plaintext with the output/ciphertext such that no one except those having secret keys can decrypt the ciphertext to get the original plaintext. They at least provide data confidentiality, and authenticated encryption schemes also provide integrity (hash functions can be used here).
How difficult it is to reverse hash- and encryption algorithms?
Hash functions by design are hard to reverse, so are encryption algorithms without knowing the secret key. But if you know the key or if you want to play with the encryption algorithm with any key, then you can easily reverse it with a decryption algorithm using the same key. So, here the simple difference is that hash functions do not have keys. (Note that keyed hash functions are often discussed in the topic of message authentication codes.)
From the theoretical/mathematical perspective, what differentiates a hash- and encryption algorithm?
Hash functions normally map long inputs into short ones while encryption algorithms often do the opposite to introduce randomness (to hide the input messages). Hash functions are deterministic while encryption algorithms are often probabilistic (otherwise the encryption scheme cannot be IND-CPA secure). Well, regarding similarities, they both generate random-looking outputs.