I have often seen these terms being used interchangably, but is that correct? Are there Private Key Algorithms that do not use Symmetric Encryption?
Private key cryptography is often used as synonym for symmetric cryptography, that is, cryptography with constructs that use a symmetric key. For instance professor Katz uses this terminology.
Symmetric keys can be used for e.g. a block cipher such as AES, a stream cipher such as RC4. HMAC - which is a keyed hash construct - also uses a symmetric key, and doesn't use a cipher at all. The term "symmetric encryption" is usually limited to constructs that provide confidentiality, although authenticated ciphers may also provide integrity and authenticity of the encrypted messages.
So the question " Are there Private Key Algorithms that do not use Symmetric Encryption?" has the response "yes" for the simple reason that HMAC it does not provide encryption, and it is certainly a private key algorithm. Symmetric encryption which is a goal rather than a construct uses certain private key algorithms to accomplish the goal.
Private key cryptography refers to a broad class of algorithms which are all "theoretically equivalent", meaning: if you have one, then you can construct each of the others (but those equivalences are essentially of theoretical interest, they do not produce efficient algorithms in general). This includes pseudorandom generators, symmetric encryption, signatures, and (preimage resistant) hash functions (the list is obviously not exhaustive). Therefore, from a theoretical point of view, a symmetric encryption is "equivalent" to any primitive in symmetric cryptography (e.g. you can construct a prg from a symmetric encryption scheme, and a symmetric cryptosystem from a prg), but practical construction might strongly differ in their design, so from a more practical viewpoint, "private key cryptography" refers to a large class of primitives which includes in particular symmetric cryptosystems.