I have a YubiKey 4 and want to use it to store a private key that should never enter a computer system's memory. After importing the key to the device I understand it never leaves it. Do the signing/decryption operations that take place for plain text using the private key happen on the device's hardware? Also, if this does happen on hardware, then can it be used like an HSM if proper compliance does not require that it generate the key on the device?
From the official specifications, published by NIST;
NIST Special Publication 800-73-4 Interfaces for Personal Identity Verification – Part 1: PIV Card Application Namespace, Data Model and Representation
A.5 Key Establishment Schemes with the PIV Key Management Key
FIPS 201 specifies a public key pair and associated X.509 Certificate for Key Management. The key management key (KMK) is further defined in SP 800-78, which defines two distinct key establishment schemes for the KMK:
A.5.1.1 RSA Key Transport with the PIV KMK
As specified in SP 800-78, the on-card private KMK can be an RSA transport key that complies with [PKCS1]. In the scenario described above, a sender encrypts a symmetric key with the KMK’s public RSA transport key. The role of the on-card KMK private RSA transport key is to decrypt the sender’s symmetric key on behalf of the cardholder and provide it to the client application cryptographic module
The client application cryptographic module would typically mean the software on the computer that you connected the card to. It means that RSA operations happens on the card, but the computer will then decrypt the full message. It's designed this way to not make the card a bottleneck for large messages.
The Yubikey mostly follows the official spec for these protocols.