Yes, that's right. It's called a [hybrid cryptosystem](https://en.wikipedia.org/wiki/Hybrid_cryptosystem), and it means that the work for the Yubikey is the same no matter how large the message is.

For encryption, a symmetric key is established for each individual message. 

With RSA, this process is called key encapsulation, because a randomly chosen key is encrypted with the recipient's public key.

With elliptic curves, the process is called key exchange. An ephemeral (one-time) key pair is created by the sender, and the one-time public key from this pair is published. The recipient uses EC Diffie-Hellman to discover the symmetric key.

To summarise: Signatures are created by the Yubikey. Encryption only requires the Yubikey to sign the outgoing message. Decryption only requires the Yubikey to recover the encapsulated key or to perform Diffie-Hellman to discover the key.

Note that when signing, you would not use the word 'encrypt' generally to describe the signing process. That is a technical implementation detail that might apply to RSA but not to EC signing.