Yes, that's right. It's called a hybrid cryptosystem, and it means that the work for the Yubikey is the same no matter how large the message is.
For encryption, a symmetric key is established for each individual message.
With RSA, this process is called key encapsulation, because a randomly chosen key is encrypted with the recipient's public key.
With elliptic curves, the process is called key exchange. An ephemeral (one-time) key pair is created by the sender, and the one-time public key from this pair is published. The recipient uses EC Diffie-Hellman to discover the symmetric key.
To summarise: Signatures are created by the Yubikey. Encryption only requires the Yubikey to sign the outgoing message. Decryption only requires the Yubikey to recover the encapsulated key or to perform Diffie-Hellman to discover the key.
Note that when signing, you would not use the word 'encrypt' generally to describe the signing process. That is a technical implementation detail that might apply to RSA but not to EC signing.