You wrote in the comments:
no, I am only interested in the MAC generation using Poly1305 with a key that is always been freshly generated for each execution of the MAC function. The key is derived using an HKDF with a shared DH key and a random value for each execution (e.g. a nonce).
In the same RFC that you mention it is written (quoting Bernstein's paper):
The original article ([Poly1305]) is titled "The Poly1305-AES
message-authentication code", and the MAC function there requires a
128-bit AES key, a 128-bit "additional key", and a 128-bit (non-
secret) nonce. AES is used there for encrypting the nonce, so as to
get a unique (and secret) 128-bit string, but as the paper states,
"There is nothing special about AES here. One can replace AES with
an arbitrary keyed function from an arbitrary set of nonces to
16-byte strings."
Now you've got HKDF, which is a keyed function, a nonce and an output which can be made 16 bytes. I say: go for it, but don't use the key for anything else, including ChaCha20 encryption or worse, calculating another authentication tag.
Just for reassurance: do please use SHA-256 or better for HKDF. Then it should be just as secure.
Do remember that you're now in unchartered territory. That means using libraries to implement this kind of functionality will be harder, compared - for instance - to just using the AAD of a ChaCha20/Poly1305 cipher to calculate a MAC.