Is this safe? Am I doing something dumb?
Well, I don't think it's actually safe; however why it's not safe would depend on the details of what your library has implemented within the Poly1305 function.
Straight Poly1305 isn't a MAC; it's actually a universal hash. That is, it takes a message and a key and produces a value; however (unlike a MAC) it's sole guarrantee is that, for two different messages $M, M'$, we'll have $\text{Poly1305}_K(M) = \text{Poly1305}_K(M')$ for only a tiny fraction of the keys (and so if the key $K$ is selected randomly, this is quite improbable.
What this doesn't guarantee is that someone looking at $\text{Poly1305}_K(M)$ values for various $M$'s would have any difficulty whatsoever about recovering $K$, and for the Poly1305 function itself, recovering $K$ is easy. Hence, if your Poly1305 function implements only the universal hash function itself, well, it's broken that way.
Now, this use of Poly1305 doesn't actually use a nonce, and so I'm not sure if you're using it straight.
To use Poly1305 as a MAC, what we do is compute the Poly1305 of the message, and then encrypt the tag with the nonce, using some cipher (AES or ChaCha, most typically). This is safe; the Poly1305 prevents someone from producing two messages that will have the same MAC (because with an unknown Poly1305 key, that's unlikely), and the AES/ChaCha cipher will prevent someone from seeing the Poly1305 output directly (and so recovering things that way).
However, we do this encryption by taking the nonce, sending that through the cipher, and then adding that to the Poly1305 output; if we use the same nonce repeatedly, that means that we'll be adding a constant value to the Poly1305, and the attacker can adjust to that.
On the other hand, things doing this generally call out the cipher being used, say, as Poly1305-AES or Poly1305-ChaCha
So, I'm not sure why what you're doing isn't safe; but it doesn't look good.
Is there a better way?
If you find it inconvenient to use a different nonce for every token, might I suggest using HMAC (based on some reasonable hash function, say SHA256)? That doesn't need a nonce, and hence you're secure (or, at least, as long as you can keep the key secret). It's not as fast as Poly1305; it wouldn't appear you care about that.
