I would have phrased it as not having evidence supporting it as safe to use.
The ChaCha permutation was not designed to be used in a sponge construction. It could be safe, but we don't have evidence of any weight similar to the weight of evidence we have suggesting that the whole ChaCha20 stream cipher is safe.
I'm not aware of any attack on the ChaCha permutation if used as a sponge, unless capacity bits are initialized to zero. ChaCha's trivial fixed point is from zero to zero. If you can force the sponge's state to that value, then you can insert one or more blocks of all zero bits between two message blocks without changing the sponge's state. (Making collision attacks trivial.)
It is (most likely) non-trivial to get the state (capacity) bits to the zero if they aren't already that value. If the capacity bits are non-zero and the number of capacity bits is sufficiently large and the ChaCha permutation would be indifferentiable from a random permutation if not for the one known fixed point, then there might not be a way to exploit the fixed point in practice.
But, again, that third point isn't something that we can assume is true with the same confidence that we have for assuming that the stream cipher algorithm, ChaCha20, is secure. There is a lack of evidence because not enough effort has been put into studying the permutation in isolation.