First of all, if the order is not important then rotating a hash value depending on the order before using it would of course be counter-intuitive. Now the hash of a specific element is fully dependent on the order.
Generally I would not advice XOR-ing hashes.
Single rotation won't work especially if you combine multiple hashes - as you've indicated it would be easy to find a collision where two hashes cancel each other out. But you've already covered this.
XOR-ing would also make collisions easier to calculate, because the XOR of two hash values can also create a collision. Of course this won't change the order of finding a collision, but a XOR is still significantly faster than calculating a hash.
I could also imagine a scheme where you XOR (rotated) hashes that have a similar highest bit set. I think this could quickly create hashes that have the initial bits set to all zero. You could also XOR hashes that have a small Hamming distance. Either of these methods would create a set of hashes that have more than a normal amount of bits set to zero. It seems likely that these have less collision resistance than the initial hash. The fact that you can also use rotated hashes for this would make it even easier to create such attacks.
Instead you could think of sorting the hashes before you hash them (using a binary compare). That way you get a unique hash for a unique set where the order is ignored, even if it can contain identical elements. The disadvantage is that you cannot calculate a new hash value by adding a hash value after the final hash calculation is performed.