I was looking at the algorithm for Twofish, and I noticed that in some places a XOR is used, but in others, they use "addition modulo-32." What makes modulo-32 special? Why not always use XOR? Why not always addition mod 32?

-

The combination between addition modulo $2^{32}$ (not modulo $32 = 2^5$) - indicated by $\boxplus$ in the diagram - and XOR (i.e. bitwise addition modulo $2$) - indicated by $\oplus$ - makes the algorithm more non-linear.
Each of them for itself is a linear operation, but over different groups (addition in $GF(2^{32})$ vs. addition in $Z/2^{32})$, and the combination is slightly non-linear over both groups.
Why modulo $2^{32}$? This operation is already implemented in many processors (Java's int addition is this, for example), and this makes implementation easy and efficient.