First, from a direct witness of the events (i.e. myself): Serpent was indeed felt as "too slow" when compared with Rijndael, by a factor of 2 to 3. The performance of Rijndael was not the best there was on a PC (RC6 was faster) but it wasn't abysmal on any platform, especially 8-bit smartcard (contrary to, say, RC6). Serpent performance was consistently lower than Rijndael's. See this page for benchmarks of the AES candidates on various software platforms at the time of the competition.
XSL attacks were discovered in 2002, hence after the choice of Rijndael as the AES; therefore, they could not have been a parameter in the choice.
In their paper, Courtois and Pierprzyk "show" that AES (i.e. Rijndael) with a 256-bit key, and Serpent with a 192-bit or 256-bit key, are "broken", which means that the attack ought to recover the key with less work than the CPU needed to evaluate the block cipher 2256 times (resp. 2192). They certainly did not go below 2128, so the break is only theoretical and unverifiable. That's part of the problem, too: the analysis is based on a number of probabilistic assumptions on an equation-solving algorithm in a situation where it is out of the question to actually run it on existing hardware. So XSL attacks are an effective break only insofar as their description is intellectually convincing. Don Coppersmith has publicly expressed that he was not convinced. And when Coppersmith says that your research stinks, it is time to worry.
Anyway, that an algorithm can be represented as a set of multivariate polynomial equations is not the key issue: any algorithm which can be implemented as a circuit is amenable to such a representation (e.g. because a NAND of x and y is equal to 1+xy, when computing modulo 2, and every boolean operator can be built out of NAND gates). Generally speaking, solving a set of multivariate polynomial equations is hard. Courtois and Pieprzyk argue that the specific structure of the equations in the case of Rijndael and Serpent can be exploited into running a dedicated variant of Shamir's re-linearization solving algorithm fast enough to be considered, academically speaking, as a "break".
The interesting part in XSL attacks is that resistance grows with the number of rounds, but not exponentially, contrary to more mundane differential or linear cryptanalysis. The bad part of XSL is that most cryptographers do not believe that it works as advertised, or at all. On the other hands, most think that Serpent is a more "conservative" design than AES, with a correspondingly "higher" security margin (but there is no really scientific way of quantifying such a "margin" so these are really all fuzzy feelings).
On the matter of attacks of debatable success, you could consider cache timing attacks, which are not an issue with the algorithm per se, but with its implementation. At least that attack could be demonstrated in lab conditions (the debate being whether the lab conditions have any similarity with whatever could occur in the field). Serpent is "naturally" immune to such attacks, because the normal implementations on software platforms will not use any table at all.