When switching desktops in either GNOME or KDE, XMMS will often skip. This occured on my K6-3 550 with 192MB of RAM and a Sound Blaster PCI512 (emu10k1) with swap enabled, but a custom kernel and no swap doesnt exhibit the problem. A friend's 733 Celery ( > 128MB RAM and unknown soundchip) also exhibited the problem. vmstat didnt report large swapping so I doubt the change in swap is the problem mpg321 and ogg123 seem to work fine as well.
Switching desktops causes large X drawing usage; X is niced to -10; this can cause the media player to get blocked out.