Yes, it must be removed if you want test server mode.
As long as there a way to deliver absolute mouse coordinates spice will pick client mode. The agent is the first option, usb tablet the second and vmmouse the third, so for testing server mode you must make sure none of these three is used.