There is no remove action for linked categories.
Actually, there is. It just doesn't use the normal UI pattern. Perforce 32910 tries to clarify this a little. Closing.