$ rpm -e docbook-dtds openjade
Failed to remove entry from /etc/sgml/catalog
/var/tmp/rpm-tmp.15271: line 59: [: : integer expression expected

It appears the section of %%postun is in the if:

CATALOG=/etc/sgml/catalog
SGML=$(cat -n ${CATALOG} | grep sgml-docbook | head -1 | (read n line;echo $n))
XML=$(cat -n ${CATALOG} | grep xml-docbook | head -1 | (read n line; echo $n))
# Do they need switching around?
if [ "${XML}" -lt "${SGML}" ]

In that sometimes either SGML or XML end up being empty/blank when it probably
ought to be considerred to be 0.