I occasionally have to add extra swap to my desktop box,
so I do a:
# swapon /usr/local/swapfile
to get another 100Mb of swap. Upon shutdown, the swapfile
isn't unused, so /usr and /usr/local can't be unmounted.
I think you need something like:
swapoff `cut -f1 -d\ /proc/swaps | tail +2`
after the "swapoff -a". Either that, or swapoff needs to
check /proc/swaps, rather than /etc/fstab to find the list
of active swaps.
fixed in initscripts-4.38-2.