Created attachment 1349460 [details]
log from pod shutdown with block PV

Description of problem:

Bring up a gluster block backed pod and you see the multipath device with 3x paths.

[cloud-user@osenode4 ~]$ sudo multipath -ll
mpathb (360014050b2a1e10336a4600ae4c2eec5) dm-32 LIO-ORG ,TCMU device
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 5:0:0:0 sda 8:0  active ready running
|-+- policy='round-robin 0' prio=1 status=enabled
| `- 6:0:0:0 sdb 8:16 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 7:0:0:0 sdc 8:32 active ready running

When the POD is moved / scaled down / deleted, the the muultipath device remains:

[cloud-user@osenode4 ~]$ sudo multipath -ll
mpathb (360014050b2a1e10336a4600ae4c2eec5) dm-32
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw

After the fact, a second flush removes it:

[cloud-user@osenode4 ~]$ sudo multipath -f mpathb
Nov 08 09:04:15 | mpathb: map in use
Nov 08 09:04:15 | failed to remove multipath map mpathb

[cloud-user@osenode4 ~]$ sudo multipath -f mpathb
[cloud-user@osenode4 ~]$ sudo multipath -ll

Log Attached.

Shutdown At: Nov 8 08:58:00

First Multipath -f At (failed): Nov 8 09:04:15

Second Multipath -f At (successful): Nov 8 09:04:38

Version-Release number of selected component (if applicable):

CNS 3.6

How reproducible:

Always

Steps to Reproduce:
1. Deploy pod with block PV
2. Scale down to 0
3. Check multipath 

Actual results:

Left over device

Expected results:

Everything gets cleaned up

Additional info: