Description of problem:

After a variety of gluster issues from BZ1535732, gluster block was left in an inoperable state.

Gluster block was working fine, there are provisioned block volumes that were in use.

Services, dependencies and iptables tables rules were all good:

[root@server ~]# lsmod | egrep 'dm_thin_pool|dm_multipath|target_core'
target_core_pscsi      19269  0
target_core_file       27502  0
target_core_iblock     27510  0
dm_thin_pool           65968  67
dm_persistent_data     74708  1 dm_thin_pool
dm_bio_prison          18209  1 dm_thin_pool
dm_multipath           27427  4 dm_service_time
target_core_user       23936  0
target_core_mod       367918  6 target_core_iblock,target_core_pscsi,iscsi_target_mod,target_core_file,target_core_user
uio                    19259  2 cnic,target_core_user
crc_t10dif             12714  2 target_core_mod,sd_mod
dm_mod                123303  441 dm_multipath,dm_log,dm_persistent_data,dm_mirror,dm_bufio,dm_thin_pool,dm_service_time

[root@server ~]# systemctl status multipathd.service
* multipathd.service - Device-Mapper Multipath Device Controller
   Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-01-17 10:46:23 PST; 2 days ago
  Process: 1241 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS)
  Process: 1232 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS)
  Process: 1218 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS)
 Main PID: 1264 (multipathd)
   Memory: 4.2M
   CGroup: /system.slice/multipathd.service
           `-1264 /sbin/multipathd

[root@server ~]# systemctl status rpcbind.socket
* rpcbind.socket - RPCbind Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-01-17 10:46:26 PST; 2 days ago
   Listen: /var/run/rpcbind.sock (Stream)
           [::]:111 (Stream)
           0.0.0.0:111 (Stream)

Jan 17 10:46:26 server.dmz systemd[1]: Listening on RPCbind Server Act....
Jan 17 10:46:26 server.dmz systemd[1]: Starting RPCbind Server Activat....


gluster block in gluster pods was not running:

[root@server ~]# ps -ef | grep gluster-blockd
root      21757 128598  0 12:44 pts/3    00:00:00 grep --color=auto gluster-blockd

saveconfig is empty:

sh-4.2# cat /etc/target/saveconfig.json
{
  "fabric_modules": [],
  "storage_objects": [],
  "targets": []
}



Version-Release number of selected component (if applicable):

CNS 3.6

oc exec glusterfs-hf25h -- rpm -qa glusterfs
glusterfs-3.8.4-44.el7rhgs.x86_64


How reproducible:

Random

Actual results:

Can no longer use or create block vols.

Expected results:


Additional info: