Created attachment 1358524 [details]
hosts

Description of problem:
Killed glusterd on node 1 out of 4 and state is shown healthy in the dashboard

Version-Release number of selected component (if applicable):
3.8.4-52

How reproducible:
100%

Steps to Reproduce:
1. killed glusterd on node 1 
2. wait 15 mins to update the status on dashboard
3. check the status of the hosts on main dashboard. 

Actual results:
showing healthy


Expected results:
should be unhealthy or degraded.

Additional info:
root@gqas004 ~]# gluster v status
Status of volume: arbiter-2
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick gqas012.sbu.lab.eng.bos.redhat.com:/b
ricks/arbiter_21                            49152     0          Y       27311
Brick gqas010.sbu.lab.eng.bos.redhat.com:/b
ricks/arbiter_2                             49154     0          Y       11163
Brick gqas004.sbu.lab.eng.bos.redhat.com:/b
ricks/arbiter_2                             49154     0          Y       28092
Brick gqas004.sbu.lab.eng.bos.redhat.com:/b
ricks/arbiter-3                             49152     0          Y       24506
Brick gqas010.sbu.lab.eng.bos.redhat.com:/b
ricks/arbiter-3                             49152     0          Y       23997
Self-heal Daemon on localhost               N/A       N/A        Y       28130
Self-heal Daemon on gqas012.sbu.lab.eng.bos
.redhat.com                                 N/A       N/A        Y       23064
Self-heal Daemon on gqas010.sbu.lab.eng.bos
.redhat.com                                 N/A       N/A        Y       27601
 
Task Status of Volume arbiter-2
------------------------------------------------------------------------------
There are no active volume tasks
 
Status of volume: testvol
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick gqas004.sbu.lab.eng.bos.redhat.com:/b
ricks/testvol_brick1                        49153     0          Y       21951
Brick gqas010.sbu.lab.eng.bos.redhat.com:/b
ricks/testvol_brick2                        49153     0          Y       21535
Brick gqas012.sbu.lab.eng.bos.redhat.com:/b
ricks/testvol_brick3                        49153     0          Y       21212
Self-heal Daemon on localhost               N/A       N/A        Y       28130
Self-heal Daemon on gqas012.sbu.lab.eng.bos
.redhat.com                                 N/A       N/A        Y       23064
Self-heal Daemon on gqas010.sbu.lab.eng.bos
.redhat.com                                 N/A       N/A        Y       27601
 
Task Status of Volume testvol
------------------------------------------------------------------------------
Task                 : Rebalance           
ID                   : c3a6f652-4ce6-4768-9a73-a3d6e857b988
Status               : completed           
 
[root@gqas004 ~]# 




root@gqas004 ~]# gluster v info
 
Volume Name: arbiter-2
Type: Distributed-Replicate
Volume ID: aadda253-b450-403c-8115-e8d8f41d0feb
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: gqas012.sbu.lab.eng.bos.redhat.com:/bricks/arbiter_21
Brick2: gqas010.sbu.lab.eng.bos.redhat.com:/bricks/arbiter_2
Brick3: gqas004.sbu.lab.eng.bos.redhat.com:/bricks/arbiter_2 (arbiter)
Brick4: gqas001.sbu.lab.eng.bos.redhat.com:/bricks/arbiter-3
Brick5: gqas004.sbu.lab.eng.bos.redhat.com:/bricks/arbiter-3
Brick6: gqas010.sbu.lab.eng.bos.redhat.com:/bricks/arbiter-3 (arbiter)
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
transport.address-family: inet
nfs.disable: on
 
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 14cec5af-14b3-407a-b849-b0e68e979ea1
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gqas001.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0
Brick2: gqas004.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1
Brick3: gqas010.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2
Brick4: gqas012.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
client.event-threads: 4
server.event-threads: 4
cluster.lookup-optimize: on
network.inode-lru-limit: 90000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
server.allow-insecure: on
performance.stat-prefetch: on
transport.address-family: inet
nfs.disable: on
[root@gqas004 ~]#