Created attachment 1144029 [details]
glustershd for server with chatty output. Heals are as a result of glusterd restarts.

Description of problem:

Since upgrading to 3.7.10, I have been experiencing inconsistent or possibly erroneous outputs for "volume heal <vol> info" commands.


Version-Release number of selected component (if applicable): 3.7.10-1


How reproducible: Unknown


Steps to Reproduce:
1. With synchronized 3.7.9 volume, upgrade all bricks to 3.7.10
2. Allow all bricks to synchronize.
3. Start using Volume in production. (In this case, as virtual machine storage)
4. gluster volume heal <vol> info.

Actual results:

Inconsistent or chatty outputs. Possibly symbolizing an out of sync volume but not likely, no bricks went down, nor was this experienced in 3.7.9.

Expected results:

Empty entry list in command output.

Additional info:

Volume Name: TriDistRepl
Type: Distributed-Replicate
Volume ID: 08a2dad5-0d43-4458-8c5b-a17add9cb3e4
Status: Started
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: rhev.styx.local:/virtual/TriDistRepl
Brick2: host.styx.local:/virtual/TriDistRepl
Brick3: gfs1.styx.local:/virtual/TriDistRepl
Brick4: repl.styx.local:/virtual/TriDistRepl
Brick5: fs1.styx.local:/media/DataPartRaid5/virtual/TriDistRepl
Brick6: gfs2.styx.local:/virtual/TriDistRepl
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
storage.owner-uid: 36
server.allow-insecure: on
storage.owner-gid: 36
network.ping-timeout: 10
features.shard-block-size: 512MB
features.shard: on

Sample Volume Heal Output:
gluster> vol heal TriDistRepl info
Brick rhev.styx.local:/virtual/TriDistRepl
Status: Connected
Number of entries: 0

Brick host.styx.local:/virtual/TriDistRepl
Status: Connected
Number of entries: 0

Brick gfs1.styx.local:/virtual/TriDistRepl
Status: Connected
Number of entries: 0

Brick repl.styx.local:/virtual/TriDistRepl
/5d8b9ed2-7dae-472f-90f7-904085d4dbf9/images/20475fd1-d0eb-4cb6-82a9-09d43a07397f/ba91e4f9-d144-41a0-901e-b7b78ae395b5
/5d8b9ed2-7dae-472f-90f7-904085d4dbf9/images/1d9867ba-37ce-4b27-af1e-2d965e5338c0/7a518bca-a2e3-4b50-a007-b2a8913a5ce4
Status: Connected
Number of entries: 2

Brick fs1.styx.local:/media/DataPartRaid5/virtual/TriDistRepl
/5d8b9ed2-7dae-472f-90f7-904085d4dbf9/images/20475fd1-d0eb-4cb6-82a9-09d43a07397f/ba91e4f9-d144-41a0-901e-b7b78ae395b5
Status: Connected
Number of entries: 1

Brick gfs2.styx.local:/virtual/TriDistRepl
/5d8b9ed2-7dae-472f-90f7-904085d4dbf9/images/f19279f0-2839-44d6-b36c-09f0995ea47f/be06560c-2edf-44fa-ae4b-cbf68961e838
/5d8b9ed2-7dae-472f-90f7-904085d4dbf9/images/c0c9d41c-e354-4e34-9e8f-d03066bd0d5c/3e9e133e-248a-404a-a93e-5ead81d43fa4
Status: Connected
Number of entries: 2


Same command ran from different brick in same volume:

gluster> vol heal TriDistRepl info
Brick rhev.styx.local:/virtual/TriDistRepl
Status: Connected
Number of entries: 0

Brick host.styx.local:/virtual/TriDistRepl
Status: Connected
Number of entries: 0

Brick gfs1.styx.local:/virtual/TriDistRepl
Status: Connected
Number of entries: 0

Brick repl.styx.local:/virtual/TriDistRepl
Status: Connected
Number of entries: 0

Brick fs1.styx.local:/media/DataPartRaid5/virtual/TriDistRepl
Status: Connected
Number of entries: 0

Brick gfs2.styx.local:/virtual/TriDistRepl
Status: Connected
Number of entries: 0


Other troubleshooting steps taken:
-Restarted bricks experiencing the chatty outputs.