Bug 1467247 - Accessing a file when source brick is down results in that FOP being hung
Accessing a file when source brick is down results in that FOP being hung
Product: GlusterFS
Classification: Community
Component: replicate (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Ravishankar N
: Triaged
Depends On: 1467250 1515572
  Show dependency treegraph
Reported: 2017-07-03 04:56 EDT by Ravishankar N
Modified: 2017-11-22 00:34 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2017-11-22 00:34:44 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Ravishankar N 2017-07-03 04:56:13 EDT
Description of problem:

Volume Name: arbvol
Type: Replicate
Volume ID: d7ccfadd-63e1-4fef-b70f-77d0e2cb2bba
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Brick1: node1:/gluster/brick1/b1
Brick2: node2:/gluster/brick1/b1
Brick3: node3:/gluster/brick1/b1 (arbiter)
Options Reconfigured:
cluster.self-heal-daemon: off
performance.strict-o-direct: on
cluster.granular-entry-heal: enable
network.ping-timeout: 10
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: on
network.remote-dio: off
performance.low-prio-threads: 32
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
nfs.disable: on
cluster.data-self-heal: off
cluster.metadata-self-heal: off
cluster.entry-self-heal: off

In a hc setup, writes, truncates and reads were happening from a single client while the bricks were continuously being killed and brought backup one by one in a loop. I/O was then getting hung from the mount.

gdb back trace  of mount:
#3  0x00007f45a762f5ee in shard_make_block_abspath (block_num=159197526, gfid=0x7f45a805a0a8 "$r\342\366{\260E\032\251\020\254>\312B\245\274", filepath=0x7f45ac9e1f90 "/.shard/2472e2f6-7bb0-451a-a910-ac3eca42a5bc.159197525", len=4096)
    at shard.c:57
#4  0x00007f45a7632605 in shard_common_resolve_shards (frame=0x7f45a8000b60, this=0x7f45a80121e0, post_res_handler=0x7f45a763a344 <shard_post_resolve_truncate_handler>) at shard.c:635
#5  0x00007f45a763337b in shard_refresh_dot_shard (frame=0x7f45a8000b60, this=0x7f45a80121e0) at shard.c:884
#6  0x00007f45a763b2a4 in shard_truncate_begin (frame=0x7f45a8000b60, this=0x7f45a80121e0) at shard.c:1989
#7  0x00007f45a763c7e3 in shard_post_lookup_truncate_handler (frame=0x7f45a8000b60, this=0x7f45a80121e0) at shard.c:2063
#8  0x00007f45a7634b79 in shard_lookup_base_file_cbk (frame=0x7f45a8000b60, cookie=0x7f45a804e130, this=0x7f45a80121e0, op_ret=0, op_errno=117, inode=0x7f45a805a0a0, buf=0x7f45a804ee68, xdata=0x7f45a8071ad0, postparent=0x7f45a804f098)
    at shard.c:1149
#9  0x00007f45a7892dd4 in dht_discover_complete (this=0x7f45a8010ab0, discover_frame=0x7f45a8060f60) at dht-common.c:577
#10 0x00007f45a78937f1 in dht_discover_cbk (frame=0x7f45a8060f60, cookie=0x7f45a800de50, this=0x7f45a8010ab0, op_ret=0, op_errno=117, inode=0x7f45a805a0a0, stbuf=0x7f45a807d890, xattr=0x7f45a8071ad0, postparent=0x7f45a807d900)
    at dht-common.c:700
#11 0x00007f45a7b71ad6 in afr_discover_done (frame=0x7f45a8066d60, this=0x7f45a800de50) at afr-common.c:2624
#12 0x00007f45a7b71e19 in afr_discover_cbk (frame=0x7f45a8066d60, cookie=0x2, this=0x7f45a800de50, op_ret=0, op_errno=0, inode=0x7f45a805a0a0, buf=0x7f45ac9e3900, xdata=0x7f45a806cb50, postparent=0x7f45ac9e3890) at afr-common.c:2669

shard_common_resolve_shards() was stuck in while (shard_idx_iter <= local->last_block) because local->last_block was -1. Turns out afr served the lookup from a bad copy (the good brick was down) and shard used the iatt values in the lookup cbk to calculate size and block count which were not correct.

xattr values of the FILE on the bad brick from which lookup was served:

[root@node2]# g /gluster/brick1/b1/FILE 
getfattr: Removing leading '/' from absolute path names
# file: gluster/brick1/b1/FILE
Comment 1 Karthik U S 2017-11-22 00:34:44 EST
This issue is fixed in master by the patch https://review.gluster.org/#/c/17673/. Closing this as 3.11 is EOL.

Note You need to log in before you can comment on or make changes to this bug.