Bug 1558948

Summary: linux untar errors out at completion during disperse volume inservice upgrade
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Nag Pavan Chilakam <nchilaka>
Component: disperseAssignee: Ashish Pandey <aspandey>
Status: CLOSED ERRATA QA Contact: Upasana <ubansal>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.4CC: amukherj, aspandey, jahernan, nchilaka, pkarampu, rhinduja, rhs-bugs, sheggodu, storage-qa-internal, ubansal
Target Milestone: ---   
Target Release: RHGS 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.12.2-13 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1570538 (view as bug list) Environment:
Last Closed: 2018-09-04 06:44:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1598746    
Bug Blocks: 1503137    

Description Nag Pavan Chilakam 2018-03-21 11:38:18 UTC 
Description of problem:
-------------------
this problem was seen a couple of times.
When we do  an inservice upgrade of ec volume, with linux untar going on,
when the linux untar seems to complete , it errors out 


also refer to https://bugzilla.redhat.com/show_bug.cgi?id=1465289#c34

[root@dhcp35-126 kern]# cat async.tar.log.log
xz: (stdin): Unexpected end of input
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now



Version-Release number of selected component (if applicable):
--------------
3.12.2-5

How reproducible:
==========
have seen this twice on 3.4.0 release and once on 3.3.1-async ie 3.8.4-54-2


Steps to Reproduce:
1.do a linux untar while inservice upgrade is going on
2.
3.

Actual results:
------------
when linux untar seems to complete, it throws errors on stdout
Comment 6 Nag Pavan Chilakam 2018-03-28 07:24:33 UTC 
sosreports@ http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/nchilaka/bug.1558948/
Comment 33 Nag Pavan Chilakam 2018-05-16 12:35:40 UTC 
I have retried on a fresh setup and below is the observation:
1)have 6 node setup on 3.8.4-54.8 and mounted volume on 2 clients 
2)from both clients doing linux untar on different dedicated dirs
upgraded node 1
3)while healing was in progress , upgraded node 2
4)now opened a new terminal of client2 and created a new dir and wrote 1000 small files using dd
5)now trying to rename the small files is resulting for most of the files(not all) with EIO as below
mv: failed to access ‘file..127’: Input/output error
md5sum: file.128: Input/output error
mv: failed to access ‘file..128’: Input/output error
md5sum: file.129: Input/output error
mv: failed to access ‘file..129’: Input/output error
md5sum: file.130: Input/output error
mv: failed to access ‘file..130’: Input/output error
md5sum: file.131: Input/output error
mv: failed to access ‘file..131’: Input/output error
md5sum: file.132: Input/output error

[root@dhcp35-126 dd]# ls file.132
file.132


Also noticed that the untar on client2 had errors at the end(not seen on c1)

[root@dhcp35-126 dir2]# cat ~/312-upgrade.log 
tar: linux-4.16.8/scripts/dtc/include-prefixes: Cannot utime: Input/output error
tar: linux-4.16.8/scripts/dtc/include-prefixes: Cannot change ownership to uid 0, gid 0: Input/output error
tar: linux-4.16.8/scripts/dtc/include-prefixes: Cannot change mode to rwxrwxr-x: Input/output error
tar: Exiting with failure status due to previous errors



[2018-05-16 12:28:50.033398] W [MSGID: 122040] [ec-common.c:990:ec_prepare_update_cbk] 0-dispvol1-disperse-1: Failed to get size and version [Input/output error]
[2018-05-16 12:28:50.034200] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=110101, remaining=000000, good=110101, bad=001010)
[2018-05-16 12:28:50.034390] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=110101, remaining=000000, good=110101, bad=001010)
[2018-05-16 12:28:50.059831] I [MSGID: 109066] [dht-rename.c:1611:dht_rename] 0-dispvol1-dht: renaming /dd/file.64 (hash=dispvol1-disperse-1/cache=dispvol1-disperse-1) => /dd/file.c6243837663ca28eb4d6ac20af10bc31.64 (hash=dispvol1-disperse-0/cache=<nul>)
[2018-05-16 12:28:50.069354] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.070375] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.070943] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.072136] W [fuse-bridge.c:1788:fuse_rename_cbk] 0-glusterfs-fuse: 1173411: /dd/file.64 -> /dd/file.c6243837663ca28eb4d6ac20af10bc31.64 => -1 (Input/output error)
[2018-05-16 12:28:50.096771] I [MSGID: 109066] [dht-rename.c:1611:dht_rename] 0-dispvol1-dht: renaming /dd/file.65 (hash=dispvol1-disperse-0/cache=dispvol1-disperse-0) => /dd/file.079fec35f542a5d72e672823edb97ae4.65 (hash=dispvol1-disperse-0/cache=<nul>)
[2018-05-16 12:28:50.117439] W [MSGID: 122040] [ec-common.c:990:ec_prepare_update_cbk] 0-dispvol1-disperse-1: Failed to get size and version [Input/output error]
[2018-05-16 12:28:50.121880] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=110101, remaining=000000, good=110101, bad=001010)
[2018-05-16 12:28:50.122206] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=110101, remaining=000000, good=110101, bad=001010)
[2018-05-16 12:28:50.145237] I [MSGID: 109066] [dht-rename.c:1611:dht_rename] 0-dispvol1-dht: renaming /dd/file.66 (hash=dispvol1-disperse-1/cache=dispvol1-disperse-1) => /dd/file.85c0a22a2f40fdf698f7ce556b21a273.66 (hash=dispvol1-disperse-0/cache=<nul>)
[2018-05-16 12:28:50.159831] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.159942] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.160693] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.162071] W [fuse-bridge.c:1788:fuse_rename_cbk] 0-glusterfs-fuse: 1173433: /dd/file.66 -> /dd/file.85c0a22a2f40fdf698f7ce556b21a273.66 => -1 (Input/output error)
[2018-05-16 12:28:50.187206] I [MSGID: 109066] [dht-rename.c:1611:dht_rename] 0-dispvol1-dht: renaming /dd/file.67 (hash=dispvol1-disperse-0/cache=dispvol1-disperse-0) => /dd/file.4a4eca15570464f0b5d5104d0fd1951b.67 (hash=dispvol1-disperse-0/cache=<nul>)
[2018-05-16 12:28:50.198233] W [MSGID: 122040] [ec-common.c:990:ec_prepare_update_cbk] 0-dispvol1-disperse-1: Failed to get size and version [Input/output error]
[2018-05-16 12:28:50.207186] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=110101, remaining=000000, good=110101, bad=001010)
[2018-05-16 12:28:50.207321] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=110101, remaining=000000, good=110101, bad=001010)
[2018-05-16 12:28:50.234347] I [MSGID: 109066] [dht-rename.c:1611:dht_rename] 0-dispvol1-dht: renaming /dd/file.68 (hash=dispvol1-disperse-1/cache=dispvol1-disperse-1) => /dd/file.549df35424cdbbc46b32629dab845e87.68 (hash=dispvol1-disperse-1/cache=<nul>)
[2018-05-16 12:28:50.248168] E [MSGID: 122037] [ec-common.c:1914:ec_update_size_version_done] 0-dispvol1-disperse-1: Failed to update version and size [Input/output error]
[2018-05-16 12:28:50.251616] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=110101, remaining=000000, good=110101, bad=001010)
[2018-05-16 12:28:50.252555] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=110101, remaining=000000, good=110101, bad=001010)
[2018-05-16 12:28:50.258203] W [MSGID: 122040] [ec-common.c:990:ec_prepare_update_cbk] 0-dispvol1-disperse-1: Failed to get size and version [Input/output error]
[2018-05-16 12:28:50.282043] I [MSGID: 109066] [dht-rename.c:1611:dht_rename] 0-dispvol1-dht: renaming /dd/file.69 (hash=dispvol1-disperse-0/cache=dispvol1-disperse-0) => /dd/file.9e48d3b56cb2fafefc4179f9ae58966e.69 (hash=dispvol1-disperse-1/cache=<nul>)
[2018-05-16 12:28:50.293663] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.293761] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.294003] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-1: Operation failed on 1 of 6 subvolumes.(up=111111, mask=111111, remaining=000000, good=111011, bad=000100)
[2018-05-16 12:28:50.294343] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.294412] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.294447] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.294480] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.295708] E [MSGID: 122037] [ec-common.c:1914:ec_update_size_version_done] 0-dispvol1-disperse-1: Failed to update version and size [Input/output error]
[2018-05-16 12:28:50.303140] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.303203] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.303244] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.303298] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.307554] I [MSGID: 122058] [ec-heal.c:2383:ec_heal_do] 0-dispvol1-disperse-1: /dd/file.69: name heal successful on 3F
[2018-05-16 12:28:50.321665] E [MSGID: 122037] [ec-common.c:1914:ec_update_size_version_done] 0-dispvol1-disperse-1: Failed to update version and size [Input/output error]
[2018-05-16 12:28:50.335662] W [MSGID: 122040] [ec-common.c:990:ec_prepare_update_cbk] 0-dispvol1-disperse-1: Failed to get size and version [Input/output error]
[2018-05-16 12:28:50.344216] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=110101, remaining=000000, good=110101, bad=001010)
[2018-05-16 12:28:50.344253] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=110101, remaining=000000, good=110101, bad=001010)
[2018-05-16 12:28:50.368612] I [MSGID: 109066] [dht-rename.c:1611:dht_rename] 0-dispvol1-dht: renaming /dd/file.70 (hash=dispvol1-disperse-0/cache=dispvol1-disperse-0) => /dd/file.e07984df589cb38090e579ce0cda6c68.70 (hash=dispvol1-disperse-1/cache=<nul>)
[2018-05-16 12:28:50.377549] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.377742] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.378608] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.378677] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.378947] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.379044] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.379089] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.378705] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-1: Operation failed on 2 of 6 subvolumes.(up=111111, mask=111111, remaining=000000, good=011011, bad=100100)
[2018-05-16 12:28:50.380664] E [MSGID: 122037] [ec-common.c:1914:ec_update_size_version_done] 0-dispvol1-disperse-1: Failed to update version and size [Input/output error]
[2018-05-16 12:28:50.388146] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.388213] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.388253] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.388295] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.392521] I [MSGID: 122058] [ec-heal.c:2383:ec_heal_do] 0-dispvol1-disperse-1: /dd/file.70: name heal successful on 3F
[2018-05-16 12:28:50.406372] E [MSGID: 122037] [ec-common.c:1914:ec_update_size_version_done] 0-dispvol1-disperse-1: Failed to update version and size [Input/output error]
[2018-05-16 12:28:50.419880] W [MSGID: 122040] [ec-common.c:990:ec_prepare_update_cbk] 0-dispvol1-disperse-1: Failed to get size and version [Input/output error]
[2018-05-16 12:28:50.426325] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=110101, remaining=000000, good=110101, bad=001010)
[2018-05-16 12:28:50.426492] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=110101, remaining=000000, good=110101, bad=001010)
[2018-05-16 12:28:50.457661] I [MSGID: 109066] [dht-rename.c:1611:dht_rename] 0-dispvol1-dht: renaming /dd/file.71 (hash=dispvol1-disperse-0/cache=dispvol1-disperse-0) => /dd/file.c2f10afb5556452c6501148d64d230cc.71 (hash=dispvol1-disperse-0/cache=<nul>)
[2018-05-16 12:28:50.477228] W [MSGID: 122040] [ec-common.c:990:ec_prepare_update_cbk] 0-dispvol1-disperse-1: Failed to get size and version [Input/output error]
[2018-05-16 12:28:50.479663] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=110101, remaining=000000, good=110101, bad=001010)
[2018-05-16 12:28:50.479868] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=110101, remaining=000000, good=110101, bad=001010)
[2018-05-16 12:28:50.505608] I [MSGID: 109066] [dht-rename.c:1611:dht_rename] 0-dispvol1-dht: renaming /dd/file.72 (hash=dispvol1-disperse-1/cache=dispvol1-disperse-1) => /dd/file.093a71c84b0cee0cc2c9b902fcf93232.72 (hash=dispvol1-disperse-0/cache=<nul>)
[2018-05-16 12:28:50.515352] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.515455] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.515547] W [dict.c:1243:dict_foreach_match] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f167) [0x7f85e92de167] -->/lib64/libglusterfs.so.0(are_dicts_equal+0x67) [0x7f85f6e5aff7] -->/lib64/libglusterfs.so.0(dict_foreach_match+0xe3) [0x7f85f6e5af83] ) 0-dict: dict|match|action is NULL [Invalid argument]
[2018-05-16 12:28:50.515786] W [MSGID: 122053] [ec-common.c:161:ec_check_status] 0-dispvol1-disperse-0: Operation failed on 2 of 6 subvolumes.(up=111111, mask=111111, remaining=000000, good=011011, bad=100100)
[2018-05-16 12:28:50.516092] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.516146] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.516188] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.518299] E [MSGID: 122037] [ec-common.c:1914:ec_update_size_version_done] 0-dispvol1-disperse-0: Failed to update version and size [Input/output error]
[2018-05-16 12:28:50.526034] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.526108] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.526150] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.526191] I [dict.c:462:dict_get] (-->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2f935) [0x7f85e92de935] -->/usr/lib64/glusterfs/3.8.4/xlator/cluster/disperse.so(+0x2e6a4) [0x7f85e92dd6a4] -->/lib64/libglusterfs.so.0(dict_get+0xec) [0x7f85f6e5917c] ) 0-dict: !this || key=glusterfs.inodelk-count [Invalid argument]
[2018-05-16 12:28:50.527377] E [MSGID: 122037] [ec-common.c:1914:ec_update_size_version_done] 0-dispvol1-disperse-1: Failed to update version and size [Input/output error]
[2018-05-16 12:28:50.530220] I [MSGID: 122058] [ec-heal.c:2383:ec_heal_do] 0-dispvol1-disperse-0: /dd/file.72: name heal successful on 3F
[2018-05-16 12:28:50.543100] E [MSGID: 122037] [ec-common.c:1914:ec_update_size_version_done] 0-dispvol1-disperse-0: Failed to update version and size [Input/output error]
[2018-05-16 12:28:50.551103] E [MSGID: 122037] [ec-common.c:1914:ec_update_size_version_done] 0-dispvol1-disperse-1: Failed to update version and size [Input/output error]
[2018-05-16 12:28:50.559154] W [MSGID: 122040] [ec-common.c:990:ec_prepare_update_cbk] 0-dispvol1-disperse-0: Failed to get size and version [Input/output error]
[2018-05-16 12:28:50.562184] W [MSGID: 122040] [ec-common.c:990:ec_prepare_update_cbk] 0-dispvol1-disperse-1: Failed to get size and version [Input/output error]
[2018-05-16 12:28:50.569197] W [MSGID: 122040] [ec-common.c:990:ec_prepare_update_cbk] 0-dispvol1-disperse-0: Failed to get size and version [Input/output error]
[2018-05-16 12:28:50.573574] W [MSGID: 122040] [ec-common.c:990:ec_prepare_update_cbk] 0-dispvol1-disperse-1: Failed to get size and version [Input/output error]
[2018-05-16 12:28:50.582317] W [MSGID: 122040] [ec-common.c:990:ec_prepare_update_cbk] 0-dispvol1-disperse-0: Failed to get size and version [Input/output error]
[2018-05-16 12:28:50.583141] W [MSGID: 122040] [ec-common.c:990:ec_prepare_update_cbk] 0-dispvol1-disperse-1: Failed to get size and version [Input/output error]


and on c1 where untar didn't error out, I did an rm -rf and see EIO
[root@dhcp35-72 dir1]# rm -rf linux-4.16.8
rm: cannot remove ‘linux-4.16.8/Documentation/ABI/obsolete’: Input/output error
rm: cannot remove ‘linux-4.16.8/Documentation/ABI/stable’: Input/output error
rm: cannot remove ‘linux-4.16.8/Documentation/PCI/endpoint’: Input/output error
rm: cannot remove ‘linux-4.16.8/Documentation/RCU/Design/Data-Structures’: Input/output error


hence moving to failed_qa
Comment 40 Nag Pavan Chilakam 2018-07-12 08:15:22 UTC 
onqa validation blocked due to BZ#1598746 refer 1598746#c13
Comment 44 errata-xmlrpc 2018-09-04 06:44:20 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607