Description of problem: ======================= Was doing an inservice upgrade from 3.4.4 to 3.5.0 on a 6 node setup With a distributed-dispersed volume and brickmux enabled setup Version-Release number of selected component (if applicable): ============================================================= 2 nodes still on 3.4.4 4 nodes on 3.5.0 (Build 3) How reproducible: ================ 1/1 Steps to Reproduce: ================== 1.Create a distributed-dispersed volume with brick mux enabled on a 3.4.4 setup 2.Mount the volume and start the IO's 3.Upgraded 2 nodes at a time and wait for healing to complete -- This completed successfully 4.Upgrade the next 2 nodes and start healing 5.Healing is not progressing from the past 5 hours (Has 150 files in heal info from then) Actual results: =============== Healing is not completing Expected results: ================ Healing should complete successfully Additional info: ================ [root@dhcp35-186 ec-vol]# tail -f shd.log [2019-05-24 12:40:17.323674] E [MSGID: 114031] [client-rpc-fops.c:1457:client3_3_inodelk_cbk] 0-ec-vol-client-9: remote operation failed [Invalid argument] [2019-05-24 12:40:27.366609] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-ec-vol-client-1: remote operation failed [Invalid argument] [2019-05-24 12:40:27.366765] E [MSGID: 114031] [client-rpc-fops.c:1457:client3_3_inodelk_cbk] 0-ec-vol-client-4: remote operation failed [Invalid argument] [2019-05-24 12:40:27.366849] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-ec-vol-client-2: remote operation failed [Invalid argument] [2019-05-24 12:40:27.370783] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-ec-vol-client-5: remote operation failed [Invalid argument] [2019-05-24 12:40:27.404860] E [MSGID: 114031] [client-rpc-fops.c:1457:client3_3_inodelk_cbk] 0-ec-vol-client-3: remote operation failed [Invalid argument] [2019-05-24 12:40:41.908206] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-ec-vol-client-5: remote operation failed [Invalid argument] [2019-05-24 12:40:41.940410] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-ec-vol-client-5: remote operation failed [Invalid argument] [2019-05-24 12:40:50.005904] E [MSGID: 114031] [client-rpc-fops.c:1457:client3_3_inodelk_cbk] 0-ec-vol-client-4: remote operation failed [Invalid argument] [2019-05-24 12:40:58.997907] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-ec-vol-client-7: remote operation failed [Invalid argument] [2019-05-24 12:41:46.825432] E [inodelk.c:513:__inode_unlock_lock] 1-ec-vol-locks: Matching lock not found for unlock 0-9223372036854775807, by 582f0e70ba7f0000 on 0x7f3c041f9c70 [2019-05-24 12:41:46.825565] E [MSGID: 115053] [server-rpc-fops.c:297:server_inodelk_cbk] 0-ec-vol-server: 871533: INODELK /linux/linux-5.1.4/drivers/infiniband/core/iwcm.c (97413fd8-9b1c-465f-a2db-d6c115d3d861), client: dhcp35-27.lab.eng.blr.redhat.com-12328-2019/05/24-07:22:48:731990-ec-vol-client-8-0-1, error-xlator: ec-vol-locks [Invalid argument] [2019-05-24 12:41:48.635832] I [dict.c:541:dict_get] (-->/usr/lib64/glusterfs/6.0/xlator/features/worm.so(+0x7241) [0x7f3c09280241] -->/usr/lib64/glusterfs/6.0/xlator/features/locks.so(+0x1c219) [0x7f3c094a7219] -->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f3c195b02f4] ) 1-dict: !this || key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument] [2019-05-24 12:41:48.865078] I [dict.c:541:dict_get] (-->/usr/lib64/glusterfs/6.0/xlator/features/worm.so(+0x7241) [0x7f3c09280241] -->/usr/lib64/glusterfs/6.0/xlator/features/locks.so(+0x1c219) [0x7f3c094a7219] -->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f3c195b02f4] ) 1-dict: !this || key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument] [2019-05-24 12:41:49.051621] I [dict.c:541:dict_get] (-->/usr/lib64/glusterfs/6.0/xlator/features/worm.so(+0x7241) [0x7f3c09280241] -->/usr/lib64/glusterfs/6.0/xlator/features/locks.so(+0x1c219) [0x7f3c094a7219] -->/lib64/libglusterfs.so.0(dict_get+0x94) [0x7f3c195b02f4] ) 1-dict: !this || key=trusted.glusterfs.enforce-mandatory-lock [Invalid argument] [2019-05-24 12:41:51.247490] E [inodelk.c:513:__inode_unlock_lock] 0-ec-vol-locks: Matching lock not found for unlock 0-9223372036854775807, by 3813439cba7f0000 on 0x7f3bfc0333d0 [2019-05-24 12:41:51.248990] E [MSGID: 115053] [server-rpc-fops.c:297:server_inodelk_cbk] 0-ec-vol-server: 865423: INODELK /linux/linux-5.1.4/drivers/iio/potentiometer/ad5272.c (2f07c222-ca25-489c-9f4a-71cf871c3f6c), client: dhcp35-27.lab.eng.blr.redhat.com-12328-2019/05/24-07:22:48:731990-ec-vol-client-2-0-1, error-xlator: ec-vol-locks [Invalid argument]
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:3249