+++ This bug was initially created as a clone of Bug #1727081 +++ Description of problem: LTP ftestxx tests reports data corruption at a 4+2 disperse volume. <<<test_output>>> ftest05 1 TFAIL : ftest05.c:395: Test[0] bad verify @ 0x3800 for val 2 count 487 xfr 2048 file_max 0xfa000. ftest05 0 TINFO : Test[0]: last_trunc = 0x4d800 ftest05 0 TINFO : Stat: size=fa000, ino=120399ba ftest05 0 TINFO : Buf: ftest05 0 TINFO : 64*0, ftest05 0 TINFO : 2, ftest05 0 TINFO : 2, ftest05 0 TINFO : 2, ftest05 0 TINFO : 2, ftest05 0 TINFO : 2, ftest05 0 TINFO : 2, ftest05 0 TINFO : 2, ftest05 0 TINFO : 2, ftest05 0 TINFO : 2, ftest05 0 TINFO : 2, ftest05 0 TINFO : ... more ftest05 0 TINFO : Bits array: ftest05 0 TINFO : 0: ftest05 0 TINFO : 0: ftest05 0 TINFO : ddx ftest05 0 TINFO : 8: ftest05 0 TINFO : ecx Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Worker Ant on 2019-07-05 00:06:33 UTC --- REVIEW: https://review.gluster.org/22999 (cluster/ec: inherit right mask from top parent) posted (#1) for review on master by Kinglong Mee --- Additional comment from Worker Ant on 2019-07-08 13:27:01 UTC --- REVIEW: https://review.gluster.org/23010 (cluster/ec: inherit healing from lock which has info) posted (#1) for review on master by Kinglong Mee --- Additional comment from Pranith Kumar K on 2019-07-10 10:30:39 UTC --- (In reply to Kinglong Mee from comment #0) > Description of problem: > > LTP ftestxx tests reports data corruption at a 4+2 disperse volume. > > <<<test_output>>> > ftest05 1 TFAIL : ftest05.c:395: Test[0] bad verify @ 0x3800 > for val 2 count 487 xfr 2048 file_max 0xfa000. > ftest05 0 TINFO : Test[0]: last_trunc = 0x4d800 > ftest05 0 TINFO : Stat: size=fa000, ino=120399ba > ftest05 0 TINFO : Buf: > ftest05 0 TINFO : 64*0, > ftest05 0 TINFO : 2, > ftest05 0 TINFO : 2, > ftest05 0 TINFO : 2, > ftest05 0 TINFO : 2, > ftest05 0 TINFO : 2, > ftest05 0 TINFO : 2, > ftest05 0 TINFO : 2, > ftest05 0 TINFO : 2, > ftest05 0 TINFO : 2, > ftest05 0 TINFO : 2, > ftest05 0 TINFO : ... more > ftest05 0 TINFO : Bits array: > ftest05 0 TINFO : 0: > ftest05 0 TINFO : 0: > ftest05 0 TINFO : ddx > ftest05 0 TINFO : 8: > ftest05 0 TINFO : ecx When I try to run this test, it is choosing /tmp as the directory where the file is created. How to change it to the mount directory? root@localhost - /mnt/ec2 15:11:08 :( ⚡ /opt/ltp/testcases/bin/ftest05 ftest05 1 TPASS : Test passed. > > > Version-Release number of selected component (if applicable): > > > How reproducible: > > > Steps to Reproduce: > 1. > 2. > 3. > > Actual results: > > > Expected results: > > > Additional info: --- Additional comment from Kinglong Mee on 2019-07-10 12:47:01 UTC --- (In reply to Pranith Kumar K from comment #3) > (In reply to Kinglong Mee from comment #0) > > Description of problem: > > > > LTP ftestxx tests reports data corruption at a 4+2 disperse volume. > > > > <<<test_output>>> > > ftest05 1 TFAIL : ftest05.c:395: Test[0] bad verify @ 0x3800 > > for val 2 count 487 xfr 2048 file_max 0xfa000. > > ftest05 0 TINFO : Test[0]: last_trunc = 0x4d800 > > ftest05 0 TINFO : Stat: size=fa000, ino=120399ba > > ftest05 0 TINFO : Buf: > > ftest05 0 TINFO : 64*0, > > ftest05 0 TINFO : 2, > > ftest05 0 TINFO : 2, > > ftest05 0 TINFO : 2, > > ftest05 0 TINFO : 2, > > ftest05 0 TINFO : 2, > > ftest05 0 TINFO : 2, > > ftest05 0 TINFO : 2, > > ftest05 0 TINFO : 2, > > ftest05 0 TINFO : 2, > > ftest05 0 TINFO : 2, > > ftest05 0 TINFO : ... more > > ftest05 0 TINFO : Bits array: > > ftest05 0 TINFO : 0: > > ftest05 0 TINFO : 0: > > ftest05 0 TINFO : ddx > > ftest05 0 TINFO : 8: > > ftest05 0 TINFO : ecx > > > When I try to run this test, it is choosing /tmp as the directory where the > file is created. How to change it to the mount directory? > root@localhost - /mnt/ec2 > 15:11:08 :( ⚡ /opt/ltp/testcases/bin/ftest05 > ftest05 1 TPASS : Test passed. You can run as, ./runltp -p -l /tmp/resut.log -o /tmp/output.log -C /tmp/failed.log -d /mnt/nfs/ -f casefilename-under-runtest When running the test at nfs client, there is a bash scripts running which reboot one node(the cluster node Ganesha.nfsd is not running on) every 600s. --- Additional comment from Kinglong Mee on 2019-07-11 10:32:22 UTC --- valgrind reports some memory leak, ==7925== 300 bytes in 6 blocks are possibly lost in loss record 880 of 1,436 ==7925== at 0x4C29BC3: malloc (vg_replace_malloc.c:299) ==7925== by 0x71828BF: __gf_default_malloc (mem-pool.h:112) ==7925== by 0x7183182: __gf_malloc (mem-pool.c:131) ==7925== by 0x713FB65: gf_strndup (mem-pool.h:189) ==7925== by 0x713FBD5: gf_strdup (mem-pool.h:206) ==7925== by 0x7144465: loc_copy (xlator.c:1276) ==7925== by 0x18EDBF1C: ec_loc_from_loc (ec-helpers.c:760) ==7925== by 0x18F02FE5: ec_manager_open (ec-inode-read.c:778) ==7925== by 0x18EE4905: __ec_manager (ec-common.c:3094) ==7925== by 0x18EE4A0F: ec_manager (ec-common.c:3112) ==7925== by 0x18F037F3: ec_open (ec-inode-read.c:929) ==7925== by 0x18ED5E85: ec_gf_open (ec.c:1146) --- Additional comment from Worker Ant on 2019-07-11 11:05:58 UTC --- REVIEW: https://review.gluster.org/23029 (cluster/ec: do loc_copy from ctx->loc in fd->lock) posted (#1) for review on master by Kinglong Mee --- Additional comment from Kinglong Mee on 2019-07-12 00:47:27 UTC --- ganesha.nfsd crash when healing name, Core was generated by `/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N N'. Program terminated with signal 11, Segmentation fault. #0 0x00007f0d5ae8c5a9 in ec_heal_name (frame=0x7f0d57c6ca28, ec=0x7f0d5b62d280, parent=0x0, name=0x7f0d57537d31 "b", participants=0x7f0d0dfffe30 "\001\001\001") at ec-heal.c:1685 1685 loc.inode = inode_new(parent->table); Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 dbus-libs-1.10.24-12.el7.x86_64 elfutils-libelf-0.172-2.el7.x86_64 elfutils-libs-0.172-2.el7.x86_64 glibc-2.17-260.el7.x86_64 gssproxy-0.7.0-21.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-34.el7.x86_64 libacl-2.2.51-14.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-59.el7.x86_64 libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-13.el7.x86_64 libgcc-4.8.5-36.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libnfsidmap-0.25-19.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 libuuid-2.23.2-59.el7.x86_64 lz4-1.7.5-2.el7.x86_64 openssl-libs-1.0.2k-16.el7.x86_64 pcre-8.32-17.el7.x86_64 systemd-libs-219-62.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64 (gdb) bt #0 0x00007f0d5ae8c5a9 in ec_heal_name (frame=0x7f0d57c6ca28, ec=0x7f0d5b62d280, parent=0x0, name=0x7f0d57537d31 "b", participants=0x7f0d0dfffe30 "\001\001\001") at ec-heal.c:1685 #1 0x00007f0d5ae93cae in ec_heal_do (this=0x7f0d5b65ac00, data=0x7f0d24e3c028, loc=0x7f0d24e3c358, partial=0) at ec-heal.c:3050 #2 0x00007f0d5ae94455 in ec_synctask_heal_wrap (opaque=0x7f0d24e3c028) at ec-heal.c:3139 #3 0x00007f0d6d1268c9 in synctask_wrap () at syncop.c:369 #4 0x00007f0d6c6bf010 in ?? () from /lib64/libc.so.6 #5 0x0000000000000000 in ?? () (gdb) frame 1 #1 0x00007f0d5ae93cae in ec_heal_do (this=0x7f0d5b65ac00, data=0x7f0d24e3c028, loc=0x7f0d24e3c358, partial=0) at ec-heal.c:3050 3050 ret = ec_heal_name(frame, ec, loc->parent, (char *)loc->name, (gdb) p loc $1 = (loc_t *) 0x7f0d24e3c358 (gdb) p *loc $2 = { path = 0x7f0d57537d00 "/nfsshare/ltp-eZQlnozjnX/ftegVRmbT/ftest05.20436/b", name = 0x7f0d57537d31 "b", inode = 0x7f0d24255b28, parent = 0x0, gfid = "\263\341\223\031\301\245I\260\234\334\017\to%\305^", pargfid = '\000' <repeats 15 times>} --- Additional comment from Xavi Hernandez on 2019-07-13 14:09:15 UTC --- Please, don't use the same bug for different issues. --- Additional comment from Worker Ant on 2019-07-14 13:03:20 UTC --- REVISION POSTED: https://review.gluster.org/23029 (cluster/ec: do loc_copy from ctx->loc in fd->lock) posted (#2) for review on master by Kinglong Mee --- Additional comment from Worker Ant on 2019-07-16 17:54:25 UTC --- REVIEW: https://review.gluster.org/23010 (cluster/ec: inherit healing from lock when it has info) merged (#4) on master by Amar Tumballi --- Additional comment from Ashish Pandey on 2019-07-17 05:27:44 UTC --- There are two patches associated with this BZ - https://review.gluster.org/#/c/glusterfs/+/22999/ - No merged and under review https://review.gluster.org/#/c/glusterfs/+/23010/ - Merged I would like to keep this bug open till both the patches get merged. -- Ashish --- Additional comment from Worker Ant on 2019-07-18 07:28:12 UTC --- REVIEW: https://review.gluster.org/23069 ((WIP)cluster/ec: Always read from good-mask) posted (#1) for review on master by Pranith Kumar Karampuri --- Additional comment from Worker Ant on 2019-07-23 06:20:08 UTC --- REVIEW: https://review.gluster.org/23073 (cluster/ec: fix data corruption) posted (#4) for review on master by Pranith Kumar Karampuri
Created attachment 1606040 [details] python script to test
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:3249