Bug 1732774 - Disperse volume : data corruption with ftruncate data in 4+2 config
Summary: Disperse volume : data corruption with ftruncate data in 4+2 config
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: disperse
Version: rhgs-3.4
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: RHGS 3.5.0
Assignee: Ashish Pandey
QA Contact: Nag Pavan Chilakam
URL:
Whiteboard:
Depends On: 1727081 1739424 1739449 1805051 1805054 1805056
Blocks: 1696809 1730914
TreeView+ depends on / blocked
 
Reported: 2019-07-24 10:31 UTC by Ashish Pandey
Modified: 2020-02-20 07:43 UTC (History)
8 users (show)

Fixed In Version: glusterfs-6.0-12
Doc Type: Bug Fix
Doc Text:
If a file on a bad brick was being healed while a write request for that file was being performed, the read that occurs during a write operation could still read the file from the bad brick. This could lead to corruption of data on good bricks. All reads are now done from good bricks only, avoiding this issue.
Clone Of: 1727081
Environment:
Last Closed: 2019-10-30 12:22:31 UTC
Target Upstream Version:


Attachments (Terms of Use)
python script to test (7.27 KB, text/plain)
2019-08-20 10:12 UTC, Nag Pavan Chilakam
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2019:3249 None None None 2019-10-30 12:22:54 UTC

Description Ashish Pandey 2019-07-24 10:31:48 UTC
+++ This bug was initially created as a clone of Bug #1727081 +++

Description of problem:

LTP ftestxx tests reports data corruption at a 4+2 disperse volume.

<<<test_output>>>
ftest05     1  TFAIL  :  ftest05.c:395:         Test[0] bad verify @ 0x3800 for val 2 count 487 xfr 2048 file_max 0xfa000.
ftest05     0  TINFO  :         Test[0]: last_trunc = 0x4d800
ftest05     0  TINFO  :         Stat: size=fa000, ino=120399ba
ftest05     0  TINFO  :         Buf:
ftest05     0  TINFO  :         64*0,
ftest05     0  TINFO  :         2,
ftest05     0  TINFO  :         2,
ftest05     0  TINFO  :         2,
ftest05     0  TINFO  :         2,
ftest05     0  TINFO  :         2,
ftest05     0  TINFO  :         2,
ftest05     0  TINFO  :         2,
ftest05     0  TINFO  :         2,
ftest05     0  TINFO  :         2,
ftest05     0  TINFO  :         2,
ftest05     0  TINFO  :          ... more
ftest05     0  TINFO  :         Bits array:
ftest05     0  TINFO  :         0:
ftest05     0  TINFO  :         0:
ftest05     0  TINFO  :         ddx
ftest05     0  TINFO  :         8:
ftest05     0  TINFO  :         ecx


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Worker Ant on 2019-07-05 00:06:33 UTC ---

REVIEW: https://review.gluster.org/22999 (cluster/ec: inherit right mask from top parent) posted (#1) for review on master by Kinglong Mee

--- Additional comment from Worker Ant on 2019-07-08 13:27:01 UTC ---

REVIEW: https://review.gluster.org/23010 (cluster/ec: inherit healing from lock which has info) posted (#1) for review on master by Kinglong Mee

--- Additional comment from Pranith Kumar K on 2019-07-10 10:30:39 UTC ---

(In reply to Kinglong Mee from comment #0)
> Description of problem:
> 
> LTP ftestxx tests reports data corruption at a 4+2 disperse volume.
> 
> <<<test_output>>>
> ftest05     1  TFAIL  :  ftest05.c:395:         Test[0] bad verify @ 0x3800
> for val 2 count 487 xfr 2048 file_max 0xfa000.
> ftest05     0  TINFO  :         Test[0]: last_trunc = 0x4d800
> ftest05     0  TINFO  :         Stat: size=fa000, ino=120399ba
> ftest05     0  TINFO  :         Buf:
> ftest05     0  TINFO  :         64*0,
> ftest05     0  TINFO  :         2,
> ftest05     0  TINFO  :         2,
> ftest05     0  TINFO  :         2,
> ftest05     0  TINFO  :         2,
> ftest05     0  TINFO  :         2,
> ftest05     0  TINFO  :         2,
> ftest05     0  TINFO  :         2,
> ftest05     0  TINFO  :         2,
> ftest05     0  TINFO  :         2,
> ftest05     0  TINFO  :         2,
> ftest05     0  TINFO  :          ... more
> ftest05     0  TINFO  :         Bits array:
> ftest05     0  TINFO  :         0:
> ftest05     0  TINFO  :         0:
> ftest05     0  TINFO  :         ddx
> ftest05     0  TINFO  :         8:
> ftest05     0  TINFO  :         ecx


When I try to run this test, it is choosing /tmp as the directory where the file is created. How to change it to the mount directory?
root@localhost - /mnt/ec2 
15:11:08 :( ⚡ /opt/ltp/testcases/bin/ftest05 
ftest05     1  TPASS  :  Test passed.


> 
> 
> Version-Release number of selected component (if applicable):
> 
> 
> How reproducible:
> 
> 
> Steps to Reproduce:
> 1.
> 2.
> 3.
> 
> Actual results:
> 
> 
> Expected results:
> 
> 
> Additional info:

--- Additional comment from Kinglong Mee on 2019-07-10 12:47:01 UTC ---

(In reply to Pranith Kumar K from comment #3)
> (In reply to Kinglong Mee from comment #0)
> > Description of problem:
> > 
> > LTP ftestxx tests reports data corruption at a 4+2 disperse volume.
> > 
> > <<<test_output>>>
> > ftest05     1  TFAIL  :  ftest05.c:395:         Test[0] bad verify @ 0x3800
> > for val 2 count 487 xfr 2048 file_max 0xfa000.
> > ftest05     0  TINFO  :         Test[0]: last_trunc = 0x4d800
> > ftest05     0  TINFO  :         Stat: size=fa000, ino=120399ba
> > ftest05     0  TINFO  :         Buf:
> > ftest05     0  TINFO  :         64*0,
> > ftest05     0  TINFO  :         2,
> > ftest05     0  TINFO  :         2,
> > ftest05     0  TINFO  :         2,
> > ftest05     0  TINFO  :         2,
> > ftest05     0  TINFO  :         2,
> > ftest05     0  TINFO  :         2,
> > ftest05     0  TINFO  :         2,
> > ftest05     0  TINFO  :         2,
> > ftest05     0  TINFO  :         2,
> > ftest05     0  TINFO  :         2,
> > ftest05     0  TINFO  :          ... more
> > ftest05     0  TINFO  :         Bits array:
> > ftest05     0  TINFO  :         0:
> > ftest05     0  TINFO  :         0:
> > ftest05     0  TINFO  :         ddx
> > ftest05     0  TINFO  :         8:
> > ftest05     0  TINFO  :         ecx
> 
> 
> When I try to run this test, it is choosing /tmp as the directory where the
> file is created. How to change it to the mount directory?
> root@localhost - /mnt/ec2 
> 15:11:08 :( ⚡ /opt/ltp/testcases/bin/ftest05 
> ftest05     1  TPASS  :  Test passed.

You can run as,
./runltp -p -l /tmp/resut.log -o /tmp/output.log -C /tmp/failed.log -d /mnt/nfs/ -f casefilename-under-runtest

When running the test at nfs client, there is a bash scripts running which reboot one node(the cluster node Ganesha.nfsd is not running on) every 600s.

--- Additional comment from Kinglong Mee on 2019-07-11 10:32:22 UTC ---

valgrind reports some memory leak,

==7925== 300 bytes in 6 blocks are possibly lost in loss record 880 of 1,436
==7925==    at 0x4C29BC3: malloc (vg_replace_malloc.c:299)
==7925==    by 0x71828BF: __gf_default_malloc (mem-pool.h:112)
==7925==    by 0x7183182: __gf_malloc (mem-pool.c:131)
==7925==    by 0x713FB65: gf_strndup (mem-pool.h:189)
==7925==    by 0x713FBD5: gf_strdup (mem-pool.h:206)
==7925==    by 0x7144465: loc_copy (xlator.c:1276)
==7925==    by 0x18EDBF1C: ec_loc_from_loc (ec-helpers.c:760)
==7925==    by 0x18F02FE5: ec_manager_open (ec-inode-read.c:778)
==7925==    by 0x18EE4905: __ec_manager (ec-common.c:3094)
==7925==    by 0x18EE4A0F: ec_manager (ec-common.c:3112)
==7925==    by 0x18F037F3: ec_open (ec-inode-read.c:929)
==7925==    by 0x18ED5E85: ec_gf_open (ec.c:1146)

--- Additional comment from Worker Ant on 2019-07-11 11:05:58 UTC ---

REVIEW: https://review.gluster.org/23029 (cluster/ec: do loc_copy from ctx->loc in fd->lock) posted (#1) for review on master by Kinglong Mee

--- Additional comment from Kinglong Mee on 2019-07-12 00:47:27 UTC ---

ganesha.nfsd crash when healing name,

Core was generated by `/usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N N'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f0d5ae8c5a9 in ec_heal_name (frame=0x7f0d57c6ca28, 
    ec=0x7f0d5b62d280, parent=0x0, name=0x7f0d57537d31 "b", 
    participants=0x7f0d0dfffe30 "\001\001\001") at ec-heal.c:1685
1685	    loc.inode = inode_new(parent->table);
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 dbus-libs-1.10.24-12.el7.x86_64 elfutils-libelf-0.172-2.el7.x86_64 elfutils-libs-0.172-2.el7.x86_64 glibc-2.17-260.el7.x86_64 gssproxy-0.7.0-21.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-34.el7.x86_64 libacl-2.2.51-14.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-59.el7.x86_64 libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-13.el7.x86_64 libgcc-4.8.5-36.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libnfsidmap-0.25-19.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 libuuid-2.23.2-59.el7.x86_64 lz4-1.7.5-2.el7.x86_64 openssl-libs-1.0.2k-16.el7.x86_64 pcre-8.32-17.el7.x86_64 systemd-libs-219-62.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt
#0  0x00007f0d5ae8c5a9 in ec_heal_name (frame=0x7f0d57c6ca28, 
    ec=0x7f0d5b62d280, parent=0x0, name=0x7f0d57537d31 "b", 
    participants=0x7f0d0dfffe30 "\001\001\001") at ec-heal.c:1685
#1  0x00007f0d5ae93cae in ec_heal_do (this=0x7f0d5b65ac00, 
    data=0x7f0d24e3c028, loc=0x7f0d24e3c358, partial=0) at ec-heal.c:3050
#2  0x00007f0d5ae94455 in ec_synctask_heal_wrap (opaque=0x7f0d24e3c028)
    at ec-heal.c:3139
#3  0x00007f0d6d1268c9 in synctask_wrap () at syncop.c:369
#4  0x00007f0d6c6bf010 in ?? () from /lib64/libc.so.6
#5  0x0000000000000000 in ?? ()
(gdb) frame 1
#1  0x00007f0d5ae93cae in ec_heal_do (this=0x7f0d5b65ac00, 
    data=0x7f0d24e3c028, loc=0x7f0d24e3c358, partial=0) at ec-heal.c:3050
3050	        ret = ec_heal_name(frame, ec, loc->parent, (char *)loc->name,
(gdb) p loc
$1 = (loc_t *) 0x7f0d24e3c358
(gdb) p *loc
$2 = {
  path = 0x7f0d57537d00 "/nfsshare/ltp-eZQlnozjnX/ftegVRmbT/ftest05.20436/b", 
  name = 0x7f0d57537d31 "b", inode = 0x7f0d24255b28, parent = 0x0, 
  gfid = "\263\341\223\031\301\245I\260\234\334\017\to%\305^", 
  pargfid = '\000' <repeats 15 times>}

--- Additional comment from Xavi Hernandez on 2019-07-13 14:09:15 UTC ---

Please, don't use the same bug for different issues.

--- Additional comment from Worker Ant on 2019-07-14 13:03:20 UTC ---

REVISION POSTED: https://review.gluster.org/23029 (cluster/ec: do loc_copy from ctx->loc in fd->lock) posted (#2) for review on master by Kinglong Mee

--- Additional comment from Worker Ant on 2019-07-16 17:54:25 UTC ---

REVIEW: https://review.gluster.org/23010 (cluster/ec: inherit healing from lock when it has info) merged (#4) on master by Amar Tumballi

--- Additional comment from Ashish Pandey on 2019-07-17 05:27:44 UTC ---



There are two patches associated with this BZ - 

https://review.gluster.org/#/c/glusterfs/+/22999/ - No merged and under review
https://review.gluster.org/#/c/glusterfs/+/23010/ - Merged 

I would like to keep this bug open till both the patches get merged.

--
Ashish

--- Additional comment from Worker Ant on 2019-07-18 07:28:12 UTC ---

REVIEW: https://review.gluster.org/23069 ((WIP)cluster/ec: Always read from good-mask) posted (#1) for review on master by Pranith Kumar Karampuri

--- Additional comment from Worker Ant on 2019-07-23 06:20:08 UTC ---

REVIEW: https://review.gluster.org/23073 (cluster/ec: fix data corruption) posted (#4) for review on master by Pranith Kumar Karampuri

Comment 8 Nag Pavan Chilakam 2019-08-20 10:12:38 UTC
Created attachment 1606040 [details]
python script to test

Comment 15 errata-xmlrpc 2019-10-30 12:22:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:3249


Note You need to log in before you can comment on or make changes to this bug.