Description of problem: Same as summary. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
All is not well in the compound fops world. I wrote and ran the following simple test on master: <script> 0 #!/bin/bash 1 . $(dirname $0)/../../include.rc 2 . $(dirname $0)/../../volume.rc 3 4 cleanup 5 6 TEST glusterd 7 TEST pidof glusterd 8 TEST $CLI volume create $V0 replica 3 $H0:$B0/${V0}{0,1,2} 9 TEST $CLI volume start $V0 10 TEST $GFS --volfile-id=$V0 --volfile-server=$H0 $M0 11 12 TEST dd if=/dev/urandom of=$M0/file bs=1024 count=10 2>/dev/null 13 md5sum_file=$(md5sum $M0/file | awk '{print $1}') 14 15 TEST $CLI volume set $V0 cluster.use-compound-fops on 16 17 TEST dd if=$M0/file of=$M0/file-copy bs=1024 count=10 2>/dev/null 18 19 EXPECT_WITHIN $UMOUNT_TIMEOUT "Y" force_umount $M0 20 TEST $GFS --volfile-id=$V0 --volfile-server=$H0 $M0 21 22 EXPECT "$md5sum_file" echo `md5sum $M0/file-copy | awk '{print $1}'` 23 24 EXPECT_WITHIN $UMOUNT_TIMEOUT "Y" force_umount $M0 25 TEST $CLI volume stop $V0 26 TEST $CLI volume delete $V0 27 28 cleanup </script> The md5sum comparison at line 22 fails. The script runs fine when I "comment out" line 15, implying the bug is in compound fops. Will post my findings after the issue is RC'd. -Krutika
REVIEW: http://review.gluster.org/15654 (compound fops: Fix file corruption issue) posted (#1) for review on master by Krutika Dhananjay (kdhananj)
REVIEW: http://review.gluster.org/15654 (compound fops: Fix file corruption issue) posted (#2) for review on master by Krutika Dhananjay (kdhananj)
REVIEW: http://review.gluster.org/15654 (compound fops: Fix file corruption issue) posted (#3) for review on master by Krutika Dhananjay (kdhananj)
REVIEW: http://review.gluster.org/15654 (compound fops: Fix file corruption issue) posted (#4) for review on master by Krutika Dhananjay (kdhananj)
REVIEW: http://review.gluster.org/15654 (compound fops: Fix file corruption issue) posted (#5) for review on master by Krutika Dhananjay (kdhananj)
REVIEW: http://review.gluster.org/15654 (compound fops: Fix file corruption issue) posted (#6) for review on master by Krutika Dhananjay (kdhananj)
COMMIT: http://review.gluster.org/15654 committed in master by Atin Mukherjee (amukherj) ------ commit 41dc5ee07ffba6d17459757abf13fae9f174e6b6 Author: Krutika Dhananjay <kdhananj> Date: Mon Oct 17 15:13:28 2016 +0530 compound fops: Fix file corruption issue 1. Address of a local variable @args is copied into state->req in server3_3_compound (). But even after the function has gone out of scope, in server_compound_resume () this pointer is accessed and dereferenced. This patch fixes that. 2. Compound fops, by virtue of NOT having a vector sizer (like the one writev has), ends up having both the header and the data (in case one of its member fops is WRITEV) in the same hdr_iobuf. This buffer was not being preserved through the lifetime of the compound fop, causing it to be overwritten by a parallel write fop, even when the writev associated with the currently executing compound fop is yet to hit the desk, thereby corrupting the file's data. This is fixed by associating the hdr_iobuf with the iobref so its memory remains valid through the lifetime of the fop. 3. Also fixed a use-after-free bug in protocol/client in compound fops cbk, missed by Linux but caught by NetBSD. Finally, big thanks to Pranith Kumar K and Raghavendra Gowdappa for their help in debugging this file corruption issue. Change-Id: I6d5c04f400ecb687c9403a17a12683a96c2bf122 BUG: 1378778 Signed-off-by: Krutika Dhananjay <kdhananj> Reviewed-on: http://review.gluster.org/15654 NetBSD-regression: NetBSD Build System <jenkins.org> Reviewed-by: Raghavendra G <rgowdapp> Smoke: Gluster Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report. glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html [2] https://www.gluster.org/pipermail/gluster-users/