Bug 1159471 - rename operation leads to core dump
Summary: rename operation leads to core dump
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: disperse
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Xavi Hernandez
QA Contact:
URL:
Whiteboard:
: 1161034 (view as bug list)
Depends On: 1161886
Blocks: 1159529 glusterfs-3.6.3
TreeView+ depends on / blocked
 
Reported: 2014-11-01 08:13 UTC by lidi
Modified: 2015-08-07 11:30 UTC (History)
5 users (show)

Fixed In Version: glusterfs-3.6.3beta1
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1161886 (view as bug list)
Environment:
Last Closed: 2015-08-07 11:30:31 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)

Description lidi 2014-11-01 08:13:42 UTC
Description of problem:
I create a disperse volume and mount on /cluster2/test.When I run fstest-20090130-RC on /cluster2/test it crashed.Here is the volume info:

[root@localhost ~]# gluster vol info
 
Volume Name: test
Type: Disperse
Volume ID: 6f2dbbab-9be9-4e6d-9422-d9858ab821be
Status: Started
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: 10.10.21.20:/sdb
Brick2: 10.10.21.21:/sdb
Brick3: 10.10.21.22:/sdb


Steps to Reproduce:
1.create the disperse volume and mount it on /cluster2/test
2.prove -r /root/pjd-fstest-20090130-RC


Additional info:

[root@localhost /]# gdb /usr/sbin/glusterfs core.2289 
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/glusterfs...done.
[New Thread 2289]
[New Thread 2290]
[New Thread 2291]
[New Thread 2292]
[New Thread 2293]
[New Thread 2298]
[New Thread 2297]
Missing separate debuginfo for /usr/lib64/libglusterfs.so.0
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/c3/3c9eb6dbb9054326e16c7a3961797babf12fcf
Missing separate debuginfo for /usr/lib64/libgfrpc.so.0
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/d3/daf2ce43390ff466bf981048f289713510a8ab
Missing separate debuginfo for /usr/lib64/libgfxdr.so.0
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/1d/e5b4dc4bd8e75ce5aadb25eca6d4e1f02454ca
Missing separate debuginfo for /usr/lib64/glusterfs//xlator/mount/fuse.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/82/635b72929a22dc83dbad9869023d6f7587cca8
Missing separate debuginfo for /usr/lib64/glusterfs//rpc-transport/socket.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/52/eecd284b267584f3415df89b38d65f5b2e4234
Missing separate debuginfo for /usr/lib64/glusterfs//xlator/protocol/client.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b3/78a3675ae28e5811de1259970797c8d26cb4b8
Missing separate debuginfo for /usr/lib64/glusterfs//xlator/cluster/disperse.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/41/eae1fe74a65f0142f50de686c7165d5293ce52
Missing separate debuginfo for /usr/lib64/glusterfs//xlator/cluster/distribute.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/d2/e35efd6e50ba68138e1fc92947c8e27417c9eb
Missing separate debuginfo for /usr/lib64/glusterfs//xlator/performance/write-behind.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/93/07f18111d4e3a2f46f4b320d20deeb515a812f
Missing separate debuginfo for /usr/lib64/glusterfs//xlator/performance/read-ahead.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/4c/fe9557a1fabadd0b15bfc3ddf50376ba609d9e
Missing separate debuginfo for /usr/lib64/glusterfs//xlator/performance/io-cache.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/a4/d55f3e304f2ce2305f3e585d5c980a063aea5c
Missing separate debuginfo for /usr/lib64/glusterfs//xlator/performance/quick-read.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b0/cfce764003ea365d2e8d969c5c8e49360a093e
Missing separate debuginfo for /usr/lib64/glusterfs//xlator/performance/open-behind.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/71/2f30ecfb735a6357100f4f415cec3972fdbc90
Missing separate debuginfo for /usr/lib64/glusterfs//xlator/performance/md-cache.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/d3/964f25728f57611200edae6671135d6c4790fd
Missing separate debuginfo for /usr/lib64/glusterfs//xlator/debug/io-stats.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/7c/69ea1368f62d7a9c12e42a63b7e73deadfeb17
Missing separate debuginfo for /usr/lib64/glusterfs//xlator/meta.so
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/f0/cb4449dc1668f45d39d6c9f517c8c4c8c9d1e8
Missing separate debuginfo for 
Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b5/d86fbcf0ccb03331e6c7c73897b96845e0a4eb
Reading symbols from /usr/lib64/libglusterfs.so.0...done.
Loaded symbols for /usr/lib64/libglusterfs.so.0
Reading symbols from /usr/lib64/libgfrpc.so.0...done.
Loaded symbols for /usr/lib64/libgfrpc.so.0
Reading symbols from /usr/lib64/libgfxdr.so.0...done.
Loaded symbols for /usr/lib64/libgfxdr.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /usr/lib64/glusterfs//xlator/mount/fuse.so...done.
Loaded symbols for /usr/lib64/glusterfs//xlator/mount/fuse.so
Reading symbols from /usr/lib64/glusterfs//rpc-transport/socket.so...done.
Loaded symbols for /usr/lib64/glusterfs//rpc-transport/socket.so
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/glusterfs//xlator/protocol/client.so...done.
Loaded symbols for /usr/lib64/glusterfs//xlator/protocol/client.so
Reading symbols from /usr/lib64/glusterfs//xlator/cluster/disperse.so...done.
Loaded symbols for /usr/lib64/glusterfs//xlator/cluster/disperse.so
Reading symbols from /usr/lib64/glusterfs//xlator/cluster/distribute.so...done.
Loaded symbols for /usr/lib64/glusterfs//xlator/cluster/distribute.so
Reading symbols from /usr/lib64/glusterfs//xlator/performance/write-behind.so...done.
Loaded symbols for /usr/lib64/glusterfs//xlator/performance/write-behind.so
Reading symbols from /usr/lib64/glusterfs//xlator/performance/read-ahead.so...done.
Loaded symbols for /usr/lib64/glusterfs//xlator/performance/read-ahead.so
Reading symbols from /usr/lib64/glusterfs//xlator/performance/io-cache.so...done.
Loaded symbols for /usr/lib64/glusterfs//xlator/performance/io-cache.so
Reading symbols from /usr/lib64/glusterfs//xlator/performance/quick-read.so...done.
Loaded symbols for /usr/lib64/glusterfs//xlator/performance/quick-read.so
Reading symbols from /usr/lib64/glusterfs//xlator/performance/open-behind.so...done.
Loaded symbols for /usr/lib64/glusterfs//xlator/performance/open-behind.so
Reading symbols from /usr/lib64/glusterfs//xlator/performance/md-cache.so...done.
Loaded symbols for /usr/lib64/glusterfs//xlator/performance/md-cache.so
Reading symbols from /usr/lib64/glusterfs//xlator/debug/io-stats.so...done.
Loaded symbols for /usr/lib64/glusterfs//xlator/debug/io-stats.so
Reading symbols from /usr/lib64/glusterfs//xlator/meta.so...done.
Loaded symbols for /usr/lib64/glusterfs//xlator/meta.so
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Core was generated by `/usr/sbin/glusterfs --volfile-server=127.0.0.1 --volfile-id=/test /cluster2/tes'.
Program terminated with signal 6, Aborted.
#0  0x00007f27b68f38a5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.107.el6.x86_64 keyutils-libs-1.4-4.el6.x86_64 krb5-libs-1.10.3-10.el6.x86_64 libcom_err-1.41.12-14.el6.x86_64 libgcc-4.4.7-3.el6.x86_64 libselinux-2.0.94-5.3.el6.x86_64 openssl-1.0.0-27.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007f27b68f38a5 in raise () from /lib64/libc.so.6
#1  0x00007f27b68f5085 in abort () from /lib64/libc.so.6
#2  0x00007f27b68eca1e in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007f27b68ecae0 in __assert_fail () from /lib64/libc.so.6
#4  0x00007f27ab4020b4 in ec_unlock_timer_add (link=0xfd2b0c) at ec-common.c:1320
#5  0x00007f27ab402284 in ec_unlock (fop=0xfd2a9c) at ec-common.c:1363
#6  0x00007f27ab4145af in ec_manager_rename (fop=0xfd2a9c, state=8) at ec-dir-write.c:1398
#7  0x00007f27ab40268b in __ec_manager (fop=0xfd2a9c, error=0) at ec-common.c:1472
#8  0x00007f27ab3ffb22 in ec_resume (fop=0xfd2a9c, error=0) at ec-common.c:305
#9  0x00007f27ab427fff in ec_combine (cbk=0x102944c, combine=0x7f27ab413894 <ec_combine_rename>) at ec-combine.c:831
#10 0x00007f27ab413ef3 in ec_rename_cbk (frame=0xfeabc4, cookie=0x2, this=0xfacc70, op_ret=0, op_errno=0, buf=0x7fff4ed32390, preoldparent=0x7fff4ed32320, postoldparent=0x7fff4ed322b0, 
    prenewparent=0x7fff4ed32240, postnewparent=0x7fff4ed321d0, xdata=0x0) at ec-dir-write.c:1281
#11 0x00007f27ab69c715 in client3_3_rename_cbk (req=0x10e2b3c, iov=0x10e2b7c, count=1, myframe=0x1023cfc) at client-rpc-fops.c:2543
#12 0x00007f27b7642b33 in rpc_clnt_handle_reply (clnt=0xfc9d40, pollin=0x10642a0) at rpc-clnt.c:763
#13 0x00007f27b7642f54 in rpc_clnt_notify (trans=0xfca0b0, mydata=0xfc9d70, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x10642a0) at rpc-clnt.c:891
#14 0x00007f27b763f544 in rpc_transport_notify (this=0xfca0b0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x10642a0) at rpc-transport.c:516
#15 0x00007f27aced1f90 in socket_event_poll_in (this=0xfca0b0) at socket.c:2225
#16 0x00007f27aced244b in socket_event_handler (fd=12, idx=3, data=0xfca0b0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2338
#17 0x00007f27b78e1dfb in event_dispatch_epoll_handler (event_pool=0xf81510, events=0xfa1520, i=0) at event-epoll.c:384
#18 0x00007f27b78e1ff5 in event_dispatch_epoll (event_pool=0xf81510) at event-epoll.c:445
#19 0x00007f27b78af21b in event_dispatch (event_pool=0xf81510) at event.c:113
#20 0x0000000000409790 in main (argc=4, argv=0x7fff4ed33be8) at glusterfsd.c:2043
(gdb)

Comment 1 Xavi Hernandez 2014-11-11 14:13:36 UTC
*** Bug 1161034 has been marked as a duplicate of this bug. ***

Comment 2 Anand Avati 2015-01-29 09:10:06 UTC
REVIEW: http://review.gluster.org/9501 (ec: Fix posix compliance failures) posted (#1) for review on release-3.6 by Xavier Hernandez (xhernandez)

Comment 3 Anand Avati 2015-02-11 09:38:08 UTC
COMMIT: http://review.gluster.org/9501 committed in release-3.6 by Raghavendra Bhat (raghavendra) 
------
commit 22035d5e37db748cbdee2596d99006cea6b5282e
Author: Xavier Hernandez <xhernandez>
Date:   Sat Nov 8 21:46:41 2014 +0100

    ec: Fix posix compliance failures
    
    This patch solves some problems that caused dispersed volumes to not
    pass posix smoke tests:
    
    * Problems in open/create with O_WRONLY
        Opening files with -w- permissions using O_WRONLY returned an EACCES
        error because internally O_WRONLY was replaced with O_RDWR.
    
    * Problems with entrylk on renames.
        When source and destination were the same, ec tried to acquire
        the same entrylk twice, causing a deadlock.
    
    * Overwrite of a variable when reordering locks.
        On a rename, if the second lock needed to be placed at the beggining
        of the list, the 'lock' variable was overwritten and later its timer
        was cancelled, cancelling the incorrect one.
    
    * Handle O_TRUNC in open.
        When O_TRUNC was received in an open call, it was blindly propagated
        to child subvolumes. This caused a discrepancy between real file
        size and the size stored into trusted.ec.size xattr. This has been
        solved by removing O_TRUNC from open and later calling ftruncate.
    
    This is a backport of http://review.gluster.org/9420
    
    Change-Id: I20c3d6e1c11be314be86879be54b728e01013798
    BUG: 1159471
    Signed-off-by: Xavier Hernandez <xhernandez>
    Reviewed-on: http://review.gluster.org/9501
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Dan Lambright <dlambrig>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Reviewed-by: Raghavendra Bhat <raghavendra>


Note You need to log in before you can comment on or make changes to this bug.