Bug 1192378 - Disperse volume: client crashed while running renames with epoll enabled
Summary: Disperse volume: client crashed while running renames with epoll enabled
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: disperse
Version: mainline
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Ashish Pandey
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks: qe_tracker_everglades 1188242 1219358 1224115 1224118 1233632
TreeView+ depends on / blocked
 
Reported: 2015-02-13 09:30 UTC by Bhaskarakiran
Modified: 2016-11-23 23:11 UTC (History)
7 users (show)

Fixed In Version: glusterfs-3.7.0beta1
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1224118 (view as bug list)
Environment:
Last Closed: 2015-05-14 17:26:29 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)
Client mount log (7.22 MB, application/zip)
2015-02-13 09:30 UTC, Bhaskarakiran
no flags Details
Client corefile (16.68 MB, application/x-bzip)
2015-02-13 09:40 UTC, Bhaskarakiran
no flags Details

Description Bhaskarakiran 2015-02-13 09:30:43 UTC
Created attachment 991272 [details]
Client mount log

Description of problem:
=======================

Enabled epoll with server.event-threads and client.event-threads set to and then started renames of files 1000 at a time on the client. After some time the client crashed. I am not sure if this has to do anything with epoll. Seen the crash after enabling that.

Version-Release number of selected component (if applicable):
=============================================================
glusterfs 3.7dev built on Feb  8 2015 01:04:29

Volume options:
==============

[root@ninja ~]# gluster volume get testvol all
Option                                  Value                                   
------                                  -----                                   
cluster.lookup-unhashed                 on                                      
cluster.min-free-disk                   10%                                     
cluster.min-free-inodes                 5%                                      
cluster.rebalance-stats                 off                                     
cluster.subvols-per-directory           (null)                                  
cluster.readdir-optimize                off                                     
cluster.rsync-hash-regex                (null)                                  
cluster.extra-hash-regex                (null)                                  
cluster.dht-xattr-name                  trusted.glusterfs.dht                   
cluster.randomize-hash-range-by-gfid    off                                     
cluster.local-volume-name               (null)                                  
cluster.weighted-rebalance              on                                      
cluster.switch-pattern                  (null)                                  
cluster.entry-change-log                on                                      
cluster.read-subvolume                  (null)                                  
cluster.read-subvolume-index            -1                                      
cluster.read-hash-mode                  1                                       
cluster.background-self-heal-count      16                                      
cluster.metadata-self-heal              on                                      
cluster.data-self-heal                  on                                      
cluster.entry-self-heal                 on                                      
cluster.self-heal-daemon                on                                      
cluster.heal-timeout                    600                                     
cluster.self-heal-window-size           1                                       
cluster.data-change-log                 on                                      
cluster.metadata-change-log             on                                      
cluster.data-self-heal-algorithm        (null)                                  
cluster.eager-lock                      on                                      
cluster.quorum-type                     none                                    
cluster.quorum-count                    (null)                                  
cluster.choose-local                    true                                    
cluster.self-heal-readdir-size          1KB                                     
cluster.post-op-delay-secs              1                                       
cluster.ensure-durability               on                                      
cluster.stripe-block-size               128KB                                   
cluster.stripe-coalesce                 true                                    
diagnostics.latency-measurement         off                                     
diagnostics.dump-fd-stats               off                                     
diagnostics.count-fop-hits              off                                     
diagnostics.brick-log-level             INFO                                    
diagnostics.client-log-level            INFO                                    
diagnostics.brick-sys-log-level         CRITICAL                                
diagnostics.client-sys-log-level        CRITICAL                                
diagnostics.brick-logger                (null)                                  
diagnostics.client-logger               (null)                                  
diagnostics.brick-log-format            (null)                                  
diagnostics.client-log-format           (null)                                  
diagnostics.brick-log-buf-size          5                                       
diagnostics.client-log-buf-size         5                                       
diagnostics.brick-log-flush-timeout     120                                     
diagnostics.client-log-flush-timeout    120                                     
performance.cache-max-file-size         0                                       
performance.cache-min-file-size         0                                       
performance.cache-refresh-timeout       1                                       
performance.cache-priority                                                      
performance.cache-size                  32MB                                    
performance.io-thread-count             16                                      
performance.high-prio-threads           16                                      
performance.normal-prio-threads         16                                      
performance.low-prio-threads            16                                      
performance.least-prio-threads          1                                       
performance.enable-least-priority       on                                      
performance.least-rate-limit            0                                       
performance.cache-size                  128MB                                   
performance.flush-behind                on                                      
performance.nfs.flush-behind            on                                      
performance.write-behind-window-size    1MB                                     
performance.nfs.write-behind-window-size1MB                                     
performance.strict-o-direct             off                                     
performance.nfs.strict-o-direct         off                                     
performance.strict-write-ordering       off                                     
performance.nfs.strict-write-ordering   off                                     
performance.lazy-open                   yes                                     
performance.read-after-open             no                                      
performance.read-ahead-page-count       4                                       
performance.md-cache-timeout            1                                       
features.encryption                     off                                     
encryption.master-key                   (null)                                  
encryption.data-key-size                256                                     
encryption.block-size                   4096                                    
network.frame-timeout                   1800                                    
network.ping-timeout                    42                                      
network.tcp-window-size                 (null)                                  
features.lock-heal                      off                                     
features.grace-timeout                  10                                      
network.remote-dio                      disable                                 
client.event-threads                    4                                       
network.tcp-window-size                 (null)                                  
network.inode-lru-limit                 16384                                   
auth.allow                              *                                       
auth.reject                             (null)                                  
transport.keepalive                     (null)                                  
server.allow-insecure                   (null)                                  
server.root-squash                      off                                     
server.anonuid                          65534                                   
server.anongid                          65534                                   
server.statedump-path                   /var/run/gluster                        
server.outstanding-rpc-limit            64                                      
features.lock-heal                      off                                     
features.grace-timeout                  (null)                                  
server.ssl                              (null)                                  
auth.ssl-allow                          *                                       
server.manage-gids                      off                                     
client.send-gids                        on                                      
server.gid-timeout                      2                                       
server.own-thread                       (null)                                  
server.event-threads                    4                                       
performance.write-behind                on                                      
performance.read-ahead                  on                                      
performance.readdir-ahead               off                                     
performance.io-cache                    on                                      
performance.quick-read                  on                                      
performance.open-behind                 on                                      
performance.stat-prefetch               on                                      
performance.client-io-threads           off                                     
performance.nfs.write-behind            on                                      
performance.nfs.read-ahead              off                                     
performance.nfs.io-cache                off                                     
performance.nfs.quick-read              off                                     
performance.nfs.stat-prefetch           off                                     
performance.nfs.io-threads              off                                     
performance.force-readdirp              true                                    
features.file-snapshot                  off                                     
features.uss                            off                                     
features.snapshot-directory             .snaps                                  
features.show-snapshot-directory        off                                     
network.compression                     off                                     
network.compression.window-size         -15                                     
network.compression.mem-level           8                                       
network.compression.min-size            0                                       
network.compression.compression-level   -1                                      
network.compression.debug               false                                   
features.limit-usage                    (null)                                  
features.quota-timeout                  0                                       
features.default-soft-limit             80%                                     
features.soft-timeout                   60                                      
features.hard-timeout                   5                                       
features.alert-time                     86400                                   
features.quota-deem-statfs              off                                     
geo-replication.indexing                off                                     
geo-replication.indexing                off                                     
geo-replication.ignore-pid-check        off                                     
geo-replication.ignore-pid-check        off                                     
features.quota                          on                                      
debug.trace                             off                                     
debug.log-history                       no                                      
debug.log-file                          no                                      
debug.exclude-ops                       (null)                                  
debug.include-ops                       (null)                                  
debug.error-gen                         off                                     
debug.error-failure                     (null)                                  
debug.error-number                      (null)                                  
debug.random-failure                    off                                     
debug.error-fops                        (null)                                  
nfs.enable-ino32                        no                                      
nfs.mem-factor                          15                                      
nfs.export-dirs                         on                                      
nfs.export-volumes                      on                                      
nfs.addr-namelookup                     off                                     
nfs.dynamic-volumes                     off                                     
nfs.register-with-portmap               on                                      
nfs.outstanding-rpc-limit               16                                      
nfs.port                                2049                                    
nfs.rpc-auth-unix                       on                                      
nfs.rpc-auth-null                       on                                      
nfs.rpc-auth-allow                      all                                     
nfs.rpc-auth-reject                     none                                    
nfs.ports-insecure                      off                                     
nfs.trusted-sync                        off                                     
nfs.trusted-write                       off                                     
nfs.volume-access                       read-write                              
nfs.export-dir                                                                  
nfs.disable                             false                                   
nfs.nlm                                 on                                      
nfs.acl                                 on                                      
nfs.mount-udp                           off                                     
nfs.mount-rmtab                         /var/lib/glusterd/nfs/rmtab             
nfs.rpc-statd                           /sbin/rpc.statd                         
nfs.server-aux-gids                     off                                     
nfs.drc                                 off                                     
nfs.drc-size                            0x20000                                 
nfs.read-size                           (1 * 1048576ULL)                        
nfs.write-size                          (1 * 1048576ULL)                        
nfs.readdir-size                        (1 * 1048576ULL)                        
features.read-only                      off                                     
features.worm                           off                                     
storage.linux-aio                       off                                     
storage.batch-fsync-mode                reverse-fsync                           
storage.batch-fsync-delay-usec          0                                       
storage.owner-uid                       -1                                      
storage.owner-gid                       -1                                      
storage.node-uuid-pathinfo              off                                     
storage.health-check-interval           30                                      
storage.build-pgfid                     off                                     
storage.bd-aio                          off                                     
cluster.server-quorum-type              off                                     
cluster.server-quorum-ratio             0                                       
changelog.changelog                     off                                     
changelog.changelog-dir                 (null)                                  
changelog.encoding                      ascii                                   
changelog.rollover-time                 15                                      
changelog.fsync-interval                5                                       
changelog.changelog-barrier-timeout     120                                     
features.barrier                        disable                                 
features.barrier-timeout                120                                     
locks.trace                             disable                                 
cluster.disperse-self-heal-daemon       enable                                  

Gluster volume status:
======================

[root@ninja ~]# gluster volume status
Status of volume: testvol
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick ninja:/rhs/brick1/b1				49152	Y	29212
Brick vertigo:/rhs/brick1/b1				49152	Y	13852
Brick ninja:/rhs/brick2/b2				49153	Y	13041
Brick vertigo:/rhs/brick2/b2				49153	Y	13864
Brick ninja:/rhs/brick3/b3				49154	Y	13053
Brick vertigo:/rhs/brick3/b3				49158	Y	13876
NFS Server on localhost					2049	Y	29220
Quota Daemon on localhost				N/A	Y	29237
NFS Server on vertigo					2049	Y	13891
Quota Daemon on vertigo					N/A	Y	13907
 
Task Status of Volume testvol
------------------------------------------------------------------------------
There are no active volume tasks

Gluster volume info:
====================
 
[root@ninja ~]# gluster volume info
 
Volume Name: testvol
Type: Disperse
Volume ID: 21ed8908-3458-4834-b93d-161b694c3e37
Status: Started
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: ninja:/rhs/brick1/b1
Brick2: vertigo:/rhs/brick1/b1
Brick3: ninja:/rhs/brick2/b2
Brick4: vertigo:/rhs/brick2/b2
Brick5: ninja:/rhs/brick3/b3
Brick6: vertigo:/rhs/brick3/b3
Options Reconfigured:
client.event-threads: 4
server.event-threads: 4
features.barrier: disable
features.quota: on
cluster.disperse-self-heal-daemon: enable
[root@ninja ~]# 


How reproducible:
=================
Tried only once

Steps to Reproduce:
1. Fuse mount 1x(4+2) disperse volume
2. Create huge number of files/directories and rename them in 1000's in parallel
3. 

Actual results:
===============
Client crashed

Expected results:
=================
No crash or other errors should be seen


Additional info:
Attaching the client mount log and corefile.

Comment 1 Bhaskarakiran 2015-02-13 09:40:15 UTC
Created attachment 991275 [details]
Client corefile

Comment 2 Bhaskarakiran 2015-02-13 09:41:08 UTC
Attached the client corefile.

Comment 5 Anand Avati 2015-03-25 13:23:34 UTC
REVIEW: http://review.gluster.org/9995 (cluster/ec: Use fd when appropriate for updating size/version) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu@redhat.com)

Comment 6 Anand Avati 2015-03-27 12:19:21 UTC
COMMIT: http://review.gluster.org/9995 committed in master by Vijay Bellur (vbellur@redhat.com) 
------
commit 1f655aed1e439935c24da86fa70f19fd9151e7e8
Author: Pranith Kumar K <pkarampu@redhat.com>
Date:   Wed Mar 25 15:32:42 2015 +0530

    cluster/ec: Use fd when appropriate for updating size/version
    
    Change-Id: I5d3aca101c8cdda406d31d06c40404fa6a2b7170
    BUG: 1192378
    Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
    Reviewed-on: http://review.gluster.org/9995
    Tested-by: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
    Reviewed-by: Dan Lambright <dlambrig@redhat.com>

Comment 7 Bhaskarakiran 2015-05-06 09:00:16 UTC
The client has crashed again with the same backtrace. Reopening the bug. This is on the latest 3.7 nightly build.

[root@rhs-client32 core]# gluster --version
glusterfs 3.7.0beta1 built on May  1 2015 01:45:36
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@rhs-client32 core]# 


(gdb) bt
#0  dht_writev_cbk (frame=0x7f653ab709e8, cookie=0x7f653ab92164, this=0x7f652c00ed80, op_ret=-1, op_errno=2, prebuf=0x0, postbuf=0x0, xdata=0x0) at dht-inode-write.c:59
#1  0x0000003a37e2fd5c in default_writev_cbk (frame=0x7f653ab92164, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=2, prebuf=<value optimized out>, 
    postbuf=0x0, xdata=0x0) at defaults.c:1019
#2  0x00007f653061d26d in ec_manager_writev (fop=0x7f652a423724, state=<value optimized out>) at ec-inode-write.c:1632
#3  0x00007f6530604b74 in __ec_manager (fop=0x7f652a423724, error=<value optimized out>) at ec-common.c:1642
#4  0x00007f6530604981 in ec_resume (fop=0x7f652a423724, error=0) at ec-common.c:313
#5  0x00007f6530620286 in ec_combine (newcbk=0x7f6529dc3e94, combine=<value optimized out>) at ec-combine.c:936
#6  0x00007f653061ec8e in ec_inode_write_cbk (frame=<value optimized out>, this=0x7f652c00d500, cookie=<value optimized out>, op_ret=-1, op_errno=<value optimized out>, 
    prestat=0x7f652af83b10, poststat=0x7f652af83aa0, xdata=0x0) at ec-inode-write.c:60
#7  0x00007f653087b94c in client3_3_writev_cbk (req=<value optimized out>, iov=<value optimized out>, count=<value optimized out>, myframe=0x7f653ab9506c) at client-rpc-fops.c:856
#8  0x0000003a3820ec45 in rpc_clnt_handle_reply (clnt=0x7f652c1171f0, pollin=0x7f65241c4740) at rpc-clnt.c:766
#9  0x0000003a382100e2 in rpc_clnt_notify (trans=<value optimized out>, mydata=0x7f652c117220, event=<value optimized out>, data=<value optimized out>) at rpc-clnt.c:894
#10 0x0000003a3820b7b8 in rpc_transport_notify (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:543
#11 0x00007f65318b7bcd in socket_event_poll_in (this=0x7f652c126e60) at socket.c:2290
#12 0x00007f65318b96fd in socket_event_handler (fd=<value optimized out>, idx=<value optimized out>, data=0x7f652c126e60, poll_in=1, poll_out=0, poll_err=0) at socket.c:2403
#13 0x0000003a37e7d4b0 in event_dispatch_epoll_handler (data=0x7f652c027700) at event-epoll.c:572
#14 event_dispatch_epoll_worker (data=0x7f652c027700) at event-epoll.c:674
#15 0x0000003d144079d1 in start_thread (arg=0x7f652af84700) at pthread_create.c:301
#16 0x0000003d13ce88fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
(gdb)

Comment 8 Niels de Vos 2015-05-14 17:26:29 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 9 Niels de Vos 2015-05-14 17:28:23 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 10 Niels de Vos 2015-05-14 17:35:16 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.0, please open a new bug report.

glusterfs-3.7.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10939
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user


Note You need to log in before you can comment on or make changes to this bug.