Bug 1225284

+++ This bug was initially created as a clone of Bug #1188145 +++

Description of problem:
=======================

while the IO is in progress on client like creating files/directories, if the feature USS is turned on/off, there's an IO error. 

Version-Release number of selected component (if applicable):
=============================================================
glusterfs 3.7dev built on Jan 29 2015 01:05:44

How reproducible:
=================
100%

Number of volumes :
===================
1

Volume Names :
==============
testvol

Volume on which the particular issue is seen [ if applicable ]
===============================================================
testvol

Type of volumes :
=================
Disperse

Volume options if available :
=============================

[root@dhcp37-120 ~]# gluster volume get testvol all
Option                                  Value                                   
------                                  -----                                   
cluster.lookup-unhashed                 on                                      
cluster.min-free-disk                   10%                                     
cluster.min-free-inodes                 5%                                      
cluster.rebalance-stats                 off                                     
cluster.subvols-per-directory           (null)                                  
cluster.readdir-optimize                off                                     
cluster.rsync-hash-regex                (null)                                  
cluster.extra-hash-regex                (null)                                  
cluster.dht-xattr-name                  trusted.glusterfs.dht                   
cluster.randomize-hash-range-by-gfid    off                                     
cluster.local-volume-name               (null)                                  
cluster.weighted-rebalance              on                                      
cluster.switch-pattern                  (null)                                  
cluster.entry-change-log                on                                      
cluster.read-subvolume                  (null)                                  
cluster.read-subvolume-index            -1                                      
cluster.read-hash-mode                  1                                       
cluster.background-self-heal-count      16                                      
cluster.metadata-self-heal              on                                      
cluster.data-self-heal                  on                                      
cluster.entry-self-heal                 on                                      
cluster.self-heal-daemon                on                                      
cluster.self-heal-window-size           1                                       
cluster.data-change-log                 on                                      
cluster.metadata-change-log             on                                      
cluster.data-self-heal-algorithm        (null)                                  
cluster.eager-lock                      on                                      
cluster.quorum-type                     none                                    
cluster.quorum-count                    (null)                                  
cluster.choose-local                    true                                    
cluster.self-heal-readdir-size          1KB                                     
cluster.post-op-delay-secs              1                                       
cluster.ensure-durability               on                                      
cluster.stripe-block-size               128KB                                   
cluster.stripe-coalesce                 true                                    
diagnostics.latency-measurement         off                                     
diagnostics.dump-fd-stats               off                                     
diagnostics.count-fop-hits              off                                     
diagnostics.brick-log-level             INFO                                    
diagnostics.client-log-level            INFO                                    
diagnostics.brick-sys-log-level         CRITICAL                                
diagnostics.client-sys-log-level        CRITICAL                                
diagnostics.brick-logger                (null)                                  
diagnostics.client-logger               (null)                                  
diagnostics.brick-log-format            (null)                                  
diagnostics.client-log-format           (null)                                  
diagnostics.brick-log-buf-size          5                                       
diagnostics.client-log-buf-size         5                                       
diagnostics.brick-log-flush-timeout     120                                     
diagnostics.client-log-flush-timeout    120                                     
performance.cache-max-file-size         0                                       
performance.cache-min-file-size         0                                       
performance.cache-refresh-timeout       1                                       
performance.cache-priority                                                      
performance.cache-size                  32MB                                    
performance.io-thread-count             16                                      
performance.high-prio-threads           16                                      
performance.normal-prio-threads         16                                      
performance.low-prio-threads            16                                      
performance.least-prio-threads          1                                       
performance.enable-least-priority       on                                      
performance.least-rate-limit            0                                       
performance.cache-size                  128MB                                   
performance.flush-behind                on                                      
performance.nfs.flush-behind            on                                      
performance.write-behind-window-size    1MB                                     
performance.nfs.write-behind-window-size1MB                                     
performance.strict-o-direct             off                                     
performance.nfs.strict-o-direct         off                                     
performance.strict-write-ordering       off                                     
performance.nfs.strict-write-ordering   off                                     
performance.lazy-open                   yes                                     
performance.read-after-open             no                                      
performance.read-ahead-page-count       4                                       
performance.md-cache-timeout            1                                       
features.encryption                     off                                     
encryption.master-key                   (null)                                  
encryption.data-key-size                256                                     
encryption.block-size                   4096                                    
network.frame-timeout                   1800                                    
network.ping-timeout                    42                                      
network.tcp-window-size                 (null)                                  
features.lock-heal                      off                                     
features.grace-timeout                  10                                      
network.remote-dio                      disable                                 
network.tcp-window-size                 (null)                                  
network.inode-lru-limit                 16384                                   
auth.allow                              *                                       
auth.reject                             (null)                                  
transport.keepalive                     (null)                                  
server.allow-insecure                   (null)                                  
server.root-squash                      off                                     
server.anonuid                          65534                                   
server.anongid                          65534                                   
server.statedump-path                   /var/run/gluster                        
server.outstanding-rpc-limit            64                                      
features.lock-heal                      off                                     
features.grace-timeout                  (null)                                  
server.ssl                              (null)                                  
auth.ssl-allow                          *                                       
server.manage-gids                      off                                     
client.send-gids                        on                                      
server.gid-timeout                      2                                       
server.own-thread                       (null)                                  
performance.write-behind                on                                      
performance.read-ahead                  on                                      
performance.readdir-ahead               off                                     
performance.io-cache                    on                                      
performance.quick-read                  on                                      
performance.open-behind                 on                                      
performance.stat-prefetch               on                                      
performance.client-io-threads           off                                     
performance.nfs.write-behind            on                                      
performance.nfs.read-ahead              off                                     
performance.nfs.io-cache                off                                     
performance.nfs.quick-read              off                                     
performance.nfs.stat-prefetch           off                                     
performance.nfs.io-threads              off                                     
performance.force-readdirp              true                                    
features.file-snapshot                  off                                     
features.uss                            on                                      
features.snapshot-directory             .snaps                                  
features.show-snapshot-directory        off                                     
network.compression                     off                                     
network.compression.window-size         -15                                     
network.compression.mem-level           8                                       
network.compression.min-size            0                                       
network.compression.compression-level   -1                                      
network.compression.debug               false                                   
features.limit-usage                    (null)                                  
features.quota-timeout                  0                                       
features.default-soft-limit             80%                                     
features.soft-timeout                   60                                      
features.hard-timeout                   5                                       
features.alert-time                     86400                                   
features.quota-deem-statfs              off                                     
geo-replication.indexing                off                                     
geo-replication.indexing                off                                     
geo-replication.ignore-pid-check        off                                     
geo-replication.ignore-pid-check        off                                     
features.quota                          on                                      
debug.trace                             off                                     
debug.log-history                       no                                      
debug.log-file                          no                                      
debug.exclude-ops                       (null)                                  
debug.include-ops                       (null)                                  
debug.error-gen                         off                                     
debug.error-failure                     (null)                                  
debug.error-number                      (null)                                  
debug.random-failure                    off                                     
debug.error-fops                        (null)                                  
nfs.enable-ino32                        no                                      
nfs.mem-factor                          15                                      
nfs.export-dirs                         on                                      
nfs.export-volumes                      on                                      
nfs.addr-namelookup                     off                                     
nfs.dynamic-volumes                     off                                     
nfs.register-with-portmap               on                                      
nfs.outstanding-rpc-limit               16                                      
nfs.port                                2049                                    
nfs.rpc-auth-unix                       on                                      
nfs.rpc-auth-null                       on                                      
nfs.rpc-auth-allow                      all                                     
nfs.rpc-auth-reject                     none                                    
nfs.ports-insecure                      off                                     
nfs.trusted-sync                        off                                     
nfs.trusted-write                       off                                     
nfs.volume-access                       read-write                              
nfs.export-dir                                                                  
nfs.disable                             false                                   
nfs.nlm                                 on                                      
nfs.acl                                 on                                      
nfs.mount-udp                           off                                     
nfs.mount-rmtab                         /var/lib/glusterd/nfs/rmtab             
nfs.rpc-statd                           /sbin/rpc.statd                         
nfs.server-aux-gids                     off                                     
nfs.drc                                 off                                     
nfs.drc-size                            0x20000                                 
nfs.read-size                           (1 * 1048576ULL)                        
nfs.write-size                          (1 * 1048576ULL)                        
nfs.readdir-size                        (1 * 1048576ULL)                        
features.read-only                      off                                     
features.worm                           off                                     
storage.linux-aio                       off                                     
storage.batch-fsync-mode                reverse-fsync                           
storage.batch-fsync-delay-usec          0                                       
storage.owner-uid                       -1                                      
storage.owner-gid                       -1                                      
storage.node-uuid-pathinfo              off                                     
storage.health-check-interval           30                                      
storage.build-pgfid                     off                                     
storage.bd-aio                          off                                     
cluster.server-quorum-type              off                                     
cluster.server-quorum-ratio             0                                       
changelog.changelog                     off                                     
changelog.changelog-dir                 (null)                                  
changelog.encoding                      ascii                                   
changelog.rollover-time                 15                                      
changelog.fsync-interval                5                                       
changelog.changelog-barrier-timeout     120                                     
features.barrier                        disable                                 
features.barrier-timeout                120                                     
locks.trace                             disable                                 
cluster.disperse-self-heal-daemon       enable                                  
[root@dhcp37-120 ~]# 



Output of gluster volume info :
===============================
[root@dhcp37-120 ~]# gluster volume info
 
Volume Name: testvol
Type: Disperse
Volume ID: ad1a31fb-2e69-4d5d-9ae0-d057879b8fd5
Status: Started
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: dhcp37-120:/rhs/brick1/b1
Brick2: dhcp37-208:/rhs/brick1/b1
Brick3: dhcp37-178:/rhs/brick1/b1
Brick4: dhcp37-183:/rhs/brick1/b1
Brick5: dhcp37-120:/rhs/brick2/b2
Brick6: dhcp37-208:/rhs/brick2/b2
Options Reconfigured:
features.quota: on
features.uss: on
[root@dhcp37-120 ~]# 


Output of gluster volume status :
=================================

[root@dhcp37-120 ~]# gluster volume status
Status of volume: testvol
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick dhcp37-120:/rhs/brick1/b1				49152	Y	16353
Brick dhcp37-208:/rhs/brick1/b1				49156	Y	30577
Brick dhcp37-178:/rhs/brick1/b1				49156	Y	30675
Brick dhcp37-183:/rhs/brick1/b1				49156	Y	30439
Brick dhcp37-120:/rhs/brick2/b2				49153	Y	4957
Brick dhcp37-208:/rhs/brick2/b2				49157	Y	30588
Snapshot Daemon on localhost				49154	Y	16412
NFS Server on localhost					2049	Y	16420
Quota Daemon on localhost				N/A	Y	16374
Snapshot Daemon on dhcp37-183				49164	Y	26945
NFS Server on dhcp37-183				2049	Y	27899
Quota Daemon on dhcp37-183				N/A	Y	30515
Snapshot Daemon on dhcp37-178				49164	Y	8563
NFS Server on dhcp37-178				2049	Y	8571
Quota Daemon on dhcp37-178				N/A	Y	30751
Snapshot Daemon on dhcp37-208				49165	Y	9581
NFS Server on dhcp37-208				2049	Y	9589
Quota Daemon on dhcp37-208				N/A	Y	30678
 
Task Status of Volume testvol
------------------------------------------------------------------------------
There are no active volume tasks


Steps to Reproduce:
===================

1. Create a 1x(4+2) disperse volume
2. start creating files/dirs from the client.
3. On one of the storage node, turn on/off the USS feature with 'gluster volume set testvol uss on/off'

Actual results:
=================

IO error on the client

Expected results:
=================

There should not be any IO error

Additional info:
================
volume statedumps will be attached to this bug.

--- Additional comment from Bhaskarakiran on 2015-03-09 03:34:01 EDT ---

The command that are used to create files/directories is 

for i in `seq 1 1000`; do dd if=/dev/urandom of=testfile.$i bs=1k count=$i ; done

for i in `seq 1 1000`; do mkdir dir.$i ; done

--- Additional comment from Pranith Kumar K on 2015-03-09 05:19:29 EDT ---

Tried re-creating the issue, but the test case worked fine. Seems like this bug is fixed as well with http://review.gluster.com/9717

--- Additional comment from Bhaskarakiran on 2015-05-05 02:06:14 EDT ---

When the USS is turned off, IO completely hangs and when turned back IO resumes. Moving back the bug.

--- Additional comment from Anand Avati on 2015-05-19 05:07:31 EDT ---

REVIEW: http://review.gluster.org/10787 (cluster/ec: Correctly cleanup delayed locks) posted (#2) for review on master by Xavier Hernandez (xhernandez)

--- Additional comment from Anand Avati on 2015-05-19 07:15:53 EDT ---

REVIEW: http://review.gluster.org/10787 (cluster/ec: Correctly cleanup delayed locks) posted (#3) for review on master by Pranith Kumar Karampuri (pkarampu)

--- Additional comment from Anand Avati on 2015-05-20 04:00:04 EDT ---

COMMIT: http://review.gluster.org/10787 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 61cfcf65f0d4ad70fc8a47395c583d4b5bf1efbe
Author: Xavier Hernandez <xhernandez>
Date:   Thu May 14 20:07:10 2015 +0200

    cluster/ec: Correctly cleanup delayed locks
    
    When a delayed lock is pending, a graph switch doesn't correctly
    terminate it. This means that the update of version and size xattrs
    is lost, causing EIO errors.
    
    This patch handles GF_EVENT_PARENT_DOWN event to correctly finish
    pending udpdates before completing the graph switch.
    
    Change-Id: I394f3b8d41df8d83cdd36636aeb62330f30a66d5
    BUG: 1188145
    Signed-off-by: Xavier Hernandez <xhernandez>
    Reviewed-on: http://review.gluster.org/10787
    Tested-by: NetBSD Build System
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>

--- Additional comment from Anand Avati on 2015-05-20 23:43:33 EDT ---

REVIEW: http://review.gluster.org/10868 (cluster/ec: Fix use after free crash) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

--- Additional comment from Anand Avati on 2015-05-21 05:07:56 EDT ---

REVIEW: http://review.gluster.org/10868 (cluster/ec: Fix use after free crash) posted (#2) for review on master by Pranith Kumar Karampuri (pkarampu)

--- Additional comment from Anand Avati on 2015-05-21 05:11:23 EDT ---

REVIEW: http://review.gluster.org/10868 (cluster/ec: Fix use after free crash) posted (#3) for review on master by Xavier Hernandez (xhernandez)

--- Additional comment from Anand Avati on 2015-05-21 06:14:24 EDT ---

REVIEW: http://review.gluster.org/10868 (cluster/ec: Fix use after free crash) posted (#4) for review on master by Pranith Kumar Karampuri (pkarampu)

--- Additional comment from Anand Avati on 2015-05-21 09:08:28 EDT ---

COMMIT: http://review.gluster.org/10868 committed in master by Pranith Kumar Karampuri (pkarampu) 
------
commit 0910bab5e5b957e11f356d525eccccfd36d334f9
Author: Pranith Kumar K <pkarampu>
Date:   Wed May 20 23:56:17 2015 +0530

    cluster/ec: Fix use after free crash
    
    ec_heal creates ec_fop_data but doesn't run ec_manager. ec_fop_data_allocate
    adds this fop to ec->pending_fops, because ec_manager is not run on this heal
    fop it is never removed from ec->pending_fops. When it is accessed after free
    it leads to crash. It is better to not to add HEAL fops to ec->pending_fops
    because we don't want graph switch to hang the mount because of a BIG
    file/directory heal.
    
    BUG: 1188145
    Change-Id: I8abdc92f06e0563192300ca4abca3909efcca9c3
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/10868
    Reviewed-by: Xavier Hernandez <xhernandez>
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Raghavendra Bhat <raghavendra>