Bug 1362540

Created attachment 1186826 [details]
Compressed core dump

Also, I can't seem to reproduce this when the test filesystem tree is small enough.

I suspect below could have caused the issue - 

In inode_table_destroy(), we first purge all the lru entries but the lru count is not adjusted accordingly. So when inode_table_prune() is called in case if the lru count was larger than lru limit (as can be seen in the core), we shall end up accessing invalid memory. 

(gdb) f 3
#3  0x00007fcad764100e in inode_table_prune (table=table@entry=0x7fcac0004040) at inode.c:1521
1521	                        __inode_retire (entry);
(gdb) p table->lru_size
$4 = 132396
(gdb) p table->lru_limit
$5 = 131072
(gdb) p table->lru
$6 = {next = 0x90, prev = 0xcafecafe}
(gdb) p &&table->lru
A syntax error in expression, near `&&table->lru'.
(gdb) p &table->lru
$7 = (struct list_head *) 0x7fcac00040b8
(gdb) 

I will send a fix for it.

(In reply to Soumya Koduri from comment #3)
> I suspect below could have caused the issue - 
> 
> In inode_table_destroy(), we first purge all the lru entries but the lru
> count is not adjusted accordingly.

This is not true. Can you point out the code where an inode is moved into/out of lru list but lru_size is not modified atomically?

I think we need to explore further to find out why lru_size and lru_list were out of sync, before considering this bug as closed with patch http://review.gluster.org/#/c/15087/

(In reply to Raghavendra G from comment #4)
> (In reply to Soumya Koduri from comment #3)
> > I suspect below could have caused the issue - 
> > 
> > In inode_table_destroy(), we first purge all the lru entries but the lru
> > count is not adjusted accordingly.
> 
> This is not true. Can you point out the code where an inode is moved
> into/out of lru list but lru_size is not modified atomically?

Sorry I missed out that inode_table_destroy is the culprit. We need to fix inode_table_destroy to update lru_size while moving inodes from lru to purge list (just like inode_table_prune).

> 
> I think we need to explore further to find out why lru_size and lru_list
> were out of sync, before considering this bug as closed with patch
> http://review.gluster.org/#/c/15087/

(In reply to Raghavendra G from comment #5)
> (In reply to Raghavendra G from comment #4)
> > (In reply to Soumya Koduri from comment #3)
> > > I suspect below could have caused the issue - 
> > > 
> > > In inode_table_destroy(), we first purge all the lru entries but the lru
> > > count is not adjusted accordingly.
> > 
> > This is not true. Can you point out the code where an inode is moved
> > into/out of lru list but lru_size is not modified atomically?
> 
> Sorry I missed out that inode_table_destroy is the culprit. We need to fix
> inode_table_destroy to update lru_size while moving inodes from lru to purge
> list (just like inode_table_prune).
> 

yes.. and that is exactly what http://review.gluster.org/#/c/15087/ does. Sorry ..am I missing anything?

REVIEW: http://review.gluster.org/15129 (inode: Adjust lru_size while retiring entries in lru list) posted (#2) for review on release-3.8 by Oleksandr Natalenko (oleksandr)

*** Bug 1365748 has been marked as a duplicate of this bug. ***

COMMIT: http://review.gluster.org/15129 committed in release-3.8 by Niels de Vos (ndevos) 
------
commit dae860ab7e1c5a205646393f2cb80a0a06986c30
Author: Soumya Koduri <skoduri>
Date:   Thu Aug 4 16:00:31 2016 +0530

    inode: Adjust lru_size while retiring entries in lru list
    
    As part of inode_table_destroy(), we first retire entries
    in the lru list but the lru_size is not adjusted accordingly.
    This may result in invalid memory reference in inode_table_prune
    if the lru_size > lru_limit.
    
    > Reviewed-on: http://review.gluster.org/15087
    > Smoke: Gluster Build System <jenkins.org>
    > CentOS-regression: Gluster Build System <jenkins.org>
    > NetBSD-regression: NetBSD Build System <jenkins.org>
    > Reviewed-by: Raghavendra G <rgowdapp>
    > Reviewed-by: Prashanth Pai <ppai>
    
    BUG: 1362540
    Change-Id: I29ee3c03b0eaa8a118d06dc0cefba85877daf963
    Signed-off-by: Soumya Koduri <skoduri>
    Signed-off-by: Oleksandr Natalenko <oleksandr>
    Reviewed-on: http://review.gluster.org/15129
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Prashanth Pai <ppai>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Niels de Vos <ndevos>

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.3, please open a new bug report.

glusterfs-3.8.3 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/announce/2016-August/000059.html
[2] https://www.gluster.org/pipermail/gluster-users/