Bug 1318428 - ./tests/basic/tier/tier-file-create.t dumping core fairly often on build machines in Linux
Summary: ./tests/basic/tier/tier-file-create.t dumping core fairly often on build mach...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: RHGS 3.1.3
Assignee: Pranith Kumar K
QA Contact: Nag Pavan Chilakam
URL:
Whiteboard:
Depends On: 1315560 1322520
Blocks: 1311817
TreeView+ depends on / blocked
 
Reported: 2016-03-16 20:45 UTC by Dustin Black
Modified: 2019-11-14 07:36 UTC (History)
12 users (show)

Fixed In Version: glusterfs-3.7.9-2
Doc Type: Bug Fix
Doc Text:
Clone Of: 1315560
Environment:
Last Closed: 2016-06-23 05:03:53 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1240 0 normal SHIPPED_LIVE Red Hat Gluster Storage 3.1 Update 3 2016-06-23 08:51:28 UTC

Description Dustin Black 2016-03-16 20:45:42 UTC
+++ This bug was initially created as a clone of Bug #1315560 +++

Description of problem:

http://www.gluster.org/pipermail/gluster-devel/2016-March/048568.html

https://build.gluster.org/job/rackspace-regression-2GB-triggered/18872/consoleFull
https://build.gluster.org/job/rackspace-regression-2GB-triggered/18793/console


I have set the author to the author of the script to begin with.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Vijay Bellur on 2016-03-07 23:52:23 EST ---

REVIEW: http://review.gluster.org/13632 (tests: Move tier-file-create.t to bad tests) posted (#1) for review on master by Krutika Dhananjay (kdhananj@redhat.com)

--- Additional comment from Vijay Bellur on 2016-03-08 01:45:07 EST ---

REVIEW: http://review.gluster.org/13632 (tests: Move tier-file-create.t to bad tests) posted (#2) for review on master by Krutika Dhananjay (kdhananj@redhat.com)

--- Additional comment from Vijay Bellur on 2016-03-08 06:29:20 EST ---

REVIEW: http://review.gluster.org/13632 (tests: Move tier-file-create.t to bad tests) posted (#3) for review on master by Krutika Dhananjay (kdhananj@redhat.com)

--- Additional comment from Vijay Bellur on 2016-03-08 15:00:44 EST ---

COMMIT: http://review.gluster.org/13632 committed in master by Jeff Darcy (jdarcy@redhat.com) 
------
commit 66d62edd08be5701407e4adcb153a676702ff8b8
Author: Krutika Dhananjay <kdhananj@redhat.com>
Date:   Tue Mar 8 10:21:14 2016 +0530

    tests: Move tier-file-create.t to bad tests
    
    Change-Id: Iaddb244699b0e2647a67a75f257e4c47e0e69e0d
    BUG: 1315560
    Signed-off-by: Krutika Dhananjay <kdhananj@redhat.com>
    Reviewed-on: http://review.gluster.org/13632
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Dan Lambright <dlambrig@redhat.com>
    Reviewed-by: Jeff Darcy <jdarcy@redhat.com>

--- Additional comment from Vijay Bellur on 2016-03-11 05:48:50 EST ---

REVIEW: http://review.gluster.org/13680 (cluster/ec: Do not ref dictionary in lookup) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu@redhat.com)

--- Additional comment from Vijay Bellur on 2016-03-14 07:40:03 EDT ---

COMMIT: http://review.gluster.org/13680 committed in master by Xavier Hernandez (xhernandez@datalab.es) 
------
commit 64cba025b13aad7fb3020a04930cfa22fbfcb859
Author: Pranith Kumar K <pkarampu@redhat.com>
Date:   Tue Mar 8 23:05:08 2016 +0530

    cluster/ec: Do not ref dictionary in lookup
    
    Problem:
    1) dict_for_each loops over the elements without any locks, so the members of
       the dictionary can be ref/unrefed while dict_for_each is executed by another
       thread leading to crashes.
    
    Basically with distributed ec + disctributed replicate as cold, hot tiers. tier
    sends a lookup which fails on ec. (By this time dict already contains ec
    xattrs) After this lookup_everywhere code path is hit in tier which triggers
    lookup on each of distribute's hash lookup but fails which leads to the cold,
    hot dht's lookup_everywhere in two parallel epoll threads where in ec when it
    tries to set trusted.ec.version/dirty/size as keys in the dictionary, the older
    values against the same key get erased. While this erasing is going on if the
    thread that is doing lookup on afr's subvolume accesses these keys either in
    dict_copy_with_ref or client xlator trying to serialize, that can either lead
    to crash or hang based on if the spin/mutex lock is called on invalid memory.
    
    2) EC deletes GF_CONTENT_KEY from the dictionary, this may lead to extra reads
       in case of lookup-everwhere for tiered volumes.
    
    Fix:
    Do dict_copy_with_ref() for the lookup-dictionary.
    This is avoiding the problem and is not actually fixing the 1st problem.
    2nd problem will be fixed.
    
    Change-Id: I5427aa14c48cb7572977d4de9a28c5ffff2b4b95
    BUG: 1315560
    Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
    Reviewed-on: http://review.gluster.org/13680
    Smoke: Gluster Build System <jenkins@build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
    Reviewed-by: Xavier Hernandez <xhernandez@datalab.es>

Comment 8 Nag Pavan Chilakam 2016-05-23 07:35:17 UTC
QA Validation:
As discussed with dev, I ran the file in continous loop of 100 times and didn't hit any core
Hence moving to verified
[root@dhcp35-191 /]# rpm -qa|grep gluster
glusterfs-fuse-3.7.9-5.el7rhgs.x86_64
glusterfs-client-xlators-3.7.9-5.el7rhgs.x86_64
glusterfs-3.7.9-5.el7rhgs.x86_64
glusterfs-server-3.7.9-5.el7rhgs.x86_64
glusterfs-api-3.7.9-5.el7rhgs.x86_64
python-gluster-3.7.9-5.el7rhgs.noarch
glusterfs-libs-3.7.9-5.el7rhgs.x86_64
glusterfs-cli-3.7.9-5.el7rhgs.x86_64
[root@dhcp35-191 /]#

Comment 10 errata-xmlrpc 2016-06-23 05:03:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1240


Note You need to log in before you can comment on or make changes to this bug.