Bug 1005417 - Catalyst workload causes FUSE related kernel stack traces after only 30K files PUT via swift
Catalyst workload causes FUSE related kernel stack traces after only 30K file...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: glusterfs (Show other bugs)
2.1
x86_64 Linux
urgent Severity urgent
: ---
: ---
Assigned To: Anand Avati
Sudhir D
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-06 16:47 EDT by Peter Portante
Modified: 2015-09-01 19:06 EDT (History)
8 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.33rhs-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-09-23 18:36:08 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/var/log/messages contents containing all stack traces (1.37 MB, text/x-log)
2013-09-06 16:47 EDT, Peter Portante
no flags Details

  None (edit)
Description Peter Portante 2013-09-06 16:47:11 EDT
Created attachment 794976 [details]
/var/log/messages contents containing all stack traces

We have an 8 node (4x2) config, 10ge front and back-end networks, Big Bend -30rhs release installed setup:

gluster-swift.noarch                    1.8.0-6.11.el6rhs
gluster-swift-account.noarch            1.8.0-6.11.el6rhs
gluster-swift-container.noarch          1.8.0-6.11.el6rhs
gluster-swift-object.noarch             1.8.0-6.11.el6rhs
gluster-swift-plugin.noarch             1.8.0-6.el6rhs
gluster-swift-proxy.noarch              1.8.0-6.11.el6rhs
glusterfs.x86_64                        3.4.0.24rhs-1.el6rhs
glusterfs-api.x86_64                    3.4.0.24rhs-1.el6rhs
glusterfs-fuse.x86_64                   3.4.0.24rhs-1.el6rhs
glusterfs-geo-replication.x86_64        3.4.0.24rhs-1.el6rhs
glusterfs-libs.x86_64                   3.4.0.24rhs-1.el6rhs
glusterfs-rdma.x86_64                   3.4.0.24rhs-1.el6rhs
glusterfs-server.x86_64                 3.4.0.24rhs-1.el6rhs

After PUTing 30,000 files using the Catalyst workload, the test hangs do to a set of object servers processes in D states:

[root@gprfs010 ~]# ps auxww | grep swift
root 3738  0.0  0.0 1745356 19700 ? D Sep05 0:01
root 3741  0.0  0.0 1745292 19648 ? D Sep05 0:01
root 3743  0.0  0.0 1745352 19632 ? D Sep05 0:01
root 3745  0.0  0.0 1745356 19656 ? D Sep05 0:01
root 3746  0.0  0.0 1745344 19644 ? D Sep05 0:01
root 3748  0.0  0.0 1745376 19672 ? D Sep05 0:01
root 3749  0.0  0.0 1745340 19644 ? D Sep05 0:01
root 3751  0.0  0.0 1745360 19648 ? D Sep05 0:01
root 3753  0.0  0.0 1745344 19640 ? D Sep05 0:01
root 3754  0.0  0.0 1745352 19672 ? D Sep05 0:01
root 3756  0.0  0.0 1745348 19636 ? D Sep05 0:01
root 3757  0.0  0.0 1745392 19696 ? D Sep05 0:01

Here is an example kernel stack trace seen, the full /var/log/messages output is attached:

Sep  5 17:54:33 gprfs010 kernel: INFO: task swift-object-se:3738 blocked for more than 120 seconds.
Sep  5 17:54:33 gprfs010 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep  5 17:54:33 gprfs010 kernel: swift-object- D 0000000000000006     0  3738   3716 0x00000080
Sep  5 17:54:33 gprfs010 kernel: ffff880625f47ba8 0000000000000086 ffff880625f47b28 ffffffff8143937e
Sep  5 17:54:33 gprfs010 kernel: ffff880625f47b28 ffff880c11836230 ffff880c11836180 0000000000000000
Sep  5 17:54:33 gprfs010 kernel: ffff880614ce8638 ffff880625f47fd8 000000000000fb88 ffff880614ce8638
Sep  5 17:54:33 gprfs010 kernel: Call Trace:
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff8143937e>] ? release_sock+0xce/0xe0
Sep  5 17:54:33 gprfs010 kernel: [<ffffffffa0577b68>] ? fuse_dentry_revalidate+0x38/0x2a0 [fuse]
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff8150f78e>] __mutex_lock_slowpath+0x13e/0x180
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff8119af27>] ? __d_lookup+0xa7/0x150
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff8150f62b>] mutex_lock+0x2b/0x50
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff811907ab>] do_lookup+0x11b/0x230
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff81190acd>] __link_path_walk+0x20d/0x1030
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff81191b7a>] path_walk+0x6a/0xe0
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff81191d4b>] do_path_lookup+0x5b/0xa0
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff811929d7>] user_path_at+0x57/0xa0
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff81186b64>] ? cp_new_stat+0xe4/0x100
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff81186d8c>] vfs_fstatat+0x3c/0x80
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff81186efb>] vfs_stat+0x1b/0x20
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff81186f24>] sys_newstat+0x24/0x50
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff810dc937>] ? audit_syscall_entry+0x1d7/0x200
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff810dc685>] ? __audit_syscall_exit+0x265/0x290
Sep  5 17:54:33 gprfs010 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Comment 2 Anand Avati 2013-09-06 19:28:15 EDT
The root cause of the hang is the occurrence of a gfid mismatch on directory /gprfc026.pmcDef.pw12.omc1.ow12.dcs2MB.ncs2MB.daw02.cw02.d8192.mtu9k.untuned.clth64.tpf/run0/insightdemo12/docs/ji2/docs/20091009/job14077. The manifestation of this was a leaked lock resulting in deadlock of further operations (leading to D state).
Comment 3 Amar Tumballi 2013-09-09 13:46:02 EDT
https://code.engineering.redhat.com/gerrit/12607
Comment 4 Amar Tumballi 2013-09-10 03:02:26 EDT
Below mail is good enough to mark the bug as VERIFIED. Can someone who has QE access, give qe-ack?

-----

On 09/09/2013 11:47 PM, Peter Portante wrote:> 
----- Original Message -----
>> From: "Anand Avati" <aavati@redhat.com>
>> Subject: Re: Status of 1005417 - Catalyst workload causes FUSE related kernel stack traces after only 30K files PUT via swift
>>
>> On 9/9/13 10:42 AM, Scott Haines wrote:
>>> Hey Avati,
>>>
>>> Just checking in on https://bugzilla.redhat.com/show_bug.cgi?id=1005417.
>>>
>>> ~ Scott
>>
>> I have submitted the fix (patch) for review upstream at
>> http://review.gluster.org/5849. Amar has provided RPMs to Peter with the
>> backport of that patch, and so far Peter has not reported any failures
>> with his tests (he has ran some over the weekend)
>>
>> Peter, any updates from your tests?
> 
> Just completed 6 * 4.78 Million file PUTs (final 4.78 set of GETs just about
> to run) with no errors what-so-ever.
> 
> This is really solid.
> 
> The large number numbers from ssbench look reasonable. The small file numbers
> look okay, but have some anomalies I think we can live with.
> 
> I will be presenting these results at 4 PM EDT today.
> 
> Thanks, -peter
> 
----
Comment 5 Scott Haines 2013-09-23 18:36:08 EDT
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. 

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1262.html

Note You need to log in before you can comment on or make changes to this bug.