Bug 839768 - firefox-10.0.4-1.el5_8-x86_64 hang when rendering pages on glusterfs client
Summary: firefox-10.0.4-1.el5_8-x86_64 hang when rendering pages on glusterfs client
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: GlusterFS
Classification: Community
Component: core
Version: 3.2.6
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
Assignee: Amar Tumballi
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-07-12 18:58 UTC by Harshavardhana
Modified: 2015-03-23 01:04 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-09-07 18:10:54 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Embargoed:


Attachments (Terms of Use)
strace firefox (1.78 MB, application/x-bzip)
2012-07-12 18:59 UTC, Harshavardhana
no flags Details
Gluster dump during firefox hang (171.98 KB, application/x-bzip)
2012-07-12 18:59 UTC, Harshavardhana
no flags Details
Sosreport (13.39 MB, application/x-bzip)
2012-07-12 19:00 UTC, Harshavardhana
no flags Details
GlusterFS config melnode1 (2.20 KB, application/x-bzip)
2012-07-13 03:14 UTC, Harshavardhana
no flags Details
GlusterFS config melnode2 (2.21 KB, application/x-bzip)
2012-07-13 03:15 UTC, Harshavardhana
no flags Details
Gluster Client/Server logs melnode1 (11.64 MB, application/x-bzip)
2012-07-13 03:16 UTC, Harshavardhana
no flags Details
Gluster Client/Server logs melnode2 (2.06 MB, application/x-bzip)
2012-07-13 03:17 UTC, Harshavardhana
no flags Details
Gluster statedump 3.3.0 (77.99 KB, application/octet-stream)
2012-08-02 18:14 UTC, Harshavardhana
no flags Details
Gluster statedump 3.3.0 (17.17 KB, application/x-gzip)
2012-08-02 18:15 UTC, Harshavardhana
no flags Details
Sosreport node1 (689.20 KB, application/octet-stream)
2012-08-03 04:56 UTC, Harshavardhana
no flags Details
Sosreport node2 (700.14 KB, application/octet-stream)
2012-08-03 04:56 UTC, Harshavardhana
no flags Details

Description Harshavardhana 2012-07-12 18:58:30 UTC
Description of problem:

Upgrade to firefox-10.0.4-1.el5_8-x86_64 from firefox-3.0.6-1.el5.x86_64.rpm.

is found that firefox 10.0.4 hangs when loading most web pages on glusterfs.

Please find attached the output of: strace -s 1024 -v /usr/bin/firefox

The hang was induced by browsing to google.com

The process was sent a SIGINT to terminate.

The server is headless, the process was using a remote X11 server for DISPLAY.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
GlusterFS version 3.2.6, RHEL5.8 Client x86_64 - install firefox-10.0.4-1 run it over GlusterFS mount, open a webpage and just hangs. 
  
Actual results:
Firefox hang

Expected results:
Firefox renders web page successfully.

Comment 1 Harshavardhana 2012-07-12 18:59:15 UTC
Created attachment 597878 [details]
strace firefox

Comment 2 Harshavardhana 2012-07-12 18:59:46 UTC
Created attachment 597879 [details]
Gluster dump during firefox hang

Comment 3 Harshavardhana 2012-07-12 19:00:26 UTC
Created attachment 597880 [details]
Sosreport

Comment 4 Harshavardhana 2012-07-12 19:15:22 UTC
Waiting on 


* tar czf gluster_configuration.tar.gz /etc/glusterd from servers

* tar czf gluster_logs_servers.tar.gz /var/log/glusterfs from servers

* tar czf gluster_logs_client.tar.gz /var/log/glusterfs from client.

Comment 5 Harshavardhana 2012-07-13 03:14:50 UTC
Created attachment 597944 [details]
GlusterFS config melnode1

The two servers are also the clients, there are no other clients, so the log archives contain both the client and server process output.

Comment 6 Harshavardhana 2012-07-13 03:15:19 UTC
Created attachment 597945 [details]
GlusterFS config melnode2

Comment 7 Harshavardhana 2012-07-13 03:16:07 UTC
Created attachment 597947 [details]
Gluster Client/Server logs melnode1

Comment 8 Harshavardhana 2012-07-13 03:17:00 UTC
Created attachment 597948 [details]
Gluster Client/Server logs melnode2

Comment 9 Amar Tumballi 2012-07-13 05:25:08 UTC
Harsha,

I will be going through the logs now, meantime, wanted to understand what would be the behavior on 3.3.0 release? Is it possible to test that here?

-Amar

Comment 10 Amar Tumballi 2012-07-19 10:35:50 UTC
[global.callpool.stack.1.frame.2]
global.callpool.stack.1.frame.2.ref_count=0
global.callpool.stack.1.frame.2.translator=u0-client-0
global.callpool.stack.1.frame.2.complete=0
global.callpool.stack.1.frame.2.parent=u0-replicate-0
global.callpool.stack.1.frame.2.wind_from=afr_lk
global.callpool.stack.1.frame.2.wind_to=priv->children[i]->fops->lk
global.callpool.stack.1.frame.2.unwind_to=afr_lk_cbk

Seems some 'missing' frame in client-protocol. Let me get more info.

Comment 11 Harshavardhana 2012-08-02 18:14:39 UTC
Created attachment 602002 [details]
Gluster statedump 3.3.0

http://pastebin.gsslab.fab.redhat.com/mypaste/bLHk/ - testing with RHS 2.0 and  firefox 10.0.4

Comment 12 Harshavardhana 2012-08-02 18:15:08 UTC
Created attachment 602003 [details]
Gluster statedump 3.3.0

Comment 13 Harshavardhana 2012-08-03 04:56:33 UTC
Created attachment 602060 [details]
Sosreport node1

Comment 14 Harshavardhana 2012-08-03 04:56:54 UTC
Created attachment 602061 [details]
Sosreport node2

Comment 15 Harshavardhana 2012-08-03 05:08:43 UTC
Amar,

Testing with RHS 2.0 / GlusterFS 3.3.0 succeeds with no infinite hangs as it would with 3.2.x, but the real problem is that now the browser waits for like 3mins to open up a 'web' link. 

You could see a .parentlock file where the firefox is issuing a WRITE locking request (F_WRLCK)

Also odd thing is that a lot of files are being cached by io-cache for some reason, could that be cause for the slowness?

Do you have any inputs?

Comment 16 Harshavardhana 2012-10-23 01:05:44 UTC
http://rhn.redhat.com/errata/RHSA-2012-1350.html - believed to be a bug in Firefox , storage.nfs_filesystem not being enabled.

Upstream mozilla relevant bug - https://bugzilla.mozilla.org/show_bug.cgi?id=433129


Note You need to log in before you can comment on or make changes to this bug.