Bug 839768 - firefox-10.0.4-1.el5_8-x86_64 hang when rendering pages on glusterfs client
firefox-10.0.4-1.el5_8-x86_64 hang when rendering pages on glusterfs client
Status: CLOSED WONTFIX
Product: GlusterFS
Classification: Community
Component: core (Show other bugs)
3.2.6
Unspecified Unspecified
medium Severity high
: ---
: ---
Assigned To: Amar Tumballi
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-12 14:58 EDT by Harshavardhana
Modified: 2015-03-22 21:04 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-09-07 14:10:54 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
strace firefox (1.78 MB, application/x-bzip)
2012-07-12 14:59 EDT, Harshavardhana
no flags Details
Gluster dump during firefox hang (171.98 KB, application/x-bzip)
2012-07-12 14:59 EDT, Harshavardhana
no flags Details
Sosreport (13.39 MB, application/x-bzip)
2012-07-12 15:00 EDT, Harshavardhana
no flags Details
GlusterFS config melnode1 (2.20 KB, application/x-bzip)
2012-07-12 23:14 EDT, Harshavardhana
no flags Details
GlusterFS config melnode2 (2.21 KB, application/x-bzip)
2012-07-12 23:15 EDT, Harshavardhana
no flags Details
Gluster Client/Server logs melnode1 (11.64 MB, application/x-bzip)
2012-07-12 23:16 EDT, Harshavardhana
no flags Details
Gluster Client/Server logs melnode2 (2.06 MB, application/x-bzip)
2012-07-12 23:17 EDT, Harshavardhana
no flags Details
Gluster statedump 3.3.0 (77.99 KB, application/octet-stream)
2012-08-02 14:14 EDT, Harshavardhana
no flags Details
Gluster statedump 3.3.0 (17.17 KB, application/x-gzip)
2012-08-02 14:15 EDT, Harshavardhana
no flags Details
Sosreport node1 (689.20 KB, application/octet-stream)
2012-08-03 00:56 EDT, Harshavardhana
no flags Details
Sosreport node2 (700.14 KB, application/octet-stream)
2012-08-03 00:56 EDT, Harshavardhana
no flags Details

  None (edit)
Description Harshavardhana 2012-07-12 14:58:30 EDT
Description of problem:

Upgrade to firefox-10.0.4-1.el5_8-x86_64 from firefox-3.0.6-1.el5.x86_64.rpm.

is found that firefox 10.0.4 hangs when loading most web pages on glusterfs.

Please find attached the output of: strace -s 1024 -v /usr/bin/firefox

The hang was induced by browsing to google.com

The process was sent a SIGINT to terminate.

The server is headless, the process was using a remote X11 server for DISPLAY.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
GlusterFS version 3.2.6, RHEL5.8 Client x86_64 - install firefox-10.0.4-1 run it over GlusterFS mount, open a webpage and just hangs. 
  
Actual results:
Firefox hang

Expected results:
Firefox renders web page successfully.
Comment 1 Harshavardhana 2012-07-12 14:59:15 EDT
Created attachment 597878 [details]
strace firefox
Comment 2 Harshavardhana 2012-07-12 14:59:46 EDT
Created attachment 597879 [details]
Gluster dump during firefox hang
Comment 3 Harshavardhana 2012-07-12 15:00:26 EDT
Created attachment 597880 [details]
Sosreport
Comment 4 Harshavardhana 2012-07-12 15:15:22 EDT
Waiting on 


* tar czf gluster_configuration.tar.gz /etc/glusterd from servers

* tar czf gluster_logs_servers.tar.gz /var/log/glusterfs from servers

* tar czf gluster_logs_client.tar.gz /var/log/glusterfs from client.
Comment 5 Harshavardhana 2012-07-12 23:14:50 EDT
Created attachment 597944 [details]
GlusterFS config melnode1

The two servers are also the clients, there are no other clients, so the log archives contain both the client and server process output.
Comment 6 Harshavardhana 2012-07-12 23:15:19 EDT
Created attachment 597945 [details]
GlusterFS config melnode2
Comment 7 Harshavardhana 2012-07-12 23:16:07 EDT
Created attachment 597947 [details]
Gluster Client/Server logs melnode1
Comment 8 Harshavardhana 2012-07-12 23:17:00 EDT
Created attachment 597948 [details]
Gluster Client/Server logs melnode2
Comment 9 Amar Tumballi 2012-07-13 01:25:08 EDT
Harsha,

I will be going through the logs now, meantime, wanted to understand what would be the behavior on 3.3.0 release? Is it possible to test that here?

-Amar
Comment 10 Amar Tumballi 2012-07-19 06:35:50 EDT
[global.callpool.stack.1.frame.2]
global.callpool.stack.1.frame.2.ref_count=0
global.callpool.stack.1.frame.2.translator=u0-client-0
global.callpool.stack.1.frame.2.complete=0
global.callpool.stack.1.frame.2.parent=u0-replicate-0
global.callpool.stack.1.frame.2.wind_from=afr_lk
global.callpool.stack.1.frame.2.wind_to=priv->children[i]->fops->lk
global.callpool.stack.1.frame.2.unwind_to=afr_lk_cbk

Seems some 'missing' frame in client-protocol. Let me get more info.
Comment 11 Harshavardhana 2012-08-02 14:14:39 EDT
Created attachment 602002 [details]
Gluster statedump 3.3.0

http://pastebin.gsslab.fab.redhat.com/mypaste/bLHk/ - testing with RHS 2.0 and  firefox 10.0.4
Comment 12 Harshavardhana 2012-08-02 14:15:08 EDT
Created attachment 602003 [details]
Gluster statedump 3.3.0
Comment 13 Harshavardhana 2012-08-03 00:56:33 EDT
Created attachment 602060 [details]
Sosreport node1
Comment 14 Harshavardhana 2012-08-03 00:56:54 EDT
Created attachment 602061 [details]
Sosreport node2
Comment 15 Harshavardhana 2012-08-03 01:08:43 EDT
Amar,

Testing with RHS 2.0 / GlusterFS 3.3.0 succeeds with no infinite hangs as it would with 3.2.x, but the real problem is that now the browser waits for like 3mins to open up a 'web' link. 

You could see a .parentlock file where the firefox is issuing a WRITE locking request (F_WRLCK)

Also odd thing is that a lot of files are being cached by io-cache for some reason, could that be cause for the slowness?

Do you have any inputs?
Comment 16 Harshavardhana 2012-10-22 21:05:44 EDT
http://rhn.redhat.com/errata/RHSA-2012-1350.html - believed to be a bug in Firefox , storage.nfs_filesystem not being enabled.

Upstream mozilla relevant bug - https://bugzilla.mozilla.org/show_bug.cgi?id=433129

Note You need to log in before you can comment on or make changes to this bug.