This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1481040 - ganesha.nfsd processes steadily using high CPU on 2 nodes.
ganesha.nfsd processes steadily using high CPU on 2 nodes.
Status: POST
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: nfs-ganesha (Show other bugs)
3.2
x86_64 Linux
unspecified Severity urgent
: ---
: ---
Assigned To: Soumya Koduri
Ambarish
:
Depends On: 1485836 1497139
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-13 15:15 EDT by Cal Calhoun
Modified: 2017-10-16 08:17 EDT (History)
17 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
ganesha_gstack_output_qe_setup (301.26 KB, text/plain)
2017-09-18 11:26 EDT, Soumya Koduri
no flags Details
ganesha_top_output_qe_setup (2.66 KB, text/plain)
2017-09-18 11:27 EDT, Soumya Koduri
no flags Details

  None (edit)
Comment 49 Soumya Koduri 2017-09-18 11:26 EDT
Created attachment 1327513 [details]
ganesha_gstack_output_qe_setup
Comment 50 Soumya Koduri 2017-09-18 11:27 EDT
Created attachment 1327514 [details]
ganesha_top_output_qe_setup
Comment 52 Soumya Koduri 2017-09-19 06:11:04 EDT
At-least from QE setup, since the high CPU usage seems to be from upcall polling, this particular issue shall be addressed as part of bug1485836
Comment 57 Kaleb KEITHLEY 2017-09-25 15:20:03 EDT
Yes, please install the -debuginfo rpm(s) so that the next core and backtrace will be more helpful.

(You don't need me on-line with you to do that, right?)
Comment 59 Daniel Gryniewicz 2017-09-26 08:47:46 EDT
Ganesha does not need to be restarted after debuginfo is installed.  debuginfo is just for gdb, not for ganesha itself.
Comment 62 Soumya Koduri 2017-09-26 10:22:37 EDT
Thats right Raghavendra. If we match the thread IDs (consuming CPU) in http://collab-shell.usersys.redhat.com/01909671/x-image/sh_remote_session_26092017.png against the ones in gdb stack (http://collab-shell.usersys.redhat.com/01909671/x-text/ganeshagdb.log), they are all upcall threads. That confirms our theory that upcall polling is adding up to high CPU usage even on customer environment. 

This is being addressed as part of bug1485836. We need to support on glusterfs to provide APIs to be able to register for asynchronous upcalls to be sent to application. Post that those APIs need to be consumed by ganesha.

Changes needed in gluster are sent upstream for review - 
https://review.gluster.org/#/c/18349/

Note You need to log in before you can comment on or make changes to this bug.