Bug 1852736 - High CPU usage on EC volume after inservice upgrade of one node in 3 node cluster
Summary: High CPU usage on EC volume after inservice upgrade of one node in 3 node clu...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: disperse
Version: rhgs-3.5
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: RHGS 3.5.z Batch Update 3
Assignee: Xavi Hernandez
QA Contact: Manisha Saini
URL:
Whiteboard:
Depends On: 1853594
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-01 08:06 UTC by Leela Venkaiah Gangavarapu
Modified: 2020-12-17 04:52 UTC (History)
10 users (show)

Fixed In Version: glusterfs-6.0-41
Doc Type: Enhancement
Doc Text:
Earlier, Gluster kept trying to heal the files which failed and remain unhealed consuming a significant amount of CPU. With this enhancement, Gluster has a better way to detect when continuous healing is necessary and reduces CPU utilization when pending heals cannot be immediately healed.
Clone Of:
Environment:
Last Closed: 2020-12-17 04:51:50 UTC
Embargoed:


Attachments (Terms of Use)
CPU usage on server and client (2.46 MB, video/webm)
2020-07-01 08:06 UTC, Leela Venkaiah Gangavarapu
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:5603 0 None None None 2020-12-17 04:52:24 UTC

Description Leela Venkaiah Gangavarapu 2020-07-01 08:06:20 UTC
Created attachment 1699439 [details]
CPU usage on server and client

Description of problem:
High CPU usage is being observed after in-service upgrade of one node in a 3 node cluster

Version-Release number of selected component (if applicable):
glusterfs-server-6.0-37.1.el7rhgs.x86_64

How reproducible:
Consistent

Steps to Reproduce:
1. A cluster with 3 nodes hosting 4X(4+2) dist-disp vol and 3X3 repl vol
2. Upgraded one of the nodes when dist-disp is ~5% full and repl is ~35% full
3. Monitoring CPU during the heal process and post heal process
4. Saw a sudden spike in CPU and still continuing even after heal is complete
5. Observe CPU spikes for gluster process on servers
6. Command used "$ top -c -p $(pgrep -d',' -f gluster)" on client and server

Actual results:
CPU Usage spikes reaching ~700-800%

Expected results:
CPU Usage should be moderate <100%

Additional info:
- No heals are pending
- The other two nodes are in `glusterfs-6.0-37.el7rhgs.x86_64` version

Comment 29 errata-xmlrpc 2020-12-17 04:51:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (glusterfs bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5603


Note You need to log in before you can comment on or make changes to this bug.