Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1380196 - [RHCS 1.3.z] RGW goes into loop causing 100% CPU utilization
[RHCS 1.3.z] RGW goes into loop causing 100% CPU utilization
Status: CLOSED ERRATA
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW (Show other bugs)
1.3.2
x86_64 Linux
unspecified Severity high
: rc
: 1.3.4
Assigned To: Marcus Watts
ceph-qe-bugs
:
Depends On:
Blocks: 1394936
  Show dependency treegraph
 
Reported: 2016-09-28 18:17 EDT by Justin Bautista
Modified: 2017-07-30 11:41 EDT (History)
13 users (show)

See Also:
Fixed In Version: RHEL: ceph-0.94.9-6.el7cp Ubuntu: ceph_0.94.9-7redhat1
Doc Type: Bug Fix
Doc Text:
Dynamic Large Objects (DLOs) containing zero-length segments could cause the GET requests to enter an infinite loop or to fail depending on the position of the zero-length segment in an object. Consequently, when a GET request entered to an infinite loop, Ceph Object Gateway nodes could consume 100% of CPU. The underlying source code has been modified, and the GET requests succeed as expected in described situation.
Story Points: ---
Clone Of:
: 1383728 (view as bug list)
Environment:
Last Closed: 2016-12-01 16:39:03 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Ceph Project Bug Tracker 17572 None None None 2016-10-13 13:18 EDT
Red Hat Product Errata RHSA-2016:2847 normal SHIPPED_LIVE Moderate: Red Hat Ceph Storage 1.3 security, bug fix, and enhancement update 2016-12-01 21:38:33 EST
Red Hat Product Errata RHSA-2016:2848 normal SHIPPED_LIVE Moderate: Red Hat Ceph Storage 1.3 security, bug fix, and enhancement update 2016-12-01 22:04:31 EST

  None (edit)
Description Justin Bautista 2016-09-28 18:17:04 EDT
Description of problem:

  Virtualized RGW host goes into loop causing 100% CPU utilization.

Version-Release number of selected component (if applicable):

  ceph-radosgw-0.94.5-12.el7cp.x86_64
  RHEL 7.2 servers running OpenStack Kilo with Ceph Storage Backend
===========================================================================

Customer reports that after setting up 3 virtualized RGW nodes several months ago & during testing they've noticed that all 3 are running at 100% CPU (95% of the time) even though they don'think they are being heavily used. We requested RGW debug logs &  they are attached to case 01705883 as 'client.radosgw.log.gz'  

The customer also provided the following details when asked for a specific timeframe to focus on:

"Between 10:10 & 10:20 AM . And it was not using ALL CPU's, but it was using more than 50% (which should have been the limit) and if we had left it alone it would have eventually used all the CPU's. Below is the partial output from sar for that timeframe ...

09:50:01 AM     all      0.07      0.00      0.04      0.00      0.00     99.89
10:00:01 AM     all      0.08      0.00      0.06      0.00      0.00     99.86
10:10:01 AM     all      0.31      0.00      0.22      0.00      0.00     99.48
10:20:01 AM     all     29.40      0.00     30.32      0.05      0.04     40.20
10:30:01 AM     all      9.51      0.00     13.63      0.01      0.01     76.84"

==================================================================================== 

Log analysis of 'log entries per second' ramps up quickly starting at the 10:09:41 mark in the debug log, and appears to go into a loop around that time.
Comment 36 shilpa 2016-11-27 11:10:30 EST
Verified on ceph-radosgw-0.94.9-7
Comment 41 errata-xmlrpc 2016-12-01 16:39:03 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2847.html

Note You need to log in before you can comment on or make changes to this bug.