Bug 1554544

Summary: pg log does not trim when write errors are added, potentially leading to OOM
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Josh Durgin <jdurgin>
Component: RADOSAssignee: Josh Durgin <jdurgin>
Status: CLOSED ERRATA QA Contact: Parikshith <pbyregow>
Severity: urgent Docs Contact: Erin Donnelly <edonnell>
Priority: urgent    
Version: 3.0CC: ceph-eng-bugs, dzafman, edonnell, hnallurv, jdurgin, kchai, pdhiran, tserlin, vumrao
Target Milestone: z2   
Target Release: 3.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: RHEL: ceph-12.2.4-2.el7cp Ubuntu: ceph_12.2.4-3redhat1 Doc Type: Bug Fix
Doc Text:
Previously, a placement group (PG) that received only unsuccessful writes—such as deleting a non-existing object—would grow its write operations log indefinitely. This caused the OSDs handling that PG to run out of memory and crash. With this fix, the PG log is now trimmed as expected regardless of whether a write is successful or not, and the OSDs do not run out of memory.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-26 17:38:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1557269    

Description Josh Durgin 2018-03-12 23:00:33 UTC 
Description of problem:
When a rados operation that would be a write returns an error, a pg log entry is added, but the pg log is not trimmed. If no successful writes are completed against this pg, the pg log will continue growing with each new error. Since the pg log is kept in memory, this can lead to out of memory conditions and thus the osds holding that pg crashing.

Version-Release number of selected component (if applicable): 3.0


How reproducible: always


Steps to Reproduce:
1. delete a non-existent object more than osd_max_pg_log_size times, e.g. 'rados -p test rm foo'
2. find the pg that would hold object foo: 'ceph osd map test foo'
3. check the length of the pg log: 'ceph query $PGID foo -f json | jq .info.stats.log_size'

Actual results:
More than osd_max_pg_log_size entries

Expected results:
Fewer than osd_max_pg_log_size entries

Additional info:
See https://github.com/ceph/ceph/blob/luminous/qa/standalone/osd/repro_long_log.sh for a simple test case, which is included in the rados suite via: https://github.com/ceph/ceph/blob/luminous/qa/suites/rados/standalone/osd.yaml
Comment 6 Vikhyat Umrao 2018-03-13 14:45:43 UTC 
Luminous backport:

http://tracker.ceph.com/issues/23323
https://github.com/ceph/ceph/pull/20851
Comment 13 Vikhyat Umrao 2018-03-27 19:43:16 UTC 
*** Bug 1551721 has been marked as a duplicate of this bug. ***
Comment 17 errata-xmlrpc 2018-04-26 17:38:40 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1259
Comment 18 Michael J. Kidd 2018-08-17 19:27:24 UTC 
*** Bug 1616039 has been marked as a duplicate of this bug. ***