Bug 2240838 - [6.1.z backport][RADOS] "currently delayed" slow ops does not provide details on why op has been delayed
Summary: [6.1.z backport][RADOS] "currently delayed" slow ops does not provide details...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RADOS
Version: 5.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: 6.1z3
Assignee: Prashant Dhange
QA Contact: Pawan
Disha Walvekar
URL:
Whiteboard:
Depends On: 2240832 2240839
Blocks: 2247624
TreeView+ depends on / blocked
 
Reported: 2023-09-26 20:58 UTC by Vikhyat Umrao
Modified: 2023-12-12 13:56 UTC (History)
10 users (show)

Fixed In Version: ceph-17.2.6-157.el9cp
Doc Type: Enhancement
Doc Text:
Feature: Report sub-events for ops which are marked as delayed. Reason: The slow ops which was marked as delayed does not provide details on why op was marked as delayed e.g is it because of "waiting for rw locks" or "waiting for missing objects" or "waiting for peered" etc ? Result: A detailed description is being provided for the op which is marked as delayed now.
Clone Of: 2240832
Environment:
Last Closed: 2023-12-12 13:55:56 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 62995 0 None None None 2023-09-26 23:11:15 UTC
Github ceph ceph pull 53690 0 None open quincy: osd/OpRequest: Add detail description for delayed op in osd log file 2023-09-26 23:20:09 UTC
Red Hat Issue Tracker RHCEPH-7554 0 None None None 2023-09-26 20:59:44 UTC
Red Hat Product Errata RHSA-2023:7740 0 None None None 2023-12-12 13:56:01 UTC

Description Vikhyat Umrao 2023-09-26 20:58:07 UTC
+++ This bug was initially created as a clone of Bug #2240832 +++

Description of problem:
With reference to BZ#2240819, the osd.0 observed slow ops and most of the slow ops were delayed but with no details on why op marked as delayed e.g is it because of "waiting for rw locks" or "waiting for missing objects" or "waiting for peered" etc.

There could be different reason for op being marked as delayed and it could be either of below reason :
  op->mark_delayed("waiting for missing object");
  op->mark_delayed("waiting for degraded object");
  op->mark_delayed("waiting for cache not full");
  op->mark_delayed("waiting for clean to repair");
  op->mark_delayed("waiting for blocked object");
  op->mark_delayed("waiting for readable");
  op->mark_delayed("waiting for readable");
          op->mark_delayed("waiting for scrub");
          op->mark_delayed("waiting for readable");
    op->mark_delayed("waiting_for_map not empty");
      op->mark_delayed("waiting for peered");
    op->mark_delayed("waiting for flush");
      op->mark_delayed("waiting for active");
      op->mark_delayed("waiting for scrub");
	op->mark_delayed("waiting for ondisk");
    op->mark_delayed("waiting for rw locks");
	op->mark_delayed("waiting for scrub");
      op->mark_delayed("waiting for scrub");
  op->mark_delayed("waiting for missing object");

Version-Release number of selected component (if applicable):
RHCS 7

How reproducible:
Frequently

Steps to Reproduce:
1. Deploy ceph cluster
2. Run extensive client workload against the ceph cluster 
3. Observe "currently delayed" slow ops

Actual results:
The delayed ops does provide details on reason for op being flagged as delayed

Expected results:
The delayed ops should provide details on reason for op being flagged as delayed

Additional info:

--- Additional comment from Vikhyat Umrao on 2023-09-26 20:44:44 UTC ---

Marking this one blocker because it is a kind of regression and causing issues in troubleshooting slow requests!

--- Additional comment from Vikhyat Umrao on 2023-09-26 20:56:51 UTC ---

The issue was reported in ODF 4.10 which is nothing but 5.1.z2 - 16.2.7-126 hence changing the reported version to 5.1!

Comment 12 errata-xmlrpc 2023-12-12 13:55:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 6.1 security, enhancements, and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:7740


Note You need to log in before you can comment on or make changes to this bug.