RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2116602 - [cee/sd][RFE]sosreport is not capturing ceph Orchestrator CLI outputs.
Summary: [cee/sd][RFE]sosreport is not capturing ceph Orchestrator CLI outputs.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: sos
Version: 8.6
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Jose Castillo
QA Contact: Daniel Záležák
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-08-09 02:29 UTC by Geo Jose
Modified: 2023-05-31 18:49 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-05-31 18:49:38 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github sosreport sos pull 3179 0 None open [ceph_mgr] Gather some commands in json format 2023-03-30 13:05:45 UTC
Red Hat Issue Tracker RHELPLAN-130551 0 None None None 2022-08-09 02:32:42 UTC

Description Geo Jose 2022-08-09 02:29:11 UTC
Description of problem:
----------------------

When generating sos report, ceph plugin won't capture ceph Orchestrator CLI outputs from RHCS5 cluster.  

Currently, this will capture the below commands and which doesn't include ceph Orchestrator CLI outputs:
~~~
# ls sos_commands/ceph/
ceph_balancer_status               ceph_df_detail      ceph_mgr_dump        ceph_osd_crush_show-tunables      ceph_osd_pool_autoscale-status  ceph_status
ceph_config_generate-minimal-conf  ceph-disk_list      ceph_mgr_metadata    ceph_osd_df                       ceph_osd_pool_ls_detail         ceph_time-sync-status
ceph_config-key_dump               ceph_features       ceph_mgr_module_ls   ceph_osd_df_tree                  ceph_osd_stat                   ceph_-v
ceph_config_log                    ceph_fs_dump        ceph_mon_dump        ceph_osd_dump                     ceph_osd_tree                   ceph_versions
ceph_crash_ls                      ceph_fs_ls          ceph_mon_stat        ceph_osd_erasure-code-profile_ls  ceph_pg_dump                    ceph-volume_lvm_list
ceph_crash_stat                    ceph_health_detail  ceph_mon_status      ceph_osd_metadata                 ceph_pg_stat                    json_output
ceph_device_ls                     ceph_insights       ceph_osd_blocked-by  ceph_osd_numa-status              ceph_quorum_status
ceph_df                            ceph_mds_stat       ceph_osd_crush_dump  ceph_osd_perf                     ceph_report
# ls sos_commands/ceph/json_output/
ceph_device_ls_--format_json-pretty       ceph_osd_dump_--format_json-pretty
ceph_df_detail_--format_json-pretty       ceph_osd_numa-status_--format_json-pretty
ceph_df_--format_json-pretty              ceph_osd_perf_--format_json-pretty
ceph_fs_dump_--format_json-pretty         ceph_osd_pool_autoscale-status_--format_json-pretty
ceph_fs_ls_--format_json-pretty           ceph_osd_pool_ls_detail_--format_json-pretty
ceph_health_detail_--format_json-pretty   ceph_osd_stat_--format_json-pretty
ceph_mds_stat_--format_json-pretty        ceph_osd_tree_--format_json-pretty
ceph_mgr_dump_--format_json-pretty        ceph_pg_dump_--format_json-pretty
ceph_mon_dump_--format_json-pretty        ceph_pg_stat_--format_json-pretty
ceph_osd_blocked-by_--format_json-pretty  ceph_status_--format_json-pretty
ceph_osd_df_--format_json-pretty          ceph_time-sync-status_--format_json-pretty
ceph_osd_df_tree_--format_json-pretty
~~~

The below commands needs to be included with ceph plugin:
~~~
- To list hosts:
# ceph orch host ls

- To list devices on a host
# ceph orch device ls

- To check the curent specification file/List services known to orchestrator
# ceph orch ls --export

- To list daemons known to orchestrator:
# ceph orch ps

- To check the configured backend and its status
# ceph orch status --detail

- To check service versions vs available and target containers
# ceph orch upgrade status

- To see the recent activities/log from cephadm:
# ceph log last cephadm
~~~
The above commands are needed only for RHCS 5 or later clusters.


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
sos-4.2-20.el8_6.noarch

Comment 6 Pavel Moravec 2022-09-15 06:58:00 UTC
Hello,
ceph plugin has been split into several ones, in https://github.com/sosreport/sos/issues/1945 / https://github.com/sosreport/sos/commit/13ce81156970034375e3a6985a7c147e281486f3 .

What new plugin should collect the above commands outputs, please? See their list in https://github.com/sosreport/sos/tree/main/sos/report/plugins .

To the "collect only on v.5 or newer" - as 1) the command is harmless (to the system) on older ceph versions, 2) it will take some time to deliver this into RHEL downstream (8.9/9.3 ?) when <5 versions will be less installed, 3) adding a version check "just" to stop preventing "subcommand not known" cmdoutputs, I suggest dont restrict the commands executions to "version 5+" condition. Does this sound acceptable?

Comment 8 Pavel Moravec 2022-09-15 14:03:27 UTC
There is no ceph plugin in upstream (in any version where this BZ will be implemented). There are plugins:

- ceph_ansible, by default enabled by presence of ceph-ansible package
- ceph_common, enabled by either ceph-(mon|rgw|osd).* container present
- ceph_mds, enabled by presence of a /var/lib/ceph/mds/*/* file
- ceph_mgr, enabled by presence of a /var/lib/ceph/mgr/*/* file
- ceph_mon, enabled by presence of a /var/lib/ceph/mon/*/* file
- ceph_osd, enabled by presence of a /var/lib/ceph/osd/*/* file
- ceph_rgw, enabled by presence of a /var/lib/ceph/radosgw/*/* file

Anyway, I realized all the commands *are* already collected in upstream sos, in ceph_mgr plugin:

https://github.com/sosreport/sos/blob/main/sos/report/plugins/ceph_mgr.py#L40-L46

(without the "--format json-pretty" option - is that sufficient?)

All that is already available in a RHEL8.8 candidate build https://people.redhat.com/pmoravec/sos-4.4-1/ - let try that ;-)


About the "on RHCS 4, write a specific command output instead of a default error one" - we dont "spoof" commands outputs like that. We either collect a command output (or a file) or not (based on any logic, if required), and we obfuscate secrets. But replacing a valid command output by some artificial text, that would be confusing. E.g. when one notices a warning or error, (s)he searches for that string in either source (KCS, source code of the program). Finding no hit there would be confusing.

Anyway, the "Error EINVAL: invalid command" suggests the user that the (sub)command is invalid..?

Comment 10 Pavel Moravec 2022-10-10 15:02:40 UTC
Just to clarify / double-confirm: it is preferred to collect the ceph commands with "--format json-pretty" argument, which is supported in every ceph release, right? I.e. patch like:

--- a/sos/report/plugins/ceph_mgr.py
+++ b/sos/report/plugins/ceph_mgr.py
@@ -37,13 +37,13 @@ class CephMGR(Plugin, RedHatPlugin, UbuntuPlugin):
         # more commands to be added later
         self.add_cmd_output([
             "ceph balancer status",
-            "ceph orch host ls",
-            "ceph orch device ls",
-            "ceph orch ls --export",
-            "ceph orch ps",
-            "ceph orch status --detail",
-            "ceph orch upgrade status",
-            "ceph log last cephadm"
+            "ceph orch host ls --format json-pretty",
+            "ceph orch device ls --format json-pretty",
+            "ceph orch ls --export --format json-pretty",
+            "ceph orch ps --format json-pretty",
+            "ceph orch status --detail --format json-pretty",
+            "ceph orch upgrade status --format json-pretty",
+            "ceph log last cephadm --format json-pretty"
         ])

?


About "ceph needs to call sosreport -e ceph" - why? Why the plugins are not enabled on ceph (respectivelly on its individual components) by default?

How could sos get know it is running on, say, ceph osd? Is there a package specific to that? A container? A kernel module or command output? Or a filename present?

FYI we split ceph plugin per components in https://github.com/sosreport/sos/commit/13ce81156970034375e3a6985a7c147e281486f3 , so a workaround of documentation is replacing "-e ceph" by "-e ceph_common,ceph_mds,ceph_mgr,ceph_mon,ceph_osd,ceph_rgw"

(could you please raise doc BZ for it? otherwise I will once I find where to file it..)


Please double-confirm the above --forman json-pretty change, plus tell us how sos can recognize individual ceph components (via either package, container, kmod, cmdoutput or filename, if possible).

Comment 16 Pavel Moravec 2022-12-29 16:01:25 UTC
Raising needinfo on Jose as he had some pending work/PR for this.

Comment 17 Jose Castillo 2023-01-03 12:57:23 UTC
I'm reworking the PR because of the recent changes to the ceph plugins upstream via 3092 and 3089. I'm aiming to have it ready asap.

Comment 26 Jan Jansky 2023-05-17 14:56:35 UTC
Can you please test candidate release for us?

https://people.redhat.com/pmoravec/sos-4.5.3-1/el8/

Comment 36 errata-xmlrpc 2023-05-31 18:49:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sos bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:3414


Note You need to log in before you can comment on or make changes to this bug.