Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
Looks like there is a bug on getting list of nodes with `pcs status`. It appears we returning NONE from `pcs status` output so fails to get list of nodes.
I am able to get pcs status:
# pcs status
Cluster name: rhel8cluster
Cluster Summary:
* Stack: corosync
* Current DC: rhel8-1.examplerh.com (version 2.1.0-8.el8-7c3f660707) - partition with quorum
* Last updated: Fri Mar 18 14:06:51 2022
* Last change: Fri Mar 18 13:59:29 2022 by root via cibadmin on rhel8-1.examplerh.com
* 3 nodes configured
* 3 resource instances configured (2 DISABLED)
Node List:
* Online: [ rhel8-1.examplerh.com rhel8-2.examplerh.com rhel8-3.examplerh.com ]
Full List of Resources:
* d-01 (ocf::pacemaker:Dummy): Stopped (disabled)
* d-02 (ocf::pacemaker:Dummy): Stopped (disabled)
* virtfence_xvm (stonith:fence_xvm): Started rhel8-1.examplerh.com
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
-----------
Fails to collect sosreports because it cannot get list of nodes.
# sos collect -o corosync,pacemaker -vvvvv --all-logs --case-id=999999 --batch
[sos_collector:__init__] Executing /usr/sbin/sos collect -o corosync,pacemaker -vvvvv --all-logs --case-id=999999 --batch
[sos_collector:__init__] Found cluster profiles: dict_keys(['jbon', 'kubernetes', 'ocp', 'ovirt', 'rhhi_virt', 'rhv', 'pacemaker', 'satellite'])
sos-collector (version 4.1)
This utility is used to collect sosreports from multiple nodes simultaneously.
It uses OpenSSH's ControlPersist feature to connect to nodes and run commands
remotely. If your system installation of OpenSSH is older than 5.6, please
upgrade.
An archive of sosreport tarballs collected from the nodes will be generated in
/var/tmp/sos.kntlhfn6 and may be provided to an appropriate support
representative.
The generated archive may contain data considered sensitive and its content
should be reviewed by the originating organization before being passed to any
third party.
No configuration changes will be made to the system running this utility or
remote systems that it connects to.
[sos_collector:configure_sos_cmd] Initial sos cmd set to sosreport --batch --case-id=999999 --all-logs -c auto
[sos_collector:prep] password not specified, assuming SSH keys
sos-collector ASSUMES that SSH keys are installed on all nodes unless the
--password option is provided.
[localhost:determine_host_policy] using local policy Red Hat Enterprise Linux
[localhost:run_command] Running command hostname
[rhel8-1.examplerh.com:get_hostname] Hostname set to rhel8-1.examplerh.com
[rhel8-1.examplerh.com:_load_sos_info] sos version is 4.1
[rhel8-1.examplerh.com:run_command] Running command sosreport -l
[rhel8-1.examplerh.com:run_command] Running command sosreport --list-presets
[rhel8-1.examplerh.com:run_command] Running command oc whoami
[sos_collector:determine_cluster] Installation matches pacemaker, checking for layered profiles
Cluster type set to Pacemaker High Availability Cluster Manager
[rhel8-1.examplerh.com:run_command] Running command pcs status
Cluster failed to enumerate nodes: 'NoneType' object is not iterable
[pacemaker] Failed to get node list: 'NoneType' object is not iterable
[sos_collector:get_nodes_from_cluster] Node list: []
[sos_collector:reduce_node_list] Node list reduced to []
The following is a list of nodes to collect from:
rhel8-1.examplerh.com
[archive:TarFileArchive] initialised empty FileCacheArchive at '/var/tmp/sos.kntlhfn6/sos-collector-999999-2022-03-18-talcu'
Connecting to nodes...
Collection would only gather from localhost due to failure to either enumerate or connect to cluster nodes. Assuming single collection from localhost is not desired.
Aborting...
[sos_collector:close_all_connections] Closing SSH connection to localhost
Version-Release number of selected component (if applicable):
sos-4.1-9.el8_5.noarch
How reproducible:
Every time
Steps to Reproduce:
1. Make sure pacemaker is started and pcs status shows nodes online.
2. sos collect -o corosync,pacemaker -vvvvv --all-logs --case-id=999999 --batch
Actual results:
It fails to get node list from "pcs status" output.
Expected results:
It should collect sosreports from all nodes in the cluster.
Additional info:
If i pass `--nodes` then it works.
# sos collect -o corosync,pacemaker -vvvvv --all-logs --case-id=999999 --batch --nodes=rhel8-1.examplerh.com,rhel8-2.examplerh.com,rhel8-3.examplerh.com
I would not recommend using the output of `pcs status` in any script. Its output is subject to change.
It's better to use an XML-based format.
[root@fastvm-rhel-8-0-23 collector]# crm_mon --one-shot --inactive --output-as=xml
<pacemaker-result api-version="2.12" request="crm_mon --one-shot --inactive --output-as=xml">
<summary>
<stack type="corosync"/>
<current_dc present="true" version="2.1.0-8.el8-7c3f660707" name="node1" id="1" with_quorum="false"/>
<last_update time="Fri Mar 18 11:54:38 2022"/>
<last_change time="Thu Mar 17 23:24:52 2022" user="root" client="cibadmin" origin="node1"/>
<nodes_configured number="2"/>
<resources_configured number="1" disabled="0" blocked="0"/>
<cluster_options stonith-enabled="true" symmetric-cluster="true" no-quorum-policy="stop" maintenance-mode="false" stop-all-resources="false"/>
</summary>
<nodes>
<node name="node1" id="1" online="true" standby="false" standby_onfail="false" maintenance="false" pending="false" unclean="false" shutdown="false" expected_up="true" is_dc="true" resources_running="0" type="member"/>
<node name="node2" id="2" online="false" standby="false" standby_onfail="false" maintenance="false" pending="false" unclean="true" shutdown="false" expected_up="false" is_dc="false" resources_running="0" type="member"/>
</nodes>
<resources>
<resource id="xvm" resource_agent="stonith:fence_xvm" role="Stopped" active="false" orphaned="false" blocked="false" managed="true" failed="false" failure_ignored="false" nodes_running_on="0"/>
</resources>
<node_history/>
<status code="0" message="OK"/>
</pacemaker-result>
In fact, this just came up on the upstream list this morning.
- [ClusterLabs] Parsing the output of crm_mon (https://lists.clusterlabs.org/pipermail/users/2022-March/030131.html)
Per kgaillot:
> The XML output is stable and only gets backward-compatible additions
> once in a long while, but the text output changes more frequently and
> significantly.
---
In the case of this bug, the `pcs status` output format has indeed changed. Here's part of the logic from clusters/pacemaker.py in sos:
~~~
def get_online_nodes(self):
for line in self.res['stdout'].splitlines():
if line.startswith('Online:'):
nodes = line.split('[')[1].split(']')[0]
return [n for n in nodes.split(' ') if n]
~~~
But the line actually starts with "* Online".
Prior to pacemaker-2.0.3, the option is `--as-xml` instead of `--output-as=xml`. pacemaker-2.0.3 is from RHEL 8.2. Some RHEL 8 users may have an older pacemaker version installed (for whatever reason) while using an up-to-date sos package. So we should support both options. Either check the pacemaker version before running the `crm_mon` command to determine which option is appropriate, or fallback to --as-xml via try/except.
sos-collector has been integrated into sos project since 4.0 times, so reassigning back to sos.
https://github.com/sosreport/sos/pull/2891 should fix the issue, currently it will reach RHEL 8.8 / 9.2.
Pavel,
As it stands `sos collect` is broken for pacemaker support due to the change in `pcs status` output. I think this should be a candidate for an 8.6.z fix, rather than treating it as a new feature to go out in 8.8.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (sos bug fix and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2022:7732