Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1859885

Summary: IndexError from kubernetes plugin when parsing nodes [rhel-7.9.z]
Product: Red Hat Enterprise Linux 7 Reporter: Miroslav Hradílek <mhradile>
Component: sosAssignee: Jan Jansky <jjansky>
Status: CLOSED ERRATA QA Contact: Maros Kopec <makopec>
Severity: low Docs Contact:
Priority: medium    
Version: 7.8CC: agk, bmr, fkrska, jhunsaker, jreznik, plambri, pmoravec, sbradley
Target Milestone: rcKeywords: ZStream
Target Release: 7.9   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: sos-3.9-5.el7_9.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1859888 (view as bug list) Environment:
Last Closed: 2021-02-02 11:59:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1859888    

Description Miroslav Hradílek 2020-07-23 09:21:06 UTC
Plugin kubernetes in sos 3.9 started parsing output of "kubectl get nodes" in order to run "kubectl describe node" for each parsed node. The problem is that sosreport does not expect empty lines in "kubectl get nodes" output and if such output should occur plugin crashes with a traceback.

There was similar bug 1635035 in the past concerning parsing namespaces from "kubectl get namespaces" with the same outcome. Although i found about this by crafted artificial output instead of real "kubectl" output i believe this could reappear as a sort of regression in the future if somebody happens to hit error from "kubectl get nodes" when using sosreport.

Not sure if i can call this regression if it is in a different part of code but behavior is the same.

Setup:
======
mkdir -p /etc/origin/master/
echo "echo 'nonempty line'; echo '   '; echo 'another nonempty'" > /usr/bin/kubectl
chmod +x /usr/bin/kubectl
mkdir -p /etc/origin/node/pods/
touch /etc/origin/node/pods/master-config.yaml

Old:
====
# rpm -q sos
sos-3.8-8.el7_8.noarch

# sosreport -vvv -o kubernetes --batch
# tar -tf /var/tmp/sosreport-sheep-2-2020-07-23-ruybyqj.tar.xz
sosreport-sheep-2-2020-07-23-ruybyqj/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/journalctl_--no-pager_--unit_kubelet
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-apiserver
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-proxy
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-scheduler
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-controller-manager
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/kubectl_version
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/kubectl_config_view
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/kubectl_get_namespaces
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/kubectl_get_nodes
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/kubectl_get_projects
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/kubectl_get_pvs
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/kubectl_get_--raw_.metrics
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/deployments/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/deployments/kubectl_get_--all-namespaces_true_deployments
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/ingresses/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/ingresses/kubectl_get_--all-namespaces_true_ingresses
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/limitranges/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/limitranges/kubectl_get_--all-namespaces_true_limitranges
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/pods/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/pods/kubectl_get_--all-namespaces_true_pods
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/policies/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/policies/kubectl_get_--all-namespaces_true_policies
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/pvc/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/pvc/kubectl_get_--all-namespaces_true_pvc
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/rc/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/rc/kubectl_get_--all-namespaces_true_rc
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/resourcequotas/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/resourcequotas/kubectl_get_--all-namespaces_true_resourcequotas
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/routes/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/routes/kubectl_get_--all-namespaces_true_routes
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/services/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/services/kubectl_get_--all-namespaces_true_services
sosreport-sheep-2-2020-07-23-ruybyqj/sos_logs/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_logs/sos.log
sosreport-sheep-2-2020-07-23-ruybyqj/sos_logs/ui.log
sosreport-sheep-2-2020-07-23-ruybyqj/sos_reports/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_reports/sos.txt
sosreport-sheep-2-2020-07-23-ruybyqj/sos_reports/sos.json
sosreport-sheep-2-2020-07-23-ruybyqj/sos_reports/sos.html
sosreport-sheep-2-2020-07-23-ruybyqj/environment
sosreport-sheep-2-2020-07-23-ruybyqj/version.txt


New (unreleased):
====
# rpm -q sos
sos-3.9-4.el7_9.noarch
# sosreport -vvv -o kubernetes --batch
. . .
# tar -tf /var/tmp/sosreport-sheep-2-2020-07-23-trcwqgn.tar.xz
sosreport-sheep-2-2020-07-23-trcwqgn/
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/kubectl_get_namespaces
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/kubectl_get_nodes
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/journalctl_--no-pager_--unit_kubelet
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-apiserver
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-proxy
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-scheduler
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-controller-manager
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/kubectl_version
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/kubectl_config_view
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/kubectl_get_namespaces.1
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/kubectl_get_projects
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/kubectl_get_pvs
sosreport-sheep-2-2020-07-23-trcwqgn/sos_logs/
sosreport-sheep-2-2020-07-23-trcwqgn/sos_logs/kubernetes-plugin-errors.txt
sosreport-sheep-2-2020-07-23-trcwqgn/sos_logs/sos.log
sosreport-sheep-2-2020-07-23-trcwqgn/sos_logs/ui.log
sosreport-sheep-2-2020-07-23-trcwqgn/sos_reports/
sosreport-sheep-2-2020-07-23-trcwqgn/sos_reports/sos.txt
sosreport-sheep-2-2020-07-23-trcwqgn/sos_reports/sos.json
sosreport-sheep-2-2020-07-23-trcwqgn/sos_reports/sos.html
sosreport-sheep-2-2020-07-23-trcwqgn/version.txt
# tar -xf /var/tmp/sosreport-sheep-2-2020-07-23-trcwqgn.tar.xz
# cat sosreport-sheep-2-2020-07-23-trcwqgn/sos_logs/kubernetes-plugin-errors.txt
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 981, in setup
    plug.setup()
  File "/usr/lib/python2.7/site-packages/sos/plugins/kubernetes.py", line 103, in setup
    node = line.split()[0]
IndexError: list index out of range

Comment 3 Pavel Moravec 2020-07-23 10:29:37 UTC
The problem arises (only) when "kubectl get nodes" returns a line without non-whitsespace characters, i.e. when it returns either empty line, or a line with whitespace only.

After a discussion with a containers guy:
- normal output of "kubectl get nodes" does not contain such lines
- that command output can contain such line when:
  - a config (file or option) is malformed
  - the command is unable to contact apiserver

Both ways mean in 99% that all other kubectl commands will fail the same way. So the harm of not collecting their output (due to the sosreport plugin traceback) is small.

So assessing the severity:
- the bug can happen only when something else is broken (definitely not common, though sosreport is rather called in these situations, right?)
- impact of the bug is quite small - one plugin will skip collecting failed commands outputs "only"

I.e. I would not mark it as a blocker bug, though it is worth considering for a z-stream.


Jake, as you know both sos and kubectl, am I right with the assessment?

Comment 4 Pavel Moravec 2020-07-23 10:30:31 UTC
The problem arises (only) when "kubectl get nodes" returns a line without non-whitsespace characters, i.e. when it returns either empty line, or a line with whitespace only.

After a discussion with a containers guy:
- normal output of "kubectl get nodes" does not contain such lines
- that command output can contain such line when:
  - a config (file or option) is malformed
  - the command is unable to contact apiserver

(the wrong option used was also the cause of similar https://bugzilla.redhat.com/show_bug.cgi?id=1635035 , but that wont happen now, right?)

Both ways mean in 99% that all other kubectl commands will fail the same way. So the harm of not collecting their output (due to the sosreport plugin traceback) is small.

So assessing the severity:
- the bug can happen only when something else is broken (definitely not common, though sosreport is rather called in these situations, right?)
- impact of the bug is quite small - one plugin will skip collecting failed commands outputs "only"

I.e. I would not mark it as a blocker bug, though it is worth considering for a z-stream.


Jake, as you know both sos and kubectl, am I right with the assessment?

Comment 5 Jake Hunsaker 2020-07-23 13:27:09 UTC
> Jake, as you know both sos and kubectl, am I right with the assessment?


Right, in normal operation the `kubectl get nodes` command should never contain empty lines or whitespace only. That being said, even in the cases where a config file is malformed or the apiserver is unavailable - the command should exit non-zero, which would cause us to not parse the output at all.

Comment 6 Pavel Moravec 2020-07-23 13:45:20 UTC
So my understanding is that this might happen only when:

- kubectl returns 0 response code (otherwise sosreport ignores its output), AND
- the output contains empty or blank lines / the output would have to change

That is improbable.

Just to make the sos code robustness, I raised https://github.com/sosreport/sos/pull/2162 , but severity of this BZ is low for me.

Comment 19 errata-xmlrpc 2021-02-02 11:59:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sos bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0333