Bug 1859888

Summary: IndexError from kubernetes plugin when parsing nodes
Product: Red Hat Enterprise Linux 8 Reporter: Miroslav Hradílek <mhradile>
Component: sosAssignee: Pavel Moravec <pmoravec>
Status: CLOSED ERRATA QA Contact: Miroslav Hradílek <mhradile>
Severity: medium Docs Contact:
Priority: medium    
Version: 8.3CC: agk, bmr, fkrska, jjansky, plambri, pmoravec, sbradley
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 8.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: sos-3.9.1-5.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1859885 Environment:
Last Closed: 2020-11-04 01:58:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1859885    
Bug Blocks:    

Description Miroslav Hradílek 2020-07-23 09:31:09 UTC 
sos-3.9.1-4.el8.noarch (unreleased) is also affected.

+++ This bug was initially created as a clone of Bug #1859885 +++

Plugin kubernetes in sos 3.9 started parsing output of "kubectl get nodes" in order to run "kubectl describe node" for each parsed node. The problem is that sosreport does not expect empty lines in "kubectl get nodes" output and if such output should occur plugin crashes with a traceback.

There was similar bug 1635035 in the past concerning parsing namespaces from "kubectl get namespaces" with the same outcome. Although i found about this by crafted artificial output instead of real "kubectl" output i believe this could reappear as a sort of regression in the future if somebody happens to hit error from "kubectl get nodes" when using sosreport.

Not sure if i can call this regression if it is in a different part of code but behavior is the same.

Setup:
======
mkdir -p /etc/origin/master/
echo "echo 'nonempty line'; echo '   '; echo 'another nonempty'" > /usr/bin/kubectl
chmod +x /usr/bin/kubectl
mkdir -p /etc/origin/node/pods/
touch /etc/origin/node/pods/master-config.yaml

Old:
====
# rpm -q sos
sos-3.8-8.el7_8.noarch

# sosreport -vvv -o kubernetes --batch
# tar -tf /var/tmp/sosreport-sheep-2-2020-07-23-ruybyqj.tar.xz
sosreport-sheep-2-2020-07-23-ruybyqj/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/journalctl_--no-pager_--unit_kubelet
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-apiserver
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-proxy
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-scheduler
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-controller-manager
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/kubectl_version
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/kubectl_config_view
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/kubectl_get_namespaces
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/kubectl_get_nodes
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/kubectl_get_projects
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/kubectl_get_pvs
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/kubectl_get_--raw_.metrics
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/deployments/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/deployments/kubectl_get_--all-namespaces_true_deployments
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/ingresses/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/ingresses/kubectl_get_--all-namespaces_true_ingresses
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/limitranges/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/limitranges/kubectl_get_--all-namespaces_true_limitranges
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/pods/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/pods/kubectl_get_--all-namespaces_true_pods
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/policies/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/policies/kubectl_get_--all-namespaces_true_policies
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/pvc/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/pvc/kubectl_get_--all-namespaces_true_pvc
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/rc/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/rc/kubectl_get_--all-namespaces_true_rc
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/resourcequotas/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/resourcequotas/kubectl_get_--all-namespaces_true_resourcequotas
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/routes/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/routes/kubectl_get_--all-namespaces_true_routes
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/services/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_commands/kubernetes/services/kubectl_get_--all-namespaces_true_services
sosreport-sheep-2-2020-07-23-ruybyqj/sos_logs/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_logs/sos.log
sosreport-sheep-2-2020-07-23-ruybyqj/sos_logs/ui.log
sosreport-sheep-2-2020-07-23-ruybyqj/sos_reports/
sosreport-sheep-2-2020-07-23-ruybyqj/sos_reports/sos.txt
sosreport-sheep-2-2020-07-23-ruybyqj/sos_reports/sos.json
sosreport-sheep-2-2020-07-23-ruybyqj/sos_reports/sos.html
sosreport-sheep-2-2020-07-23-ruybyqj/environment
sosreport-sheep-2-2020-07-23-ruybyqj/version.txt


New (unreleased):
====
# rpm -q sos
sos-3.9-4.el7_9.noarch
# sosreport -vvv -o kubernetes --batch
. . .
# tar -tf /var/tmp/sosreport-sheep-2-2020-07-23-trcwqgn.tar.xz
sosreport-sheep-2-2020-07-23-trcwqgn/
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/kubectl_get_namespaces
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/kubectl_get_nodes
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/journalctl_--no-pager_--unit_kubelet
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-apiserver
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-proxy
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-scheduler
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/journalctl_--no-pager_--unit_kube-controller-manager
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/kubectl_version
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/kubectl_config_view
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/kubectl_get_namespaces.1
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/kubectl_get_projects
sosreport-sheep-2-2020-07-23-trcwqgn/sos_commands/kubernetes/kubectl_get_pvs
sosreport-sheep-2-2020-07-23-trcwqgn/sos_logs/
sosreport-sheep-2-2020-07-23-trcwqgn/sos_logs/kubernetes-plugin-errors.txt
sosreport-sheep-2-2020-07-23-trcwqgn/sos_logs/sos.log
sosreport-sheep-2-2020-07-23-trcwqgn/sos_logs/ui.log
sosreport-sheep-2-2020-07-23-trcwqgn/sos_reports/
sosreport-sheep-2-2020-07-23-trcwqgn/sos_reports/sos.txt
sosreport-sheep-2-2020-07-23-trcwqgn/sos_reports/sos.json
sosreport-sheep-2-2020-07-23-trcwqgn/sos_reports/sos.html
sosreport-sheep-2-2020-07-23-trcwqgn/version.txt
# tar -xf /var/tmp/sosreport-sheep-2-2020-07-23-trcwqgn.tar.xz
# cat sosreport-sheep-2-2020-07-23-trcwqgn/sos_logs/kubernetes-plugin-errors.txt
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sos/sosreport.py", line 981, in setup
    plug.setup()
  File "/usr/lib/python2.7/site-packages/sos/plugins/kubernetes.py", line 103, in setup
    node = line.split()[0]
IndexError: list index out of range
Comment 6 errata-xmlrpc 2020-11-04 01:58:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sos bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:4534