Bug 1635035

Defering to 7.7, too late for 7.6 (where the bug is supposed to be present as well as the code hasnt changed since sos 3.5).


If you can reproduce it on your system, could you please provide me access to it?

Or at least, provide either:

1) Add to /usr/lib/python2.7/site-packages/sos/plugins/kubernetes.py the print statement:

        kn = self.get_command_output('%s get namespaces' % kube_cmd)
        print("kube_cmd='%s', kn=%s" % (kube_cmd, kn))
        knsps = [n.split()[0] for n in kn['output'].splitlines()[1:] if n]

re-run sosreport that will fail and provide stdout output.

2) Or provide output of:

kubectl get namespaces
kubectl --kubeconfig=/etc/origin/master/admin.kubeconfig get namespaces

commands (whose parsing to get namespaces is failing)


(FYI in 02193703, I think the sosreport was killed by OOM killer as it was executing another plugin (logs) - see https://bugzilla.redhat.com/show_bug.cgi?id=1183244)

        if self.check_is_master():
            kube_cmd = "kubectl "
            if path.exists('/etc/origin/master/admin.kubeconfig'):
-                kube_cmd += "--config=/etc/origin/master/admin.kubeconfig"
+                kube_cmd += "--kubeconfig=/etc/origin/master/admin.kubeconfig"


Greetings
Klaas

(In reply to Klaas Demter from comment #3)
>         if self.check_is_master():
>             kube_cmd = "kubectl "
>             if path.exists('/etc/origin/master/admin.kubeconfig'):
> -                kube_cmd += "--config=/etc/origin/master/admin.kubeconfig"
> +                kube_cmd +=
> "--kubeconfig=/etc/origin/master/admin.kubeconfig"
> 
> 
> Greetings
> Klaas

That is upstream commit https://github.com/sosreport/sos/commit/63ad6c2 included in 3.6 we rebase to in RHEL 7.6.

Do you suggest this fixes this BZ?

Yeah, that fixes it for me (Case 02192312)

Hey Pavel,

I can confirm that https://github.com/sosreport/sos/commit/63ad6c2 fixed this issue within my environment as well.

.. and here is the problem that is basically workarounded only in sos 3.6:

python
Python 2.7.15 (default, Sep 21 2018, 23:26:48) 
[GCC 8.1.1 20180712 (Red Hat 8.1.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> kn = {'status': 1, 'output': u'Error: unknown flag: --config\n\n\nExamples:\n'}
>>> kn
{'status': 1, 'output': u'Error: unknown flag: --config\n\n\nExamples:\n'}
>>> [n.split()[0] for n in kn['output'].splitlines()[1:] if n]
[u'Examples:']
>>>

(so far so good, let use longer output snippet, with line containing spaces)

>>> kn={'status': 1, 'output': u'Error: unknown flag: --config\n\n\nExamples:\n  # List all pods in ps output format.\n  kubectl get pods\n  \n  # List all pods in ps output format with more information (such as node name).\n  kubectl get pods -o wide\n  \n'}
>>> [n.split()[0] for n in kn['output'].splitlines()[1:] if n]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: list index out of range
>>>

So the problem is when the output contains '\n  \n' substring (line with spaces only) where:
- this line is nonemtpy, so "if n" is True
- splitting this line returns empty list
- we try to access 1st item of the list

So the code *is* wrong, and it works in 3.6 only b'cos the output does not contain lines with spaces only. Ideally the assignment should be like:

knsps = [n.split()[0] for n in kn['output'].splitlines()[1:] if n and len(n.split())]


So until something gets broken, the code in 3.6 *will* work well. But I will leave the BZ open / to be fixed for the improvement in 

https://github.com/sosreport/sos/pull/1442

For verification:

1) mimic you are Kubernetes master and fake kubectl command to return the problematic output:

mkdir -p /etc/origin/master/

echo "echo 'nonempty line'; echo '   '; echo 'another nonempty'" > /usr/bin/kubectl

chmod a+x /usr/bin/kubectl

2) run sosreport:

sosreport -o kubernetes --batch --build

3) check if kubernetes plugin does not raise above exception from Description

posted to upstream

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2295