Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2120617

Summary: sos collect doesn't work with OCP4.12
Product: Red Hat Enterprise Linux 8 Reporter: Nadia Pinaeva <npinaeva>
Component: sosAssignee: Jake Hunsaker <jhunsaker>
Status: CLOSED ERRATA QA Contact: Miroslav Hradílek <mhradile>
Severity: high Docs Contact:
Priority: unspecified    
Version: ---CC: agk, bmr, cww, erich, jhunsaker, knarra, lmiksik, mhradile, plambri, pmoravec, sbradley, slaznick, theute
Target Milestone: rcKeywords: OtherQA, Reopened, Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: sos-4.3-4.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2130209 2130922 (view as bug list) Environment:
Last Closed: 2022-11-08 10:50:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2130209, 2130922    

Description Nadia Pinaeva 2022-08-23 11:46:21 UTC 
Description of problem:
1. node roles parsing can't handle miltiple roles
# sos collect --no-local -vvv

[sos_collector:get_nodes_from_cluster] Node list: []
[sos_collector:reduce_node_list] Node list reduced to []

=>  Use role=worker, since it can be parsed properly.

2. No way to figure out the error, /var/tmp/sos.wxnmrktg doesn't exist
# sos collect --no-local -c ocp.role=worker -vvv

WARNING: Unable to read configuration file /etc/sos/sos.conf : No such file or directory
[sos_collector:__init__] Executing /usr/local/bin/sos collect --no-local -c ocp.role=worker -vvv
[sos_collector:__init__] Found cluster profiles: dict_keys(['jbon', 'kubernetes', 'ocp', 'rhosp', 'ovirt', 'rhhi_virt', 'rhv', 'pacemaker', 'satellite'])

sos-collector (version 4.4)
...
[sos_collector:configure_sos_cmd] Initial sos cmd set to sosreport --batch -c auto
[sos_collector:prep] password not specified, assuming SSH keys
sos-collector ASSUMES that SSH keys are installed on all nodes unless the
--password option is provided.

Optionally, please enter the case id that you are generating this report for []: 

[localhost:run_command] Running command hostname
[thinkpad:_get_hostname] Hostname set to thinkpad
[localhost:determine_host_policy] using local policy Ubuntu
[ocp] oc base command set to oc
[thinkpad:run_command] Running command oc whoami
[sos_collector:determine_cluster] Installation matches ocp, checking for layered profiles
Cluster type set to OpenShift Container Platform v4
[thinkpad:run_command] Running command oc auth can-i '*' '*'
[ocp] Creating new temporary project 'sos-collect-tmp'
[thinkpad:run_command] Running command oc new-project sos-collect-tmp
[thinkpad:run_command] Running command oc get nodes -o wide
[sos_collector:get_nodes_from_cluster] Node list: ['ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq', 'ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw', 'ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw']
[sos_collector:reduce_node_list] Node list reduced to ['ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq', 'ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw', 'ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw']

The following is a list of nodes to collect from:
	ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq
	ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw
	ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw


Press ENTER to continue with these nodes, or press CTRL-C to quit


[archive:TarFileArchive] initialised empty FileCacheArchive at '/var/tmp/sos.wxnmrktg/sos-collector-2022-08-23-zkgib'
[archive:TarFileArchive] created directory at 'sos_logs' in FileCacheArchive '/var/tmp/sos.wxnmrktg/sos-collector-2022-08-23-zkgib'

Connecting to nodes...
[ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq:_connect] Starting sos collector container 'ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq-sos-collector'
[ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw:_connect] Starting sos collector container 'ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw-sos-collector'
[ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw:_connect] Starting sos collector container 'ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw-sos-collector'
[ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq:_connect] Unable to deploy sos collect pod
[ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw:_connect] Unable to deploy sos collect pod
[ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq:_connect] Debug pod deployment failed: 
[ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw:_connect] Unable to deploy sos collect pod
[ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw:_connect] Debug pod deployment failed: 
[ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw:_connect] Debug pod deployment failed: 
[ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw:read_file] Reading file /etc/os-release
[ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq:read_file] Reading file /etc/os-release
[ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw:read_file] Reading file /etc/os-release
[ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw:read_file] Reading file /etc/os-release
[ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq:read_file] Reading file /etc/os-release
[ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw:read_file] Reading file /etc/os-release
[ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw:run_command] Running command cat /etc/os-release
[ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq:run_command] Running command cat /etc/os-release
[ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw:run_command] Running command cat /etc/os-release
[ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq:_read_file] Error reading /etc/os-release: [' pods "ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq-sos-collector" not found\r\n']
[ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw:_read_file] Error reading /etc/os-release: [' pods "ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw-sos-collector" not found\r\n']
[ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw:_read_file] Error reading /etc/os-release: [' pods "ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw-sos-collector" not found\r\n']
[ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq:determine_host_policy] loaded policy Ubuntu for host
[ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw:determine_host_policy] loaded policy Ubuntu for host
[ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw:determine_host_policy] loaded policy Ubuntu for host
[ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq:_disconnect] Calling delete on pod 'ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq-sos-collector' failed: {'status': 1, 'output': '', 'truncated': False}
[ci-ln-yt40m6t-72292-8xrb4-worker-a-hg2nq:disconnect] Unable to successfully disconnect, see log for more details
[ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw:_disconnect] Calling delete on pod 'ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw-sos-collector' failed: {'status': 1, 'output': '', 'truncated': False}
[ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw:disconnect] Unable to successfully disconnect, see log for more details
[ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw:_disconnect] Calling delete on pod 'ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw-sos-collector' failed: {'status': 1, 'output': '', 'truncated': False}
[ci-ln-yt40m6t-72292-8xrb4-worker-c-rfddw:disconnect] Unable to successfully disconnect, see log for more details
[thinkpad:run_command] Running command oc delete project sos-collect-tmp
[thinkpad:run_command] Running command oc wait namespace/sos-collect-tmp --for=delete --timeout=30s
[thinkpad:run_command] Running command oc project default
No nodes connected. Aborting...

3. After a manual tweak to see the error

diff --git a/sos/utilities.py b/sos/utilities.py
index 2046c8fd..c83e693a 100644
--- a/sos/utilities.py
+++ b/sos/utilities.py
@@ -156,7 +156,7 @@ def sos_get_command_output(command, timeout=TIMEOUT_DEFAULT, stderr=False,
             proc.terminate()
             raise SoSTimeoutError
         time.sleep(0.01)
-
+    stderr = True
     cmd_env = os.environ.copy()
     # ensure consistent locale for collected command output
     cmd_env['LC_ALL'] = 'C.UTF-8'


we can see podSecurity failure
# sos collect --no-local -c ocp.role=worker -vvv

[ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw:_connect] Debug pod deployment failed: Error from server (Forbidden): error when creating "/var/tmp/sos.2wd3loxw/tmpsgo1_0ih": pods "ci-ln-yt40m6t-72292-8xrb4-worker-b-lrjnw-sos-collector" is forbidden: violates PodSecurity "restricted:v1.24": host namespaces (hostNetwork=true, hostPID=true, hostIPC=true), privileged (container "sos-collector-tmp" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "sos-collector-tmp" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "sos-collector-tmp" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volumes "host", "run", "varlog", "machine-id" use restricted volume type "hostPath"), runAsNonRoot != true (pod or container "sos-collector-tmp" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "sos-collector-tmp" must not set runAsUser=0), seccompProfile (pod or container "sos-collector-tmp" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")


Version-Release number of selected component (if applicable):
Used sos collect 4.4 from upstream (all other sos collect version should fail the same way, knowing the problem)

How reproducible:
Always

Additional info:
Works with older versions of Openshift
Comment 1 Nadia Pinaeva 2022-09-07 12:34:17 UTC 
https://access.redhat.com/articles/6973044
here are some instructions on fixing the issue
Comment 2 Standa Laznicka 2022-09-08 12:05:00 UTC 
Those instructions are not at all related to the problem. You seem to be running a privileged pod - `(container "sos-collector-tmp" must not set securityContext.privileged=true)`, but you are trying to run it in a namespace that is not privileged.

Please follow the steps in https://docs.openshift.com/container-platform/4.11/authentication/understanding-and-managing-pod-security-admission.html#security-context-constraints-psa-opting_understanding-and-managing-pod-security-admission to exclude your namespace from automatic label synchronization and then apply the steps in https://kubernetes.io/docs/concepts/security/pod-security-admission/#pod-security-admission-labels-for-namespaces to get a privileged namespace that you can run your pods in.
Comment 3 Eric Rich 2022-09-08 12:13:38 UTC 
Re-Opening this bug for SOS! 

In short we need to run the following: 

> oc label namespace <namespace> security.openshift.io/scc.podSecurityLabelSync=false
> oc label namespace <namespace> pod-security.kubernetes.io/enforce=privileged

After creating the namespace in https://github.com/sosreport/sos/blob/main/sos/collector/clusters/ocp.py#L140-L143 

Or just create the namespace with these labels (IE: oc create -f namespace.yaml).
Comment 5 Jake Hunsaker 2022-09-08 14:09:34 UTC 
Will those labels cause any issues on older OCP clusters? Or can we just add them to the stock namespace config? If we need to version-gate the use of those labels that's fine too, but if they can be added to older clusters without causing problems we can just make this an across the board change.
Comment 6 Eric Rich 2022-09-12 13:48:50 UTC 
must-gather its self is going to have the same issues in 4.12 too: https://issues.redhat.com/browse/OCPBUGS-1085
Comment 8 Pavel Moravec 2022-09-23 20:47:00 UTC 
Not knowing OCP much, but do I get it right a change like:

https://github.com/sosreport/sos/commit/765f5f283bdb4747b0069f2f5d3381134b4b9a95

is required but additionally for two other commands / labels adding?
Comment 10 Jake Hunsaker 2022-09-26 14:34:25 UTC 
No additional labels needed, just what is in that commit.
Comment 11 Pavel Moravec 2022-09-27 08:10:15 UTC 
(In reply to Jake Hunsaker from comment #10)
> No additional labels needed, just what is in that commit.

Ah I was blind, I mis-matched the requirements.

So just the https://github.com/sosreport/sos/commit/765f5f283bdb4747b0069f2f5d3381134b4b9a95 commit is required to backport to 8.7 (and implicitly to 9.1, 8.8and 9.2).

Waiting on cww to grant exception+ .
Comment 19 RamaKasturi 2022-09-30 11:37:14 UTC 
Tested with the scratch build provided above in comment 17 and i see that it works fine. Detailed verification steps are present in the jira here, thanks !!

https://issues.redhat.com/browse/RHELPLAN-131968
Comment 20 RamaKasturi 2022-09-30 11:38:58 UTC 
(In reply to RamaKasturi from comment #19)
> Tested with the scratch build provided above in comment 17 and i see that it
> works fine. Detailed verification steps are present in the jira here, thanks
> !!
> 
> https://issues.redhat.com/browse/RHELPLAN-131968

My bad s/scratch build/official build
Comment 25 errata-xmlrpc 2022-11-08 10:50:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sos bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7732