Bug 1732170 - Unable to use `oc debug` after apps are created
Summary: Unable to use `oc debug` after apps are created
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Auth
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.2.0
Assignee: Standa Laznicka
QA Contact: Chuan Yu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-22 21:06 UTC by Stefanie Forrester
Modified: 2019-08-03 00:54 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-03 00:54:07 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Stefanie Forrester 2019-07-22 21:06:28 UTC
Description of problem: On a newly-provisioned 4.1.6 cluster, I can use `oc debug` to inspect nodes. But after creating 10 apps, `oc debug` fails. It gives me this error:

$ oc debug -t node/ip-10-0-131-59.ec2.internal
Error from server (Forbidden): pods "ip-10-0-131-59ec2internal-debug" is forbidden: unable to validate against any security context constraint: [provider restricted: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used provider restricted: .spec.securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1000740000, 1000749999] spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[0].securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used]


Version-Release number of selected component (if applicable): 4.1.6


How reproducible: every time


Steps to Reproduce:
1. Create a new cluster on https://cloud.stage.openshift.com/.
2. Log in as your uhc user and try `oc debug` on a node. It should work at first (you might need to be a member of the `osd-sre-admins` group).
3. Create 10 sample apps. There will be some evictions after running this:
    for num in {0..10}; do oc new-project test-${num}; oc new-app cakephp-mysql-persistent -n test-${num} ; done
4. Try `oc debug` on a node again. It won't work this time, unless you're kubeadmin.


Actual results: oc debug gives this error:

Error from server (Forbidden): pods "ip-10-0-131-59ec2internal-debug" is forbidden: unable to validate against any security context constraint: [provider restricted: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used provider restricted: .spec.securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1000740000, 1000749999] spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[0].securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used]


Expected results: oc debug should continue to work, even when there are apps on the cluster.


Additional info:

Here's some output from my testing.

[dakini@nibbana ~]$ oc whoami
sedgar@redhat.com

[dakini@nibbana ~]$ oc debug -t node/ip-10-0-131-59.ec2.internal
Starting pod/ip-10-0-131-59ec2internal-debug ...
To use host binaries, run `chroot /host`
sh-4.2# chroot /host
sh-4.4# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p3   31G  5.8G   26G  19% /

# try making some apps
for num in {0..10}; do oc new-project dakinitest20190719-${num}; oc new-app cakephp-mysql-persistent -n dakinitest20190719-${num} ; done

[dakini@nibbana ~]$ for num in {0..10}; do oc get pods -n dakinitest20190719-${num} -o wide; done |grep Evicted
cakephp-mysql-persistent-1-build   0/1     Evicted     0          17m   <none>        ip-10-0-130-35.ec2.internal   <none>           <none>
cakephp-mysql-persistent-1-build   0/1     Evicted     0          17m   <none>        ip-10-0-130-35.ec2.internal   <none>           <none>

[dakini@nibbana ~]$ oc debug -t node/ip-10-0-131-59.ec2.internal
Error from server (Forbidden): pods "ip-10-0-131-59ec2internal-debug" is forbidden: unable to validate against any security context constraint: [provider restricted: .spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used provider restricted: .spec.securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.containers[0].securityContext.securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1000740000, 1000749999] spec.containers[0].securityContext.privileged: Invalid value: true: Privileged containers are not allowed spec.containers[0].securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.containers[0].securityContext.hostPID: Invalid value: true: Host PID is not allowed to be used]

# This shows some output of `oc debug` working as kubeadmin, after the above command failed:

[dakini@nibbana ~]$ oc whoami
kube:admin

[dakini@nibbana ~]$ oc debug -t node/ip-10-0-131-59.ec2.internal
Starting pod/ip-10-0-131-59ec2internal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# df -h |head
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p3   31G  9.4G   22G  31% /

Comment 1 Standa Laznicka 2019-07-25 13:01:47 UTC
Hi Stefanie,

It's kind of hard to understand the issue because it's not clear which permissions you have - osd-sre-admins is not a group that's shipped by default in a cluster.

Are you sure that your user "sedgar@redhat.com" isn't assigned a rolebinding (thus namespace-based), which would cause just one pass of `num=1; oc new-project dakinitest20190719-${num}; oc new-app cakephp-mysql-persistent -n dakinitest20190719-${num}` for you to be unable to issue the `debug` command successfully?

Comment 2 Stefan Schimanski 2019-08-02 10:38:30 UTC
Also we have to know the SCCs before creating the apps and after. Did they change?

Comment 3 Stefanie Forrester 2019-08-03 00:54:07 UTC
Yeah, it is SCC related, sorry for the noise. It's not a bug. My team worked it out here: https://github.com/openshift/managed-cluster-config/pull/58 TL;DR: run `oc debug` from the `default` namespace and it will work.


Note You need to log in before you can comment on or make changes to this bug.