Description of problem: During our ci job , we find many cases failed with create normal pods with errors; error when creating "pod.yaml": pods "kubernetes-metadata-volume-example" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] Version-Release number of selected component (if applicable): 4.8.0-0.nightly-2021-04-09-222447 How reproducible: always Steps to Reproduce: 1. Create a new project; 2. Create a pod or apps Actual results: 2. Pod created with error: error when creating "pod.yaml": pods "kubernetes-metadata-volume-example" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] Expected results: No such issue. Additional info:
The must-gather from the ci job is : http://10.73.131.57:9000/openshift-must-gather/must-gather-12698-056432821.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=openshift%2F20210411%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210411T225212Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=0726452f7e85f63b0a4d977d2c8b64d7b9fa5471c8cfd34b89e9c69a2e4db1d9
I've checked some failed cases: For example 1 , we could see logs from ci job: 15:48:58 INFO> Shell Commands: oc new-project bbi30 --kubeconfig=/home/jenkins/ws/workspace/ocp-common/Runner/workdir/ocp4_testuser-36.kubeconfig Now using project "bbi30" on server "https://api.qeci-18869.qe.azure.devcluster.openshift.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname 15:48:59 INFO> Exit Status: 0 15:49:00 INFO> oc get projects bbi30 --output=yaml --kubeconfig=/home/jenkins/ws/workspace/ocp-common/Runner/workdir/ocp4_testuser-36.kubeconfig 15:49:00 INFO> After 1 iterations and 1 seconds: apiVersion: project.openshift.io/v1 kind: Project metadata: annotations: openshift.io/description: "" openshift.io/display-name: "" openshift.io/requester: testuser-36 creationTimestamp: "2021-04-11T15:48:59Z" labels: kubernetes.io/metadata.name: bbi30 name: bbi30 resourceVersion: "108949" uid: ee0e4199-776e-4209-b726-6db65240b865 spec: finalizers: - kubernetes status: phase: Active GivenI obtain test data file "pods/hello-pod.json" ==>@ features/step_definitions/file.rb:1 WhenI run the :create client command with: ==>@ features/step_definitions/cli.rb:13 f hello-pod.json 15:49:00 INFO> Shell Commands: oc create -f hello-pod.json --kubeconfig=/home/jenkins/ws/workspace/ocp-common/Runner/workdir/ocp4_testuser-36.kubeconfig STDERR: Error from server (Forbidden): error when creating "hello-pod.json": pods "hello-openshift" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 15:49:01 INFO> Exit Status: 1 And then check from the must-gather see logs like : [yinzhou@dhcp-141-223 must-gather-12698-056432821]$ ack bbi30 must-gather.local.4464082293993668349/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-e7eb2a1576eb129e402ea6f709926082d10104bb222c609821aea9174e370029/namespaces/openshift-kube-controller-manager/pods/kube-controller-manager-qeci-18869-cz6dg-master-1/kube-controller-manager/kube-controller-manager/logs/current.log 11755:2021-04-11T15:48:59.335331006Z E0411 15:48:59.335280 1 publisher.go:168] syncing "bbi30" failed: configmaps "kube-root-ca.crt" already exists 11756:2021-04-11T15:49:08.873090476Z E0411 15:49:08.873032 1 tokens_controller.go:269] error synchronizing serviceaccount bbi30/builder: secrets "builder-token-xwcdn" is forbidden: unable to create new content in namespace bbi30 because it is being terminated 11757:2021-04-11T15:49:08.903969718Z E0411 15:49:08.903893 1 tokens_controller.go:269] error synchronizing serviceaccount bbi30/builder: secrets "builder-token-7flnd" is forbidden: unable to create new content in namespace bbi30 because it is being terminated 11758:2021-04-11T15:49:08.919997947Z E0411 15:49:08.919951 1 tokens_controller.go:269] error synchronizing serviceaccount bbi30/default: secrets "default-token-mg58w" is forbidden: unable to create new content in namespace bbi30 because it is being terminated 11759:2021-04-11T15:49:08.927983062Z E0411 15:49:08.927932 1 tokens_controller.go:269] error synchronizing serviceaccount bbi30/deployer: secrets "deployer-token-zggvn" is forbidden: unable to create new content in namespace bbi30 because it is being terminated 11760:2021-04-11T15:49:08.951394397Z E0411 15:49:08.951343 1 tokens_controller.go:269] error synchronizing serviceaccount bbi30/deployer: secrets "deployer-token-mh9s7" is forbidden: unable to create new content in namespace bbi30 because it is being terminated 11761:2021-04-11T15:49:09.093427929Z E0411 15:49:09.093320 1 tokens_controller.go:269] error synchronizing serviceaccount bbi30/deployer: secrets "deployer-token-lvn22" is forbidden: unable to create new content in namespace bbi30 because it is being terminated 11772:2021-04-11T15:49:14.245453667Z I0411 15:49:14.245404 1 namespace_controller.go:185] Namespace has been deleted bbi30
Another example : logs from ci job: 15:48:37 INFO> Shell Commands: oc new-project ch9ks --kubeconfig=/home/jenkins/ws/workspace/ocp-common/Runner/workdir/ocp4_testuser-36.kubeconfig Now using project "ch9ks" on server "https://api.qeci-18869.qe.azure.devcluster.openshift.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname 15:48:38 INFO> Exit Status: 0 15:48:39 INFO> oc get projects ch9ks --output=yaml --kubeconfig=/home/jenkins/ws/workspace/ocp-common/Runner/workdir/ocp4_testuser-36.kubeconfig 15:48:39 INFO> After 1 iterations and 1 seconds: apiVersion: project.openshift.io/v1 kind: Project metadata: annotations: openshift.io/description: "" openshift.io/display-name: "" openshift.io/requester: testuser-36 creationTimestamp: "2021-04-11T15:48:37Z" labels: kubernetes.io/metadata.name: ch9ks name: ch9ks resourceVersion: "108687" uid: e01997d1-5061-464c-a423-160584941f04 spec: finalizers: - kubernetes status: phase: Active GivenI obtain test data file "templates/ocp16295/pod.yaml" ==>@ features/step_definitions/file.rb:1 WhenI run the :create client command with: ==>@ features/step_definitions/cli.rb:13 f pod.yaml 15:48:39 INFO> Shell Commands: oc create -f pod.yaml --kubeconfig=/home/jenkins/ws/workspace/ocp-common/Runner/workdir/ocp4_testuser-36.kubeconfig STDERR: Error from server (Forbidden): error when creating "pod.yaml": pods "kubernetes-metadata-volume-example" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 15:48:40 INFO> Exit Status: 1 Logs from must-gather: [yinzhou@dhcp-141-223 must-gather-12698-056432821]$ ack ch9ks must-gather.local.4464082293993668349/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-e7eb2a1576eb129e402ea6f709926082d10104bb222c609821aea9174e370029/namespaces/openshift-kube-controller-manager/pods/kube-controller-manager-qeci-18869-cz6dg-master-1/kube-controller-manager/kube-controller-manager/logs/current.log 11746:2021-04-11T15:48:47.158879324Z I0411 15:48:47.158780 1 garbagecollector.go:471] "Processing object" object="ch9ks/builder-dockercfg-pxw2m" objectUID=b33f1c65-576e-42ab-9f6a-7790039c2dfd kind="Secret" virtual=false 11747:2021-04-11T15:48:47.168144355Z I0411 15:48:47.168098 1 garbagecollector.go:471] "Processing object" object="ch9ks/default-dockercfg-hrplc" objectUID=ad110c26-d781-4ec0-857f-883486b95fb0 kind="Secret" virtual=false 11748:2021-04-11T15:48:47.171457103Z I0411 15:48:47.171410 1 garbagecollector.go:580] "Deleting object" object="ch9ks/builder-dockercfg-pxw2m" objectUID=b33f1c65-576e-42ab-9f6a-7790039c2dfd kind="Secret" propagationPolicy=Background 11749:2021-04-11T15:48:47.172335215Z I0411 15:48:47.172293 1 garbagecollector.go:580] "Deleting object" object="ch9ks/default-dockercfg-hrplc" objectUID=ad110c26-d781-4ec0-857f-883486b95fb0 kind="Secret" propagationPolicy=Background 11750:2021-04-11T15:48:47.182276956Z I0411 15:48:47.182229 1 garbagecollector.go:471] "Processing object" object="ch9ks/deployer-dockercfg-p9fjb" objectUID=e66a2757-b4c3-48d8-a2f7-f7a8fd97d2a5 kind="Secret" virtual=false 11751:2021-04-11T15:48:47.188572546Z I0411 15:48:47.188537 1 garbagecollector.go:580] "Deleting object" object="ch9ks/deployer-dockercfg-p9fjb" objectUID=e66a2757-b4c3-48d8-a2f7-f7a8fd97d2a5 kind="Secret" propagationPolicy=Background 11753:2021-04-11T15:48:52.905676217Z I0411 15:48:52.905625 1 namespace_controller.go:185] Namespace has been deleted ch9ks
Example 3 : Logs from ci job: 15:36:45 INFO> Shell Commands: oc new-project wm4wu --kubeconfig=/home/jenkins/ws/workspace/ocp-common/Runner/workdir/ocp4_testuser-28.kubeconfig Now using project "wm4wu" on server "https://api.qeci-18869.qe.azure.devcluster.openshift.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname 15:36:46 INFO> Exit Status: 0 15:36:47 INFO> oc get projects wm4wu --output=yaml --kubeconfig=/home/jenkins/ws/workspace/ocp-common/Runner/workdir/ocp4_testuser-28.kubeconfig 15:36:47 INFO> After 1 iterations and 1 seconds: apiVersion: project.openshift.io/v1 kind: Project metadata: annotations: openshift.io/description: "" openshift.io/display-name: "" openshift.io/requester: testuser-28 creationTimestamp: "2021-04-11T15:36:45Z" labels: kubernetes.io/metadata.name: wm4wu name: wm4wu resourceVersion: "101308" uid: e53eaac5-f039-4fd5-8d19-470873d3b522 spec: finalizers: - kubernetes status: phase: Active WhenI run the :new_app client command with: ==>@ features/step_definitions/cli.rb:13 app_repo openshift/ruby~https://github.com/openshift/ruby-hello-world.git 15:36:47 INFO> Shell Commands: oc new-app openshift/ruby\~https://github.com/openshift/ruby-hello-world.git --kubeconfig=/home/jenkins/ws/workspace/ocp-common/Runner/workdir/ocp4_testuser-28.kubeconfig --> Found image 6cef4d6 (11 days old) in image stream "openshift/ruby" under tag "2.7-ubi8" for "openshift/ruby" Ruby 2.7 -------- Ruby 2.7 available as container is a base platform for building and running various Ruby 2.7 applications and frameworks. Ruby is the interpreted scripting language for quick and easy object-oriented programming. It has many features to process text files and to do system management tasks (as in Perl). It is simple, straight-forward, and extensible. Tags: builder, ruby, ruby27, ruby-27 * A source build using source code from https://github.com/openshift/ruby-hello-world.git will be created * The resulting image will be pushed to image stream tag "ruby-hello-world:latest" * Use 'oc start-build' to trigger a new build --> Creating resources ... imagestream.image.openshift.io "ruby-hello-world" created buildconfig.build.openshift.io "ruby-hello-world" created deployment.apps "ruby-hello-world" created service "ruby-hello-world" created --> Success Build scheduled, use 'oc logs -f buildconfig/ruby-hello-world' to track its progress. Application is not exposed. You can expose services to the outside world by executing one or more of the commands below: 'oc expose service/ruby-hello-world' Run 'oc status' to view your app. 15:36:49 INFO> Exit Status: 0 Thenthe step should succeed ==>@ features/step_definitions/common.rb:4 GivenI obtain test data file "pods/pod_with_two_containers.json" ==>@ features/step_definitions/file.rb:1 WhenI run the :create client command with: ==>@ features/step_definitions/cli.rb:13 f pod_with_two_containers.json 15:36:49 INFO> Shell Commands: oc create -f pod_with_two_containers.json --kubeconfig=/home/jenkins/ws/workspace/ocp-common/Runner/workdir/ocp4_testuser-28.kubeconfig STDERR: Error from server (Forbidden): error when creating "pod_with_two_containers.json": pods "doublecontainers" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 15:36:50 INFO> Exit Status: 1 Logs from must-gather; must-gather.local.4464082293993668349/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-e7eb2a1576eb129e402ea6f709926082d10104bb222c609821aea9174e370029/namespaces/openshift-kube-controller-manager/pods/kube-controller-manager-qeci-18869-cz6dg-master-1/kube-controller-manager/kube-controller-manager/logs/current.log 10542:2021-04-11T15:36:48.054805897Z I0411 15:36:48.054742 1 replica_set.go:559] "Too few replicas" replicaSet="wm4wu/ruby-hello-world-85858f65d5" need=1 creating=1 10543:2021-04-11T15:36:48.055037301Z I0411 15:36:48.054980 1 event.go:291] "Event occurred" object="wm4wu/ruby-hello-world" kind="Deployment" apiVersion="apps/v1" type="Normal" reason="ScalingReplicaSet" message="Scaled up replica set ruby-hello-world-85858f65d5 to 1" 10544:2021-04-11T15:36:48.069866541Z I0411 15:36:48.069829 1 replica_set.go:584] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet wm4wu/ruby-hello-world-85858f65d5 10545:2021-04-11T15:36:48.069929242Z I0411 15:36:48.069900 1 event.go:291] "Event occurred" object="wm4wu/ruby-hello-world-85858f65d5" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message="Error creating: pods \"ruby-hello-world-85858f65d5-\" is forbidden: unable to validate against any security context constraint: [provider \"node-exporter\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]" 10546:2021-04-11T15:36:48.071471067Z I0411 15:36:48.071429 1 deployment_controller.go:490] "Error syncing deployment" deployment="wm4wu/ruby-hello-world" err="Operation cannot be fulfilled on deployments.apps \"ruby-hello-world\": the object has been modified; please apply your changes to the latest version and try again" 10547:2021-04-11T15:36:48.089349757Z E0411 15:36:48.089303 1 replica_set.go:532] sync "wm4wu/ruby-hello-world-85858f65d5" failed with pods "ruby-hello-world-85858f65d5-" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 10548:2021-04-11T15:36:48.089439158Z I0411 15:36:48.089386 1 replica_set.go:559] "Too few replicas" replicaSet="wm4wu/ruby-hello-world-85858f65d5" need=1 creating=1 10549:2021-04-11T15:36:48.095810661Z I0411 15:36:48.095735 1 replica_set.go:584] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet wm4wu/ruby-hello-world-85858f65d5 10550:2021-04-11T15:36:48.095841862Z E0411 15:36:48.095808 1 replica_set.go:532] sync "wm4wu/ruby-hello-world-85858f65d5" failed with pods "ruby-hello-world-85858f65d5-" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 10551:2021-04-11T15:36:48.095923163Z I0411 15:36:48.095884 1 replica_set.go:559] "Too few replicas" replicaSet="wm4wu/ruby-hello-world-85858f65d5" need=1 creating=1 10552:2021-04-11T15:36:48.095966864Z I0411 15:36:48.095918 1 event.go:291] "Event occurred" object="wm4wu/ruby-hello-world-85858f65d5" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message="Error creating: pods \"ruby-hello-world-85858f65d5-\" is forbidden: unable to validate against any security context constraint: [provider \"node-exporter\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]" 10553:2021-04-11T15:36:48.096673075Z I0411 15:36:48.096632 1 deployment_controller.go:490] "Error syncing deployment" deployment="wm4wu/ruby-hello-world" err="Operation cannot be fulfilled on deployments.apps \"ruby-hello-world\": the object has been modified; please apply your changes to the latest version and try again" 10554:2021-04-11T15:36:48.112565632Z I0411 15:36:48.112504 1 replica_set.go:584] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet wm4wu/ruby-hello-world-85858f65d5 10555:2021-04-11T15:36:48.112616133Z E0411 15:36:48.112566 1 replica_set.go:532] sync "wm4wu/ruby-hello-world-85858f65d5" failed with pods "ruby-hello-world-85858f65d5-" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 10556:2021-04-11T15:36:48.112685734Z I0411 15:36:48.112654 1 replica_set.go:559] "Too few replicas" replicaSet="wm4wu/ruby-hello-world-85858f65d5" need=1 creating=1 10557:2021-04-11T15:36:48.112839637Z I0411 15:36:48.112812 1 event.go:291] "Event occurred" object="wm4wu/ruby-hello-world-85858f65d5" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message="Error creating: pods \"ruby-hello-world-85858f65d5-\" is forbidden: unable to validate against any security context constraint: [provider \"node-exporter\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]" 10558:2021-04-11T15:36:48.117008704Z I0411 15:36:48.116962 1 deployment_controller.go:490] "Error syncing deployment" deployment="wm4wu/ruby-hello-world" err="Operation cannot be fulfilled on deployments.apps \"ruby-hello-world\": the object has been modified; please apply your changes to the latest version and try again" 10559:2021-04-11T15:36:48.134365286Z I0411 15:36:48.134308 1 replica_set.go:584] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet wm4wu/ruby-hello-world-85858f65d5 10560:2021-04-11T15:36:48.134469787Z E0411 15:36:48.134377 1 replica_set.go:532] sync "wm4wu/ruby-hello-world-85858f65d5" failed with pods "ruby-hello-world-85858f65d5-" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 10561:2021-04-11T15:36:48.134526488Z I0411 15:36:48.134493 1 replica_set.go:559] "Too few replicas" replicaSet="wm4wu/ruby-hello-world-85858f65d5" need=1 creating=1 10562:2021-04-11T15:36:48.135335801Z I0411 15:36:48.135285 1 event.go:291] "Event occurred" object="wm4wu/ruby-hello-world-85858f65d5" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message="Error creating: pods \"ruby-hello-world-85858f65d5-\" is forbidden: unable to validate against any security context constraint: [provider \"node-exporter\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]" 10563:2021-04-11T15:36:48.145859672Z I0411 15:36:48.145691 1 replica_set.go:584] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet wm4wu/ruby-hello-world-85858f65d5 10564:2021-04-11T15:36:48.145859672Z E0411 15:36:48.145753 1 replica_set.go:532] sync "wm4wu/ruby-hello-world-85858f65d5" failed with pods "ruby-hello-world-85858f65d5-" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 10565:2021-04-11T15:36:48.145859672Z I0411 15:36:48.145809 1 event.go:291] "Event occurred" object="wm4wu/ruby-hello-world-85858f65d5" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message="Error creating: pods \"ruby-hello-world-85858f65d5-\" is forbidden: unable to validate against any security context constraint: [provider \"node-exporter\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]" 10567:2021-04-11T15:36:48.228802315Z I0411 15:36:48.226821 1 replica_set.go:559] "Too few replicas" replicaSet="wm4wu/ruby-hello-world-85858f65d5" need=1 creating=1 10568:2021-04-11T15:36:48.236642442Z I0411 15:36:48.236546 1 replica_set.go:584] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet wm4wu/ruby-hello-world-85858f65d5 10569:2021-04-11T15:36:48.236642442Z E0411 15:36:48.236613 1 replica_set.go:532] sync "wm4wu/ruby-hello-world-85858f65d5" failed with pods "ruby-hello-world-85858f65d5-" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 10570:2021-04-11T15:36:48.237176950Z I0411 15:36:48.237140 1 event.go:291] "Event occurred" object="wm4wu/ruby-hello-world-85858f65d5" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message="Error creating: pods \"ruby-hello-world-85858f65d5-\" is forbidden: unable to validate against any security context constraint: [provider \"node-exporter\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]" 10571:2021-04-11T15:36:48.397882253Z I0411 15:36:48.397828 1 replica_set.go:559] "Too few replicas" replicaSet="wm4wu/ruby-hello-world-85858f65d5" need=1 creating=1 10572:2021-04-11T15:36:48.404742664Z I0411 15:36:48.404691 1 replica_set.go:584] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet wm4wu/ruby-hello-world-85858f65d5 10573:2021-04-11T15:36:48.404797065Z I0411 15:36:48.404732 1 event.go:291] "Event occurred" object="wm4wu/ruby-hello-world-85858f65d5" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message="Error creating: pods \"ruby-hello-world-85858f65d5-\" is forbidden: unable to validate against any security context constraint: [provider \"node-exporter\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]" 10574:2021-04-11T15:36:48.404797065Z E0411 15:36:48.404779 1 replica_set.go:532] sync "wm4wu/ruby-hello-world-85858f65d5" failed with pods "ruby-hello-world-85858f65d5-" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 10575:2021-04-11T15:36:48.725914665Z I0411 15:36:48.725853 1 replica_set.go:559] "Too few replicas" replicaSet="wm4wu/ruby-hello-world-85858f65d5" need=1 creating=1 10576:2021-04-11T15:36:48.732550773Z I0411 15:36:48.732509 1 replica_set.go:584] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet wm4wu/ruby-hello-world-85858f65d5 10577:2021-04-11T15:36:48.732585873Z E0411 15:36:48.732552 1 replica_set.go:532] sync "wm4wu/ruby-hello-world-85858f65d5" failed with pods "ruby-hello-world-85858f65d5-" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 10578:2021-04-11T15:36:48.732646974Z I0411 15:36:48.732612 1 event.go:291] "Event occurred" object="wm4wu/ruby-hello-world-85858f65d5" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message="Error creating: pods \"ruby-hello-world-85858f65d5-\" is forbidden: unable to validate against any security context constraint: [provider \"node-exporter\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]" 10580:2021-04-11T15:36:49.373809357Z I0411 15:36:49.373678 1 replica_set.go:559] "Too few replicas" replicaSet="wm4wu/ruby-hello-world-85858f65d5" need=1 creating=1 10581:2021-04-11T15:36:49.385425846Z I0411 15:36:49.385379 1 replica_set.go:584] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet wm4wu/ruby-hello-world-85858f65d5 10582:2021-04-11T15:36:49.385460446Z I0411 15:36:49.385431 1 event.go:291] "Event occurred" object="wm4wu/ruby-hello-world-85858f65d5" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message="Error creating: pods \"ruby-hello-world-85858f65d5-\" is forbidden: unable to validate against any security context constraint: [provider \"node-exporter\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]" 10583:2021-04-11T15:36:49.385460446Z E0411 15:36:49.385434 1 replica_set.go:532] sync "wm4wu/ruby-hello-world-85858f65d5" failed with pods "ruby-hello-world-85858f65d5-" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 10604:2021-04-11T15:36:50.666014184Z I0411 15:36:50.665945 1 replica_set.go:559] "Too few replicas" replicaSet="wm4wu/ruby-hello-world-85858f65d5" need=1 creating=1 10605:2021-04-11T15:36:50.675454037Z I0411 15:36:50.675376 1 replica_set.go:584] Slow-start failure. Skipping creation of 1 pods, decrementing expectations for ReplicaSet wm4wu/ruby-hello-world-85858f65d5 10606:2021-04-11T15:36:50.675454037Z E0411 15:36:50.675443 1 replica_set.go:532] sync "wm4wu/ruby-hello-world-85858f65d5" failed with pods "ruby-hello-world-85858f65d5-" is forbidden: unable to validate against any security context constraint: [provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount] 10607:2021-04-11T15:36:50.675496137Z I0411 15:36:50.675455 1 event.go:291] "Event occurred" object="wm4wu/ruby-hello-world-85858f65d5" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message="Error creating: pods \"ruby-hello-world-85858f65d5-\" is forbidden: unable to validate against any security context constraint: [provider \"node-exporter\": Forbidden: not usable by user or serviceaccount, provider \"privileged\": Forbidden: not usable by user or serviceaccount]" 10612:2021-04-11T15:36:53.235688281Z I0411 15:36:53.235621 1 replica_set.go:559] "Too few replicas" replicaSet="wm4wu/ruby-hello-world-85858f65d5" need=1 creating=1 10614:2021-04-11T15:36:57.096223505Z I0411 15:36:57.096169 1 garbagecollector.go:471] "Processing object" object="wm4wu/ruby-hello-world-1-build" objectUID=bfde1354-cb48-4ae8-846c-426aee56e717 kind="Pod" virtual=false 10615:2021-04-11T15:36:57.328238796Z E0411 15:36:57.328172 1 tokens_controller.go:269] error synchronizing serviceaccount wm4wu/builder: secrets "builder-token-kgqtp" is forbidden: unable to create new content in namespace wm4wu because it is being terminated 10616:2021-04-11T15:36:57.354000772Z E0411 15:36:57.353945 1 tokens_controller.go:269] error synchronizing serviceaccount wm4wu/default: secrets "default-token-lr27j" is forbidden: unable to create new content in namespace wm4wu because it is being terminated 10617:2021-04-11T15:36:57.372436242Z E0411 15:36:57.372394 1 tokens_controller.go:269] error synchronizing serviceaccount wm4wu/builder: secrets "builder-token-qgggv" is forbidden: unable to create new content in namespace wm4wu because it is being terminated 10618:2021-04-11T15:36:57.381191370Z E0411 15:36:57.381147 1 tokens_controller.go:269] error synchronizing serviceaccount wm4wu/default: secrets "default-token-v5bs5" is forbidden: unable to create new content in namespace wm4wu because it is being terminated 10619:2021-04-11T15:36:57.401511867Z E0411 15:36:57.401467 1 tokens_controller.go:269] error synchronizing serviceaccount wm4wu/deployer: secrets "deployer-token-n9tn8" is forbidden: unable to create new content in namespace wm4wu because it is being terminated 10620:2021-04-11T15:36:57.423647990Z E0411 15:36:57.423596 1 tokens_controller.go:269] error synchronizing serviceaccount wm4wu/deployer: secrets "deployer-token-bhxxl" is forbidden: unable to create new content in namespace wm4wu because it is being terminated 10621:2021-04-11T15:36:57.546203882Z E0411 15:36:57.546047 1 tokens_controller.go:269] error synchronizing serviceaccount wm4wu/default: secrets "default-token-kbfz4" is forbidden: unable to create new content in namespace wm4wu because it is being terminated 10622:2021-04-11T15:36:57.549447429Z I0411 15:36:57.549369 1 deployment_controller.go:583] "Deployment has been deleted" deployment="wm4wu/ruby-hello-world" 10675:2021-04-11T15:37:08.744251507Z I0411 15:37:08.744163 1 namespace_controller.go:185] Namespace has been deleted wm4wu 11679:2021-04-11T15:46:49.000473106Z I0411 15:46:49.000425 1 deployment_controller.go:583] "Deployment has been deleted" deployment="wm4wu/ruby-hello-world"
Project with issue: 15:36:45 INFO> Shell Commands: oc new-project wm4wu --kubeconfig=/home/jenkins/ws/workspace/ocp-common/Runner/workdir/ocp4_testuser-28.kubeconfig Now using project "wm4wu" on server "https://api.qeci-18869.qe.azure.devcluster.openshift.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname 15:36:46 INFO> Exit Status: 0 15:36:47 INFO> oc get projects wm4wu --output=yaml --kubeconfig=/home/jenkins/ws/workspace/ocp-common/Runner/workdir/ocp4_testuser-28.kubeconfig 15:36:47 INFO> After 1 iterations and 1 seconds: apiVersion: project.openshift.io/v1 kind: Project metadata: annotations: openshift.io/description: "" openshift.io/display-name: "" openshift.io/requester: testuser-28 creationTimestamp: "2021-04-11T15:36:45Z" labels: kubernetes.io/metadata.name: wm4wu name: wm4wu resourceVersion: "101308" uid: e53eaac5-f039-4fd5-8d19-470873d3b522 spec: finalizers: - kubernetes status: phase: Active Project without issue: 02:35:19 INFO> Shell Commands: oc new-project yfx-w --kubeconfig=/home/jenkins/ws/workspace/ocp-common/Runner/workdir/ocp4_testuser-12.kubeconfig Now using project "yfx-w" on server "https://api.qeci-18896.qe.azure.devcluster.openshift.com:6443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-example to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname 02:35:20 INFO> Exit Status: 0 02:35:21 INFO> oc get projects yfx-w --output=yaml --kubeconfig=/home/jenkins/ws/workspace/ocp-common/Runner/workdir/ocp4_testuser-12.kubeconfig 02:35:21 INFO> After 1 iterations and 1 seconds: apiVersion: project.openshift.io/v1 kind: Project metadata: annotations: openshift.io/description: "" openshift.io/display-name: "" openshift.io/requester: testuser-12 openshift.io/sa.scc.mcs: s0:c85,c50 openshift.io/sa.scc.supplemental-groups: 1007240000/10000 openshift.io/sa.scc.uid-range: 1007240000/10000 creationTimestamp: "2021-04-12T02:35:19Z" managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: f:openshift.io/sa.scc.mcs: {} f:openshift.io/sa.scc.supplemental-groups: {} f:openshift.io/sa.scc.uid-range: {} manager: cluster-policy-controller operation: Update time: "2021-04-12T02:35:19Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:openshift.io/description: {} f:openshift.io/display-name: {} f:openshift.io/requester: {} f:status: f:phase: {} manager: openshift-apiserver operation: Update time: "2021-04-12T02:35:19Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:spec: f:finalizers: {} manager: openshift-controller-manager operation: Update time: "2021-04-12T02:35:19Z" name: yfx-w resourceVersion: "115283" selfLink: /api/v1/namespaces/yfx-w uid: 9343c422-9627-4e46-a736-9e9a4a341d37 spec: finalizers: - kubernetes status: phase: Active seems this related to the annotations not added in acceptable time.
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Keywords if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.
404 The requested URL /zhouying/must-gather.local.159865815163817258/ was not found on this server. Please reopen if this happens again.
The LifecycleStale keyword was removed because the bug got commented on recently. The bug assignee was notified.
Have you considered to wait between project creation and pod creation? Kubernetes is an eventual consistent system with most of its policy features like RBAC and SCCs. When you create a project and RBAC rules for the privileged pod, controller have to sync first. I bet your test is racing with that. That's normal and expected. You have to add synchronization or polling yourself.
This relates to https://bugzilla.redhat.com/show_bug.cgi?id=1970331
closing out as a duplicate of #1970331 as it is the same root cause. *** This bug has been marked as a duplicate of bug 1970331 ***