Description of problem: I don't see '.metadata.annotations["openshift.io/scc"]' from the pods anymore in openshift-cnv namespace Version-Release number of selected component (if applicable): 2.4/4.5 How reproducible: Steps to Reproduce: 1. Install CNV 2. Check '.metadata.annotations["openshift.io/scc"]' if exists in pods. 3. Actual results: '.metadata.annotations["openshift.io/scc"]' is missing from the pods of openshift-cnv Expected results: .metadata.annotations["openshift.io/scc"]' should exist in the pods of openshift-cnv Additional info: Currently, I see it only on the operator pods in openshift-cnv, but other pods in openshift-cnv this seems not to be present. [root@f19-h05-000-r620 auth]# for i in $(oc get pods -n openshift-cnv | grep -v NAME | awk '{print $1}' | grep operator); do echo $i ; oc get pod $i -n openshift-cnv -o yaml | grep scc; done cdi-operator-6c5dd976f6-rdmc8 openshift.io/scc: restricted cluster-network-addons-operator-7bb5947744-gzlvr openshift.io/scc: anyuid hco-operator-567bd5d79b-scsgs openshift.io/scc: restricted hostpath-provisioner-operator-6f7b8b8776-txvpv openshift.io/scc: restricted kubevirt-ssp-operator-6957bf4d48-s9x9w openshift.io/scc: restricted node-maintenance-operator-7dbffc5b86-62phg openshift.io/scc: restricted virt-operator-649545d85f-j7wbm openshift.io/scc: restricted virt-operator-649545d85f-rgppc openshift.io/scc: restricted vm-import-operator-857b68d545-x8gst openshift.io/scc: restricted
The SCC labels disappear because the admission control is disabled for the whole openshift-cnv namespace. that happens because of the "openshift.io/run-level:0" label which is being set by the Kubemacpool. I am moving this bug to the network team. There is already a PR in US: https://github.com/k8snetworkplumbingwg/kubemacpool/pull/204
Tested today with "cluster-network-addons-operator/images/v2.4.0-34" image and I don't see the "openshift.io/scc" in the annotations of any of the pods.
[kbidarka@kbidarka-host get_current_version]$ for i in $(oc get pods -n openshift-cnv | grep -v NAME | awk '{print $1}'); do echo $i ; oc get pod $i -n openshift-cnv -o yaml | grep scc; done bridge-marker-42x9l bridge-marker-48gdw bridge-marker-8crsr bridge-marker-f2kvl bridge-marker-hqcw2 bridge-marker-tlrc8 cdi-apiserver-57b469b699-fd6hd cdi-deployment-7f59c8d85-xt4ks cdi-operator-698647ddc4-xx9bs cdi-uploadproxy-66dd8d79cf-nl7bf cluster-network-addons-operator-66b9b577dd-ltr56 hco-operator-5888587f79-lgpmk hostpath-provisioner-operator-74cd6c7979-qvk29 kube-cni-linux-bridge-plugin-2klqz kube-cni-linux-bridge-plugin-429ph kube-cni-linux-bridge-plugin-dzhnb kube-cni-linux-bridge-plugin-jpd9x kube-cni-linux-bridge-plugin-pwbxc kube-cni-linux-bridge-plugin-zwmwj kubemacpool-mac-controller-manager-865d98484c-sf8qp kubemacpool-mac-controller-manager-865d98484c-tm295 kubevirt-ssp-operator-584c955dc4-bh48r nmstate-handler-2zw9s nmstate-handler-4lrvn nmstate-handler-6zrhv nmstate-handler-89cqb nmstate-handler-b668c nmstate-handler-p9ddh node-maintenance-operator-5c4f945bc5-f2csn ovs-cni-amd64-259cb ovs-cni-amd64-m4sjs ovs-cni-amd64-n6tvm ovs-cni-amd64-swtqh ovs-cni-amd64-wkh6w ovs-cni-amd64-wkmnl virt-api-7b6f88bc54-mskm7 virt-api-7b6f88bc54-wbrck virt-controller-dcf5b79f7-cmrp2 virt-controller-dcf5b79f7-rjhl4 virt-handler-blbks virt-handler-r2g65 virt-handler-vqsph virt-operator-7ffd8cfb5d-j5xmd virt-operator-7ffd8cfb5d-rsmc7 vm-import-controller-64c97966b9-jgqmq vm-import-operator-7b758665c9-vwn84
What about the run-level (https://bugzilla.redhat.com/show_bug.cgi?id=1847594#c1)? Kedar, could you show the namespace?
openshift.io/run-level: "1" currentlyy, I see this value from the setup. [kbidarka@kbidarka-host cnv-tests]$ oc get namespace openshift-cnv -o yaml apiVersion: v1 kind: Namespace metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"labels":{"openshift.io/cluster-monitoring":"true","openshift.io/run-level":"1"},"name":"openshift-cnv"}} openshift.io/sa.scc.mcs: s0:c24,c9 openshift.io/sa.scc.supplemental-groups: 1000570000/10000 openshift.io/sa.scc.uid-range: 1000570000/10000 creationTimestamp: "2020-07-02T08:41:47Z" labels: control-plane: mac-controller-manager mutatepods.kubemacpool.io: ignore name: openshift-cnv networkaddonsoperator.network.kubevirt.io/version: sha256_ecbcbe6e8ed9015ed23aa3a93440fc3f4728ee79b97c1cfcf9152d05 olm.operatorgroup.uid/9e86dc3e-e83f-4873-b898-d74d0411c551: "" olm.operatorgroup.uid/10b8ea93-46b6-457e-a8e1-59341008ad0b: "" openshift.io/cluster-monitoring: "true" openshift.io/run-level: "1" managedFields:
CNAO was setting run-level "0". This issue seems unrelated. The fact that SCC is missing on all pods now (not only on a subset) suggests that the setting was placed even before CNAO started. Kedar, would you please try to pinpoint what is the source of this issue? Deploying a fresh cluster and installing HCO first would be a good start. Check that HCO and other operators have that annotation applied before you apply HyperConverged CR to install all the components.
Tried the following to check, Install just HCO and other operators, without applying HyperConverged CR But, we still see the same issue, "openshift.io/scc" is still missing from the annotations. [kbidarka@kbidarka-host cnv-tests]$ oc get nodes NAME STATUS ROLES AGE VERSION host-172-16-0-18 Ready master 113m v1.18.3+6025c28 host-172-16-0-43 Ready master 113m v1.18.3+6025c28 host-172-16-0-50 Ready worker 104m v1.18.3+6025c28 host-172-16-0-79 Ready worker 104m v1.18.3+6025c28 host-172-16-0-81 Ready master 113m v1.18.3+6025c28 host-172-16-0-95 Ready worker 104m v1.18.3+6025c28 [kbidarka@kbidarka-host cnv-tests]$ oc get pods -n openshift-cnv NAME READY STATUS RESTARTS AGE cdi-operator-74776646d7-lh2g4 1/1 Running 0 10m cluster-network-addons-operator-856dcd8884-6hlnt 1/1 Running 0 10m hco-operator-58fcd9dbd-cx2jg 1/1 Running 0 10m hostpath-provisioner-5mxb5 1/1 Running 0 5m12s hostpath-provisioner-8kgjv 1/1 Running 0 5m12s hostpath-provisioner-bbrwh 1/1 Running 0 5m12s hostpath-provisioner-operator-55587fd4bd-h7nwx 1/1 Running 0 10m kubevirt-ssp-operator-6c6658b98-tb8w8 1/1 Running 0 10m node-maintenance-operator-69cfccff77-ktm4l 1/1 Running 0 11m virt-operator-5ff895bb7b-4fq9s 1/1 Running 0 8m virt-operator-5ff895bb7b-sbgfq 1/1 Running 0 10m vm-import-operator-859f8ffdd6-q48d6 1/1 Running 0 10m [kbidarka@kbidarka-host cnv-tests]$ for i in $(oc get pods -n openshift-cnv | grep -v NAME | awk '{print $1}'); do echo $i ; oc get pod $i -n openshift-cnv -o yaml | grep scc; done cdi-operator-74776646d7-lh2g4 cluster-network-addons-operator-856dcd8884-6hlnt hco-operator-58fcd9dbd-cx2jg hostpath-provisioner-5mxb5 hostpath-provisioner-8kgjv hostpath-provisioner-bbrwh hostpath-provisioner-operator-55587fd4bd-h7nwx kubevirt-ssp-operator-6c6658b98-tb8w8 node-maintenance-operator-69cfccff77-ktm4l virt-operator-5ff895bb7b-4fq9s virt-operator-5ff895bb7b-sbgfq vm-import-operator-859f8ffdd6-q48d6 [kbidarka@kbidarka-host cnv-tests]$ oc get namespace openshift-cnv -o yaml apiVersion: v1 kind: Namespace metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"labels":{"openshift.io/cluster-monitoring":"true","openshift.io/run-level":"1"},"name":"openshift-cnv"}} openshift.io/sa.scc.mcs: s0:c25,c0 openshift.io/sa.scc.supplemental-groups: 1000600000/10000 openshift.io/sa.scc.uid-range: 1000600000/10000 creationTimestamp: "2020-07-16T11:20:58Z" labels: olm.operatorgroup.uid/19426d15-2153-4830-87a1-287fed31f8f1: "" olm.operatorgroup.uid/b67dbcd5-5e84-4da8-9b30-c7e0583417db: "" openshift.io/cluster-monitoring: "true" openshift.io/run-level: "1" managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: f:openshift.io/sa.scc.mcs: {} f:openshift.io/sa.scc.supplemental-groups: {} f:openshift.io/sa.scc.uid-range: {} manager: cluster-policy-controller operation: Update time: "2020-07-16T11:20:58Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:kubectl.kubernetes.io/last-applied-configuration: {} f:labels: .: {} f:openshift.io/cluster-monitoring: {} f:openshift.io/run-level: {} f:status: f:phase: {} manager: oc operation: Update time: "2020-07-16T11:20:58Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:labels: f:olm.operatorgroup.uid/19426d15-2153-4830-87a1-287fed31f8f1: {} f:olm.operatorgroup.uid/b67dbcd5-5e84-4da8-9b30-c7e0583417db: {} manager: olm operation: Update time: "2020-07-16T11:22:41Z" name: openshift-cnv resourceVersion: "78129" selfLink: /api/v1/namespaces/openshift-cnv uid: 52890913-9bad-42a8-b1b0-7968ce710802 spec: finalizers: - kubernetes status: phase: Active
I just tried reproducing this with hco-bundle-registry:v2.3.0-479 on OCP 4.5.1 and after installing the operators all the SCC are there for me: [stirabos@crc ~]$ oc get pods -n openshift-cnv NAME READY STATUS RESTARTS AGE cdi-operator-564688bcfb-zzwjw 1/1 Running 0 6m31s cluster-network-addons-operator-6c89cf84c7-6kjw8 1/1 Running 0 6m33s hco-operator-758d6766c5-7vmlg 1/1 Running 0 6m34s hostpath-provisioner-operator-665b797645-lv594 1/1 Running 0 6m29s kubevirt-ssp-operator-5bcf9b4f75-2xjpz 1/1 Running 0 6m32s node-maintenance-operator-c8f6478f4-tbvgr 1/1 Running 0 6m30s virt-operator-59f8fc785f-dpp6s 1/1 Running 0 67s virt-operator-59f8fc785f-htqmt 1/1 Running 0 68s vm-import-operator-64f5bb44cc-8bc5v 1/1 Running 0 6m28s [stirabos@crc ~]$ for i in $(oc get pods -n openshift-cnv | grep -v NAME | awk '{print $1}'); do echo $i ; oc get pod $i -n openshift-cnv -o yaml | grep scc; done cdi-operator-564688bcfb-zzwjw openshift.io/scc: restricted cluster-network-addons-operator-6c89cf84c7-6kjw8 openshift.io/scc: anyuid hco-operator-758d6766c5-7vmlg openshift.io/scc: restricted hostpath-provisioner-operator-665b797645-lv594 openshift.io/scc: restricted kubevirt-ssp-operator-5bcf9b4f75-2xjpz openshift.io/scc: restricted node-maintenance-operator-c8f6478f4-tbvgr openshift.io/scc: restricted virt-operator-59f8fc785f-dpp6s openshift.io/scc: restricted virt-operator-59f8fc785f-htqmt openshift.io/scc: restricted vm-import-operator-64f5bb44cc-8bc5v openshift.io/scc: restricted
Then I created HCO CR, the deployment successfully completed and scc annotation are still there: I'm not able to reproduce this. [stirabos@crc ~]$ oc get pods -n openshift-cnv NAME READY STATUS RESTARTS AGE bridge-marker-c5wls 1/1 Running 0 15m cdi-apiserver-78b48bff87-7bccs 1/1 Running 0 15m cdi-deployment-78b7bc7f75-cv72g 1/1 Running 0 15m cdi-operator-6b7f8479d6-q6l2c 1/1 Running 0 17m cdi-uploadproxy-7d6558df97-zwldb 1/1 Running 0 15m cluster-network-addons-operator-68c449c9b7-w2767 1/1 Running 0 17m hco-operator-789755b5d7-25fzv 1/1 Running 0 17m hostpath-provisioner-operator-5684955459-mpxc7 1/1 Running 0 17m kube-cni-linux-bridge-plugin-mhvgg 1/1 Running 0 15m kubemacpool-mac-controller-manager-777cd6c67c-lg2z5 1/1 Running 0 15m kubemacpool-mac-controller-manager-777cd6c67c-t6t88 1/1 Running 0 15m kubevirt-node-labeller-sc6xh 1/1 Running 0 10m kubevirt-ssp-operator-74d9d95969-fht4s 1/1 Running 0 17m nmstate-handler-pz99f 1/1 Running 0 15m node-maintenance-operator-544f7c8676-67hqn 1/1 Running 0 17m ovs-cni-amd64-7frfc 1/1 Running 0 15m virt-api-866bf47b7b-2d27q 1/1 Running 0 13m virt-api-866bf47b7b-7n8zn 1/1 Running 0 13m virt-controller-ccd48cd56-jjnc8 1/1 Running 0 11m virt-controller-ccd48cd56-wflsc 1/1 Running 0 11m virt-handler-sg6pf 1/1 Running 0 11m virt-operator-585bbc87b5-s9mcm 1/1 Running 0 17m virt-operator-585bbc87b5-wsq49 1/1 Running 0 16m virt-template-validator-85c4c9cd54-2vxng 1/1 Running 0 11m virt-template-validator-85c4c9cd54-hvkt2 1/1 Running 0 11m vm-import-controller-74649dd8d6-8whpk 1/1 Running 0 15m vm-import-operator-697f669bc-7mdbx 1/1 Running 0 17m [stirabos@crc ~]$ for i in $(oc get pods -n openshift-cnv | grep -v NAME | awk '{print $1}'); do echo $i ; oc get pod $i -n openshift-cnv -o yaml | grep scc; done bridge-marker-c5wls openshift.io/scc: bridge-marker cdi-apiserver-78b48bff87-7bccs openshift.io/scc: restricted cdi-deployment-78b7bc7f75-cv72g openshift.io/scc: containerized-data-importer cdi-operator-6b7f8479d6-q6l2c openshift.io/scc: restricted cdi-uploadproxy-7d6558df97-zwldb openshift.io/scc: restricted cluster-network-addons-operator-68c449c9b7-w2767 openshift.io/scc: anyuid hco-operator-789755b5d7-25fzv openshift.io/scc: restricted hostpath-provisioner-operator-5684955459-mpxc7 openshift.io/scc: restricted kube-cni-linux-bridge-plugin-mhvgg openshift.io/scc: linux-bridge kubemacpool-mac-controller-manager-777cd6c67c-lg2z5 openshift.io/scc: restricted kubemacpool-mac-controller-manager-777cd6c67c-t6t88 openshift.io/scc: restricted kubevirt-node-labeller-sc6xh openshift.io/scc: kubevirt-node-labeller kubevirt-ssp-operator-74d9d95969-fht4s openshift.io/scc: restricted nmstate-handler-pz99f openshift.io/scc: nmstate node-maintenance-operator-544f7c8676-67hqn openshift.io/scc: restricted ovs-cni-amd64-7frfc openshift.io/scc: ovs-cni-marker virt-api-866bf47b7b-2d27q openshift.io/scc: restricted virt-api-866bf47b7b-7n8zn openshift.io/scc: restricted virt-controller-ccd48cd56-jjnc8 openshift.io/scc: restricted virt-controller-ccd48cd56-wflsc openshift.io/scc: restricted virt-handler-sg6pf openshift.io/scc: kubevirt-handler virt-operator-585bbc87b5-s9mcm openshift.io/scc: restricted virt-operator-585bbc87b5-wsq49 openshift.io/scc: restricted virt-template-validator-85c4c9cd54-2vxng openshift.io/scc: restricted virt-template-validator-85c4c9cd54-hvkt2 openshift.io/scc: restricted vm-import-controller-74649dd8d6-8whpk openshift.io/scc: restricted vm-import-operator-697f669bc-7mdbx openshift.io/scc: restricted
I also have to report that I deployed from the UI as an end user letting OLM console create the missing namespace for me while Kedar triggers the script on QE automation. Maybe the issue is there and not in the code. Still unclear.
[kbidarka@kbidarka-host cnv-tests]$ for i in $(oc get pods -n openshift-cnv | grep -v NAME | awk '{print $1}'); do echo $i ; oc get pod $i -n openshift-cnv -o yaml | grep scc; done bridge-marker-4bx9d openshift.io/scc: bridge-marker bridge-marker-d57mc openshift.io/scc: bridge-marker bridge-marker-fpwfg openshift.io/scc: bridge-marker bridge-marker-gtbkm openshift.io/scc: bridge-marker bridge-marker-l6crw openshift.io/scc: bridge-marker bridge-marker-wdx7b openshift.io/scc: bridge-marker cdi-apiserver-569788b897-x7njn openshift.io/scc: restricted cdi-deployment-84c977d8f9-f6jqj openshift.io/scc: containerized-data-importer cdi-operator-59c6cd9fff-z92nm openshift.io/scc: restricted cdi-uploadproxy-6b86f768c-ccdbs openshift.io/scc: restricted cluster-network-addons-operator-7ff9db5c68-hn66p openshift.io/scc: anyuid hco-operator-5f4788b848-fdxw5 openshift.io/scc: restricted hostpath-provisioner-2c4ng openshift.io/scc: hostpath-provisioner hostpath-provisioner-operator-7d4d65c89b-rcxxb openshift.io/scc: restricted hostpath-provisioner-x9hwm openshift.io/scc: hostpath-provisioner hostpath-provisioner-zpkhr openshift.io/scc: hostpath-provisioner kube-cni-linux-bridge-plugin-f8c5s openshift.io/scc: linux-bridge kube-cni-linux-bridge-plugin-k2vlc openshift.io/scc: linux-bridge kube-cni-linux-bridge-plugin-ljvwc openshift.io/scc: linux-bridge kube-cni-linux-bridge-plugin-pkbzw openshift.io/scc: linux-bridge kube-cni-linux-bridge-plugin-wpz4s openshift.io/scc: linux-bridge kube-cni-linux-bridge-plugin-xdfmm openshift.io/scc: linux-bridge kubemacpool-mac-controller-manager-5f5f55bcc7-pplrm openshift.io/scc: restricted kubemacpool-mac-controller-manager-5f5f55bcc7-wpczp openshift.io/scc: restricted kubevirt-node-labeller-fvmlm openshift.io/scc: kubevirt-node-labeller kubevirt-node-labeller-gxpxn openshift.io/scc: kubevirt-node-labeller kubevirt-node-labeller-jflzg openshift.io/scc: kubevirt-node-labeller kubevirt-ssp-operator-6657fdcc68-r77ts openshift.io/scc: restricted nmstate-handler-8b62v openshift.io/scc: nmstate nmstate-handler-ddh6b openshift.io/scc: nmstate nmstate-handler-ldhnx openshift.io/scc: nmstate nmstate-handler-tj7vv openshift.io/scc: nmstate nmstate-handler-vcxsv openshift.io/scc: nmstate nmstate-handler-xznw8 openshift.io/scc: nmstate node-maintenance-operator-768598f58f-2fwtp openshift.io/scc: restricted ovs-cni-amd64-4twnh openshift.io/scc: ovs-cni-marker ovs-cni-amd64-7x2tr openshift.io/scc: ovs-cni-marker ovs-cni-amd64-8gzff openshift.io/scc: ovs-cni-marker ovs-cni-amd64-g7qr7 openshift.io/scc: ovs-cni-marker ovs-cni-amd64-lppp4 openshift.io/scc: ovs-cni-marker ovs-cni-amd64-n9v9g openshift.io/scc: ovs-cni-marker virt-api-b6f94b95d-58sdp openshift.io/scc: restricted virt-api-b6f94b95d-vwhd8 openshift.io/scc: restricted virt-controller-7d6cf877cd-8kts6 openshift.io/scc: restricted virt-controller-7d6cf877cd-mzbxm openshift.io/scc: restricted virt-handler-24xk6 openshift.io/scc: kubevirt-handler virt-handler-2zgw8 openshift.io/scc: kubevirt-handler virt-handler-vcjjr openshift.io/scc: kubevirt-handler virt-operator-565c54984b-jfq99 openshift.io/scc: restricted virt-operator-565c54984b-k8njl openshift.io/scc: restricted virt-template-validator-85bd457694-fq97k openshift.io/scc: restricted virt-template-validator-85bd457694-gn86l openshift.io/scc: restricted vm-import-controller-74b954cc6b-8n5vz openshift.io/scc: restricted vm-import-operator-5b87cd6f48-hmtnj openshift.io/scc: restricted
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:3194