Bug 1847594
| Summary: | pods in openshift-cnv namespace no longer have openshift.io/scc under metadata.annotations | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Kedar Bidarkar <kbidarka> |
| Component: | Networking | Assignee: | Petr Horáček <phoracek> |
| Status: | CLOSED ERRATA | QA Contact: | Israel Pinto <ipinto> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 2.4.0 | CC: | cnv-qe-bugs, ncredi, stirabos |
| Target Milestone: | --- | ||
| Target Release: | future | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | cluster-network-addons-operator-container-v2.4.0-34 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-07-28 19:10:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1834839, 1843219 | ||
The SCC labels disappear because the admission control is disabled for the whole openshift-cnv namespace. that happens because of the "openshift.io/run-level:0" label which is being set by the Kubemacpool. I am moving this bug to the network team. There is already a PR in US: https://github.com/k8snetworkplumbingwg/kubemacpool/pull/204 Tested today with "cluster-network-addons-operator/images/v2.4.0-34" image and I don't see the "openshift.io/scc" in the annotations of any of the pods. [kbidarka@kbidarka-host get_current_version]$ for i in $(oc get pods -n openshift-cnv | grep -v NAME | awk '{print $1}'); do echo $i ; oc get pod $i -n openshift-cnv -o yaml | grep scc; done
bridge-marker-42x9l
bridge-marker-48gdw
bridge-marker-8crsr
bridge-marker-f2kvl
bridge-marker-hqcw2
bridge-marker-tlrc8
cdi-apiserver-57b469b699-fd6hd
cdi-deployment-7f59c8d85-xt4ks
cdi-operator-698647ddc4-xx9bs
cdi-uploadproxy-66dd8d79cf-nl7bf
cluster-network-addons-operator-66b9b577dd-ltr56
hco-operator-5888587f79-lgpmk
hostpath-provisioner-operator-74cd6c7979-qvk29
kube-cni-linux-bridge-plugin-2klqz
kube-cni-linux-bridge-plugin-429ph
kube-cni-linux-bridge-plugin-dzhnb
kube-cni-linux-bridge-plugin-jpd9x
kube-cni-linux-bridge-plugin-pwbxc
kube-cni-linux-bridge-plugin-zwmwj
kubemacpool-mac-controller-manager-865d98484c-sf8qp
kubemacpool-mac-controller-manager-865d98484c-tm295
kubevirt-ssp-operator-584c955dc4-bh48r
nmstate-handler-2zw9s
nmstate-handler-4lrvn
nmstate-handler-6zrhv
nmstate-handler-89cqb
nmstate-handler-b668c
nmstate-handler-p9ddh
node-maintenance-operator-5c4f945bc5-f2csn
ovs-cni-amd64-259cb
ovs-cni-amd64-m4sjs
ovs-cni-amd64-n6tvm
ovs-cni-amd64-swtqh
ovs-cni-amd64-wkh6w
ovs-cni-amd64-wkmnl
virt-api-7b6f88bc54-mskm7
virt-api-7b6f88bc54-wbrck
virt-controller-dcf5b79f7-cmrp2
virt-controller-dcf5b79f7-rjhl4
virt-handler-blbks
virt-handler-r2g65
virt-handler-vqsph
virt-operator-7ffd8cfb5d-j5xmd
virt-operator-7ffd8cfb5d-rsmc7
vm-import-controller-64c97966b9-jgqmq
vm-import-operator-7b758665c9-vwn84
What about the run-level (https://bugzilla.redhat.com/show_bug.cgi?id=1847594#c1)? Kedar, could you show the namespace? openshift.io/run-level: "1" currentlyy, I see this value from the setup.
[kbidarka@kbidarka-host cnv-tests]$ oc get namespace openshift-cnv -o yaml
apiVersion: v1
kind: Namespace
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"labels":{"openshift.io/cluster-monitoring":"true","openshift.io/run-level":"1"},"name":"openshift-cnv"}}
openshift.io/sa.scc.mcs: s0:c24,c9
openshift.io/sa.scc.supplemental-groups: 1000570000/10000
openshift.io/sa.scc.uid-range: 1000570000/10000
creationTimestamp: "2020-07-02T08:41:47Z"
labels:
control-plane: mac-controller-manager
mutatepods.kubemacpool.io: ignore
name: openshift-cnv
networkaddonsoperator.network.kubevirt.io/version: sha256_ecbcbe6e8ed9015ed23aa3a93440fc3f4728ee79b97c1cfcf9152d05
olm.operatorgroup.uid/9e86dc3e-e83f-4873-b898-d74d0411c551: ""
olm.operatorgroup.uid/10b8ea93-46b6-457e-a8e1-59341008ad0b: ""
openshift.io/cluster-monitoring: "true"
openshift.io/run-level: "1"
managedFields:
CNAO was setting run-level "0". This issue seems unrelated. The fact that SCC is missing on all pods now (not only on a subset) suggests that the setting was placed even before CNAO started. Kedar, would you please try to pinpoint what is the source of this issue? Deploying a fresh cluster and installing HCO first would be a good start. Check that HCO and other operators have that annotation applied before you apply HyperConverged CR to install all the components. Tried the following to check,
Install just HCO and other operators, without applying HyperConverged CR
But, we still see the same issue, "openshift.io/scc" is still missing from the annotations.
[kbidarka@kbidarka-host cnv-tests]$ oc get nodes
NAME STATUS ROLES AGE VERSION
host-172-16-0-18 Ready master 113m v1.18.3+6025c28
host-172-16-0-43 Ready master 113m v1.18.3+6025c28
host-172-16-0-50 Ready worker 104m v1.18.3+6025c28
host-172-16-0-79 Ready worker 104m v1.18.3+6025c28
host-172-16-0-81 Ready master 113m v1.18.3+6025c28
host-172-16-0-95 Ready worker 104m v1.18.3+6025c28
[kbidarka@kbidarka-host cnv-tests]$ oc get pods -n openshift-cnv
NAME READY STATUS RESTARTS AGE
cdi-operator-74776646d7-lh2g4 1/1 Running 0 10m
cluster-network-addons-operator-856dcd8884-6hlnt 1/1 Running 0 10m
hco-operator-58fcd9dbd-cx2jg 1/1 Running 0 10m
hostpath-provisioner-5mxb5 1/1 Running 0 5m12s
hostpath-provisioner-8kgjv 1/1 Running 0 5m12s
hostpath-provisioner-bbrwh 1/1 Running 0 5m12s
hostpath-provisioner-operator-55587fd4bd-h7nwx 1/1 Running 0 10m
kubevirt-ssp-operator-6c6658b98-tb8w8 1/1 Running 0 10m
node-maintenance-operator-69cfccff77-ktm4l 1/1 Running 0 11m
virt-operator-5ff895bb7b-4fq9s 1/1 Running 0 8m
virt-operator-5ff895bb7b-sbgfq 1/1 Running 0 10m
vm-import-operator-859f8ffdd6-q48d6 1/1 Running 0 10m
[kbidarka@kbidarka-host cnv-tests]$ for i in $(oc get pods -n openshift-cnv | grep -v NAME | awk '{print $1}'); do echo $i ; oc get pod $i -n openshift-cnv -o yaml | grep scc; done
cdi-operator-74776646d7-lh2g4
cluster-network-addons-operator-856dcd8884-6hlnt
hco-operator-58fcd9dbd-cx2jg
hostpath-provisioner-5mxb5
hostpath-provisioner-8kgjv
hostpath-provisioner-bbrwh
hostpath-provisioner-operator-55587fd4bd-h7nwx
kubevirt-ssp-operator-6c6658b98-tb8w8
node-maintenance-operator-69cfccff77-ktm4l
virt-operator-5ff895bb7b-4fq9s
virt-operator-5ff895bb7b-sbgfq
vm-import-operator-859f8ffdd6-q48d6
[kbidarka@kbidarka-host cnv-tests]$ oc get namespace openshift-cnv -o yaml
apiVersion: v1
kind: Namespace
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Namespace","metadata":{"annotations":{},"labels":{"openshift.io/cluster-monitoring":"true","openshift.io/run-level":"1"},"name":"openshift-cnv"}}
openshift.io/sa.scc.mcs: s0:c25,c0
openshift.io/sa.scc.supplemental-groups: 1000600000/10000
openshift.io/sa.scc.uid-range: 1000600000/10000
creationTimestamp: "2020-07-16T11:20:58Z"
labels:
olm.operatorgroup.uid/19426d15-2153-4830-87a1-287fed31f8f1: ""
olm.operatorgroup.uid/b67dbcd5-5e84-4da8-9b30-c7e0583417db: ""
openshift.io/cluster-monitoring: "true"
openshift.io/run-level: "1"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:openshift.io/sa.scc.mcs: {}
f:openshift.io/sa.scc.supplemental-groups: {}
f:openshift.io/sa.scc.uid-range: {}
manager: cluster-policy-controller
operation: Update
time: "2020-07-16T11:20:58Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:openshift.io/cluster-monitoring: {}
f:openshift.io/run-level: {}
f:status:
f:phase: {}
manager: oc
operation: Update
time: "2020-07-16T11:20:58Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:labels:
f:olm.operatorgroup.uid/19426d15-2153-4830-87a1-287fed31f8f1: {}
f:olm.operatorgroup.uid/b67dbcd5-5e84-4da8-9b30-c7e0583417db: {}
manager: olm
operation: Update
time: "2020-07-16T11:22:41Z"
name: openshift-cnv
resourceVersion: "78129"
selfLink: /api/v1/namespaces/openshift-cnv
uid: 52890913-9bad-42a8-b1b0-7968ce710802
spec:
finalizers:
- kubernetes
status:
phase: Active
I just tried reproducing this with hco-bundle-registry:v2.3.0-479 on OCP 4.5.1 and after installing the operators all the SCC are there for me:
[stirabos@crc ~]$ oc get pods -n openshift-cnv
NAME READY STATUS RESTARTS AGE
cdi-operator-564688bcfb-zzwjw 1/1 Running 0 6m31s
cluster-network-addons-operator-6c89cf84c7-6kjw8 1/1 Running 0 6m33s
hco-operator-758d6766c5-7vmlg 1/1 Running 0 6m34s
hostpath-provisioner-operator-665b797645-lv594 1/1 Running 0 6m29s
kubevirt-ssp-operator-5bcf9b4f75-2xjpz 1/1 Running 0 6m32s
node-maintenance-operator-c8f6478f4-tbvgr 1/1 Running 0 6m30s
virt-operator-59f8fc785f-dpp6s 1/1 Running 0 67s
virt-operator-59f8fc785f-htqmt 1/1 Running 0 68s
vm-import-operator-64f5bb44cc-8bc5v 1/1 Running 0 6m28s
[stirabos@crc ~]$ for i in $(oc get pods -n openshift-cnv | grep -v NAME | awk '{print $1}'); do echo $i ; oc get pod $i -n openshift-cnv -o yaml | grep scc; done
cdi-operator-564688bcfb-zzwjw
openshift.io/scc: restricted
cluster-network-addons-operator-6c89cf84c7-6kjw8
openshift.io/scc: anyuid
hco-operator-758d6766c5-7vmlg
openshift.io/scc: restricted
hostpath-provisioner-operator-665b797645-lv594
openshift.io/scc: restricted
kubevirt-ssp-operator-5bcf9b4f75-2xjpz
openshift.io/scc: restricted
node-maintenance-operator-c8f6478f4-tbvgr
openshift.io/scc: restricted
virt-operator-59f8fc785f-dpp6s
openshift.io/scc: restricted
virt-operator-59f8fc785f-htqmt
openshift.io/scc: restricted
vm-import-operator-64f5bb44cc-8bc5v
openshift.io/scc: restricted
Then I created HCO CR, the deployment successfully completed and scc annotation are still there:
I'm not able to reproduce this.
[stirabos@crc ~]$ oc get pods -n openshift-cnv
NAME READY STATUS RESTARTS AGE
bridge-marker-c5wls 1/1 Running 0 15m
cdi-apiserver-78b48bff87-7bccs 1/1 Running 0 15m
cdi-deployment-78b7bc7f75-cv72g 1/1 Running 0 15m
cdi-operator-6b7f8479d6-q6l2c 1/1 Running 0 17m
cdi-uploadproxy-7d6558df97-zwldb 1/1 Running 0 15m
cluster-network-addons-operator-68c449c9b7-w2767 1/1 Running 0 17m
hco-operator-789755b5d7-25fzv 1/1 Running 0 17m
hostpath-provisioner-operator-5684955459-mpxc7 1/1 Running 0 17m
kube-cni-linux-bridge-plugin-mhvgg 1/1 Running 0 15m
kubemacpool-mac-controller-manager-777cd6c67c-lg2z5 1/1 Running 0 15m
kubemacpool-mac-controller-manager-777cd6c67c-t6t88 1/1 Running 0 15m
kubevirt-node-labeller-sc6xh 1/1 Running 0 10m
kubevirt-ssp-operator-74d9d95969-fht4s 1/1 Running 0 17m
nmstate-handler-pz99f 1/1 Running 0 15m
node-maintenance-operator-544f7c8676-67hqn 1/1 Running 0 17m
ovs-cni-amd64-7frfc 1/1 Running 0 15m
virt-api-866bf47b7b-2d27q 1/1 Running 0 13m
virt-api-866bf47b7b-7n8zn 1/1 Running 0 13m
virt-controller-ccd48cd56-jjnc8 1/1 Running 0 11m
virt-controller-ccd48cd56-wflsc 1/1 Running 0 11m
virt-handler-sg6pf 1/1 Running 0 11m
virt-operator-585bbc87b5-s9mcm 1/1 Running 0 17m
virt-operator-585bbc87b5-wsq49 1/1 Running 0 16m
virt-template-validator-85c4c9cd54-2vxng 1/1 Running 0 11m
virt-template-validator-85c4c9cd54-hvkt2 1/1 Running 0 11m
vm-import-controller-74649dd8d6-8whpk 1/1 Running 0 15m
vm-import-operator-697f669bc-7mdbx 1/1 Running 0 17m
[stirabos@crc ~]$ for i in $(oc get pods -n openshift-cnv | grep -v NAME | awk '{print $1}'); do echo $i ; oc get pod $i -n openshift-cnv -o yaml | grep scc; done
bridge-marker-c5wls
openshift.io/scc: bridge-marker
cdi-apiserver-78b48bff87-7bccs
openshift.io/scc: restricted
cdi-deployment-78b7bc7f75-cv72g
openshift.io/scc: containerized-data-importer
cdi-operator-6b7f8479d6-q6l2c
openshift.io/scc: restricted
cdi-uploadproxy-7d6558df97-zwldb
openshift.io/scc: restricted
cluster-network-addons-operator-68c449c9b7-w2767
openshift.io/scc: anyuid
hco-operator-789755b5d7-25fzv
openshift.io/scc: restricted
hostpath-provisioner-operator-5684955459-mpxc7
openshift.io/scc: restricted
kube-cni-linux-bridge-plugin-mhvgg
openshift.io/scc: linux-bridge
kubemacpool-mac-controller-manager-777cd6c67c-lg2z5
openshift.io/scc: restricted
kubemacpool-mac-controller-manager-777cd6c67c-t6t88
openshift.io/scc: restricted
kubevirt-node-labeller-sc6xh
openshift.io/scc: kubevirt-node-labeller
kubevirt-ssp-operator-74d9d95969-fht4s
openshift.io/scc: restricted
nmstate-handler-pz99f
openshift.io/scc: nmstate
node-maintenance-operator-544f7c8676-67hqn
openshift.io/scc: restricted
ovs-cni-amd64-7frfc
openshift.io/scc: ovs-cni-marker
virt-api-866bf47b7b-2d27q
openshift.io/scc: restricted
virt-api-866bf47b7b-7n8zn
openshift.io/scc: restricted
virt-controller-ccd48cd56-jjnc8
openshift.io/scc: restricted
virt-controller-ccd48cd56-wflsc
openshift.io/scc: restricted
virt-handler-sg6pf
openshift.io/scc: kubevirt-handler
virt-operator-585bbc87b5-s9mcm
openshift.io/scc: restricted
virt-operator-585bbc87b5-wsq49
openshift.io/scc: restricted
virt-template-validator-85c4c9cd54-2vxng
openshift.io/scc: restricted
virt-template-validator-85c4c9cd54-hvkt2
openshift.io/scc: restricted
vm-import-controller-74649dd8d6-8whpk
openshift.io/scc: restricted
vm-import-operator-697f669bc-7mdbx
openshift.io/scc: restricted
I also have to report that I deployed from the UI as an end user letting OLM console create the missing namespace for me while Kedar triggers the script on QE automation. Maybe the issue is there and not in the code. Still unclear. [kbidarka@kbidarka-host cnv-tests]$ for i in $(oc get pods -n openshift-cnv | grep -v NAME | awk '{print $1}'); do echo $i ; oc get pod $i -n openshift-cnv -o yaml | grep scc; done
bridge-marker-4bx9d
openshift.io/scc: bridge-marker
bridge-marker-d57mc
openshift.io/scc: bridge-marker
bridge-marker-fpwfg
openshift.io/scc: bridge-marker
bridge-marker-gtbkm
openshift.io/scc: bridge-marker
bridge-marker-l6crw
openshift.io/scc: bridge-marker
bridge-marker-wdx7b
openshift.io/scc: bridge-marker
cdi-apiserver-569788b897-x7njn
openshift.io/scc: restricted
cdi-deployment-84c977d8f9-f6jqj
openshift.io/scc: containerized-data-importer
cdi-operator-59c6cd9fff-z92nm
openshift.io/scc: restricted
cdi-uploadproxy-6b86f768c-ccdbs
openshift.io/scc: restricted
cluster-network-addons-operator-7ff9db5c68-hn66p
openshift.io/scc: anyuid
hco-operator-5f4788b848-fdxw5
openshift.io/scc: restricted
hostpath-provisioner-2c4ng
openshift.io/scc: hostpath-provisioner
hostpath-provisioner-operator-7d4d65c89b-rcxxb
openshift.io/scc: restricted
hostpath-provisioner-x9hwm
openshift.io/scc: hostpath-provisioner
hostpath-provisioner-zpkhr
openshift.io/scc: hostpath-provisioner
kube-cni-linux-bridge-plugin-f8c5s
openshift.io/scc: linux-bridge
kube-cni-linux-bridge-plugin-k2vlc
openshift.io/scc: linux-bridge
kube-cni-linux-bridge-plugin-ljvwc
openshift.io/scc: linux-bridge
kube-cni-linux-bridge-plugin-pkbzw
openshift.io/scc: linux-bridge
kube-cni-linux-bridge-plugin-wpz4s
openshift.io/scc: linux-bridge
kube-cni-linux-bridge-plugin-xdfmm
openshift.io/scc: linux-bridge
kubemacpool-mac-controller-manager-5f5f55bcc7-pplrm
openshift.io/scc: restricted
kubemacpool-mac-controller-manager-5f5f55bcc7-wpczp
openshift.io/scc: restricted
kubevirt-node-labeller-fvmlm
openshift.io/scc: kubevirt-node-labeller
kubevirt-node-labeller-gxpxn
openshift.io/scc: kubevirt-node-labeller
kubevirt-node-labeller-jflzg
openshift.io/scc: kubevirt-node-labeller
kubevirt-ssp-operator-6657fdcc68-r77ts
openshift.io/scc: restricted
nmstate-handler-8b62v
openshift.io/scc: nmstate
nmstate-handler-ddh6b
openshift.io/scc: nmstate
nmstate-handler-ldhnx
openshift.io/scc: nmstate
nmstate-handler-tj7vv
openshift.io/scc: nmstate
nmstate-handler-vcxsv
openshift.io/scc: nmstate
nmstate-handler-xznw8
openshift.io/scc: nmstate
node-maintenance-operator-768598f58f-2fwtp
openshift.io/scc: restricted
ovs-cni-amd64-4twnh
openshift.io/scc: ovs-cni-marker
ovs-cni-amd64-7x2tr
openshift.io/scc: ovs-cni-marker
ovs-cni-amd64-8gzff
openshift.io/scc: ovs-cni-marker
ovs-cni-amd64-g7qr7
openshift.io/scc: ovs-cni-marker
ovs-cni-amd64-lppp4
openshift.io/scc: ovs-cni-marker
ovs-cni-amd64-n9v9g
openshift.io/scc: ovs-cni-marker
virt-api-b6f94b95d-58sdp
openshift.io/scc: restricted
virt-api-b6f94b95d-vwhd8
openshift.io/scc: restricted
virt-controller-7d6cf877cd-8kts6
openshift.io/scc: restricted
virt-controller-7d6cf877cd-mzbxm
openshift.io/scc: restricted
virt-handler-24xk6
openshift.io/scc: kubevirt-handler
virt-handler-2zgw8
openshift.io/scc: kubevirt-handler
virt-handler-vcjjr
openshift.io/scc: kubevirt-handler
virt-operator-565c54984b-jfq99
openshift.io/scc: restricted
virt-operator-565c54984b-k8njl
openshift.io/scc: restricted
virt-template-validator-85bd457694-fq97k
openshift.io/scc: restricted
virt-template-validator-85bd457694-gn86l
openshift.io/scc: restricted
vm-import-controller-74b954cc6b-8n5vz
openshift.io/scc: restricted
vm-import-operator-5b87cd6f48-hmtnj
openshift.io/scc: restricted
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:3194 |
Description of problem: I don't see '.metadata.annotations["openshift.io/scc"]' from the pods anymore in openshift-cnv namespace Version-Release number of selected component (if applicable): 2.4/4.5 How reproducible: Steps to Reproduce: 1. Install CNV 2. Check '.metadata.annotations["openshift.io/scc"]' if exists in pods. 3. Actual results: '.metadata.annotations["openshift.io/scc"]' is missing from the pods of openshift-cnv Expected results: .metadata.annotations["openshift.io/scc"]' should exist in the pods of openshift-cnv Additional info: Currently, I see it only on the operator pods in openshift-cnv, but other pods in openshift-cnv this seems not to be present. [root@f19-h05-000-r620 auth]# for i in $(oc get pods -n openshift-cnv | grep -v NAME | awk '{print $1}' | grep operator); do echo $i ; oc get pod $i -n openshift-cnv -o yaml | grep scc; done cdi-operator-6c5dd976f6-rdmc8 openshift.io/scc: restricted cluster-network-addons-operator-7bb5947744-gzlvr openshift.io/scc: anyuid hco-operator-567bd5d79b-scsgs openshift.io/scc: restricted hostpath-provisioner-operator-6f7b8b8776-txvpv openshift.io/scc: restricted kubevirt-ssp-operator-6957bf4d48-s9x9w openshift.io/scc: restricted node-maintenance-operator-7dbffc5b86-62phg openshift.io/scc: restricted virt-operator-649545d85f-j7wbm openshift.io/scc: restricted virt-operator-649545d85f-rgppc openshift.io/scc: restricted vm-import-operator-857b68d545-x8gst openshift.io/scc: restricted