Bug 1892372 - operator pod with OLM webhooks is getting terminated and created several times during the installation
Summary: operator pod with OLM webhooks is getting terminated and created several time...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.5
Hardware: Unspecified
OS: Unspecified
medium
urgent
Target Milestone: ---
: 4.5.z
Assignee: Alexander Greene
QA Contact: kuiwang
URL:
Whiteboard:
Depends On: 1868712
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-10-28 15:04 UTC by OpenShift BugZilla Robot
Modified: 2021-03-03 04:40 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: When OLM is reconciling a CSV that is in the InstallReady phase, OLM will create the CA from scratch even if it had previously created valid and unexpired CAs. Consequence: New CAs Certs are created each time the operator is installed. Fix: Rather than regenerate the CA Certs from scratch, OLM now reuses existing certs created for the operator if they are available, unexpired, and valid. Result: OLM no longer creates new CA Certs if an operator re-enters the InstallReady phase.
Clone Of:
Environment:
Last Closed: 2021-03-03 04:40:29 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github operator-framework operator-lifecycle-manager pull 1865 0 None closed [release-4.5] Bug 1892372: OLM should use valid CA certs if available 2021-02-11 15:35:43 UTC
Red Hat Product Errata RHSA-2020:5194 0 None None None 2020-12-01 10:49:34 UTC
Red Hat Product Errata RHSA-2021:0428 0 None None None 2021-03-03 04:40:56 UTC

Comment 5 kuiwang 2020-11-23 06:10:32 UTC
verify it on 4.5. LGTM

--
[root@preserve-olm-env 1892372]#  oc get pod -n openshift-operator-lifecycle-manager
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-6465b8c97c-499vf   1/1     Running   0          35m
olm-operator-8454788696-85kc5       1/1     Running   0          35m
packageserver-7d8669cc44-6v4n9      1/1     Running   0          25m
packageserver-7d8669cc44-mjj8j      1/1     Running   1          25m
[root@preserve-olm-env 1892372]# oc exec catalog-operator-6465b8c97c-499vf -n openshift-operator-lifecycle-manager -- olm --version
OLM version: 0.15.1
git commit: fa71ab38bea652e03006d2df77b66058f3ff6c60
[root@preserve-olm-env 1892372]# 

[root@preserve-olm-env 1892372]# cat cs.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: hco-catalogsource
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/kubevirt/hco-container-registry:latest
  displayName: Openshift Virtualization
  publisher: grpc
[root@preserve-olm-env 1892372]# oc apply -f cs.yaml
catalogsource.operators.coreos.com/hco-catalogsource created
[root@preserve-olm-env 1892372]# 


Install "KubeVirt HyperConverged Cluster Operator" in channel 1.2.0 using OperatorHub.

[root@preserve-olm-env 1892372]# oc get csv -n kubevirt-hyperconverged
NAME                                      DISPLAY                                    VERSION   REPLACES   PHASE
kubevirt-hyperconverged-operator.v1.2.0   KubeVirt HyperConverged Cluster Operator   1.2.0                Installing
[root@preserve-olm-env 1892372]# oc get csv -n kubevirt-hyperconverged
NAME                                      DISPLAY                                    VERSION   REPLACES   PHASE
kubevirt-hyperconverged-operator.v1.2.0   KubeVirt HyperConverged Cluster Operator   1.2.0                Succeeded


[root@preserve-olm-env 1892372]# oc get pods -n kubevirt-hyperconverged
NAME                                               READY   STATUS    RESTARTS   AGE
cdi-operator-5dd7d8f78c-4gpgb                      1/1     Running   0          5m14s
cluster-network-addons-operator-67db4ccc65-8fls8   1/1     Running   0          5m15s
hco-operator-56855b88f8-dtltg                      1/1     Running   0          5m15s
hco-webhook-7476f997cb-f79xx                       1/1     Running   0          5m15s
hostpath-provisioner-operator-7cbb985658-gr6kh     1/1     Running   0          5m13s
kubevirt-ssp-operator-65f9d8974d-zn9dj             1/1     Running   0          5m14s
node-maintenance-operator-84d4758-7zc9d            1/1     Running   0          5m14s
virt-operator-5dbdf888f-f5sxf                      1/1     Running   0          4m34s
virt-operator-5dbdf888f-rgrx4                      1/1     Running   0          4m34s
vm-import-operator-696fffbff5-785cr                1/1     Running   0          5m13s
[root@preserve-olm-env 1892372]# oc get rs -n kubevirt-hyperconverged
NAME                                         DESIRED   CURRENT   READY   AGE
cdi-operator-5dd7d8f78c                      1         1         1       5m22s
cluster-network-addons-operator-67db4ccc65   1         1         1       5m23s
hco-operator-56855b88f8                      1         1         1       5m24s
hco-webhook-7476f997cb                       1         1         1       5m23s
hostpath-provisioner-operator-7cbb985658     1         1         1       5m21s
kubevirt-ssp-operator-65f9d8974d             1         1         1       5m22s
node-maintenance-operator-84d4758            1         1         1       5m22s
virt-operator-5dbdf888f                      2         2         2       5m23s
vm-import-operator-696fffbff5                1         1         1       5m21s
[root@preserve-olm-env 1892372]# 


[root@preserve-olm-env 1892372]# cat cr.yaml 
apiVersion: hco.kubevirt.io/v1beta1
kind: HyperConverged
metadata:
  name: kubevirt-hyperconverged
  namespace: kubevirt-hyperconverged
spec:
  version: 1.2.0
[root@preserve-olm-env 1892372]# oc apply -f cr.yaml 
hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged created
[root@preserve-olm-env 1892372]# 


[root@preserve-olm-env 1892372]# oc get pods -n kubevirt-hyperconverged
NAME                                                  READY   STATUS    RESTARTS   AGE
bridge-marker-clx75                                   1/1     Running   0          9m10s
bridge-marker-dswxm                                   1/1     Running   0          9m10s
bridge-marker-s6rvh                                   1/1     Running   0          9m10s
bridge-marker-svfgv                                   1/1     Running   0          9m10s
bridge-marker-wc92d                                   1/1     Running   0          9m10s
bridge-marker-xc9bz                                   1/1     Running   0          9m10s
cdi-apiserver-5558cdb57d-4hjn8                        1/1     Running   0          8m47s
cdi-deployment-6c6798bfdd-l4crb                       1/1     Running   0          8m47s
cdi-operator-5dd7d8f78c-4gpgb                         1/1     Running   0          15m
cdi-uploadproxy-79cdb778f7-llthv                      1/1     Running   0          8m47s
cluster-network-addons-operator-67db4ccc65-8fls8      1/1     Running   0          15m
hco-operator-56855b88f8-dtltg                         1/1     Running   0          15m
hco-webhook-7476f997cb-f79xx                          1/1     Running   0          15m
hostpath-provisioner-operator-7cbb985658-gr6kh        1/1     Running   0          15m
kube-cni-linux-bridge-plugin-4m4kd                    1/1     Running   0          9m11s
kube-cni-linux-bridge-plugin-89pps                    1/1     Running   0          9m11s
kube-cni-linux-bridge-plugin-n8rln                    1/1     Running   0          9m11s
kube-cni-linux-bridge-plugin-phkh5                    1/1     Running   0          9m11s
kube-cni-linux-bridge-plugin-sw2b7                    1/1     Running   0          9m11s
kube-cni-linux-bridge-plugin-z86wb                    1/1     Running   0          9m11s
kubemacpool-mac-controller-manager-67c9778c74-z29gs   1/1     Running   0          9m11s
kubevirt-node-labeller-kkvks                          1/1     Running   0          6m44s
kubevirt-node-labeller-l87js                          1/1     Running   0          6m44s
kubevirt-node-labeller-t2vgm                          1/1     Running   0          6m44s
kubevirt-ssp-operator-65f9d8974d-zn9dj                1/1     Running   0          15m
nmstate-handler-b5rwr                                 1/1     Running   0          9m8s
nmstate-handler-bkzz5                                 1/1     Running   0          9m8s
nmstate-handler-g44w8                                 1/1     Running   0          9m8s
nmstate-handler-mn6jz                                 1/1     Running   0          9m8s
nmstate-handler-vcw2b                                 1/1     Running   0          9m8s
nmstate-handler-wbxrt                                 1/1     Running   0          9m8s
nmstate-webhook-7558cdf884-qrpmk                      1/1     Running   0          9m8s
nmstate-webhook-7558cdf884-t8gw2                      1/1     Running   0          9m8s
node-maintenance-operator-84d4758-7zc9d               1/1     Running   0          15m
ovs-cni-amd64-5jdwc                                   1/1     Running   0          9m7s
ovs-cni-amd64-ctd47                                   1/1     Running   0          9m7s
ovs-cni-amd64-kgghs                                   1/1     Running   0          9m7s
ovs-cni-amd64-l9hlj                                   1/1     Running   0          9m7s
ovs-cni-amd64-sv7g6                                   1/1     Running   0          9m7s
ovs-cni-amd64-sxl45                                   1/1     Running   0          9m8s
virt-api-8cdc565dd-hg89q                              1/1     Running   0          8m36s
virt-api-8cdc565dd-lc72l                              1/1     Running   0          8m36s
virt-controller-7bd5c67d87-2p4fp                      1/1     Running   0          7m38s
virt-controller-7bd5c67d87-smfzx                      1/1     Running   0          7m38s
virt-handler-glfnz                                    1/1     Running   0          7m38s
virt-handler-rmpqp                                    1/1     Running   0          7m38s
virt-handler-s4rgz                                    1/1     Running   0          7m38s
virt-operator-5dbdf888f-f5sxf                         1/1     Running   0          14m
virt-operator-5dbdf888f-rgrx4                         1/1     Running   0          14m
virt-template-validator-ffb4ff754-tnrh5               1/1     Running   0          6m48s
virt-template-validator-ffb4ff754-wvsls               1/1     Running   0          6m48s
vm-import-controller-5954454bbc-bp69z                 1/1     Running   0          8m56s
vm-import-operator-696fffbff5-785cr                   1/1     Running   0          15m
[root@preserve-olm-env 1892372]# oc get rs -n kubevirt-hyperconverged
NAME                                            DESIRED   CURRENT   READY   AGE
cdi-apiserver-5558cdb57d                        1         1         1       9m
cdi-deployment-6c6798bfdd                       1         1         1       9m1s
cdi-operator-5dd7d8f78c                         1         1         1       15m
cdi-uploadproxy-79cdb778f7                      1         1         1       9m
cluster-network-addons-operator-67db4ccc65      1         1         1       15m
hco-operator-56855b88f8                         1         1         1       15m
hco-webhook-7476f997cb                          1         1         1       15m
hostpath-provisioner-operator-7cbb985658        1         1         1       15m
kubemacpool-mac-controller-manager-67c9778c74   1         1         1       9m24s
kubevirt-ssp-operator-65f9d8974d                1         1         1       15m
nmstate-webhook-7558cdf884                      2         2         2       9m24s
node-maintenance-operator-84d4758               1         1         1       15m
virt-api-8cdc565dd                              2         2         2       8m49s
virt-controller-7bd5c67d87                      2         2         2       7m51s
virt-operator-5dbdf888f                         2         2         2       15m
virt-template-validator-ffb4ff754               2         2         2       7m1s
vm-import-controller-5954454bbc                 1         1         1       9m10s
vm-import-operator-696fffbff5                   1         1         1       15m
[root@preserve-olm-env 1892372]# oc get csv -n kubevirt-hyperconverged
NAME                                      DISPLAY                                    VERSION   REPLACES   PHASE
kubevirt-hyperconverged-operator.v1.2.0   KubeVirt HyperConverged Cluster Operator   1.2.0                Succeeded
[root@preserve-olm-env 1892372]# 

--

Comment 8 errata-xmlrpc 2020-12-01 10:48:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.5.21 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5194

Comment 9 Alexander Greene 2020-12-03 19:58:42 UTC
Moving back to Verified state, QE had already verified the fix and the errata bot should not have closed this BZ.

Comment 12 errata-xmlrpc 2021-03-03 04:40:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.5.33 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0428


Note You need to log in before you can comment on or make changes to this bug.