Description of problem: Hi, When working on an operator I've detected an issue when trying to upgrade it. The operator has been created with operator-sdk version 1.2.0. In this version, a K8s Service object is automatically included as part of the operator's bundle: ``` msoriano@localhost:~/go/src/github.com/3scale/apicast-operator (master)$ cat bundle/manifests/apicast-operator-controller-manager-metrics-service_v1_service.yaml apiVersion: v1 kind: Service metadata: creationTimestamp: null labels: control-plane: controller-manager name: apicast-operator-controller-manager-metrics-service spec: ports: - name: https port: 8443 targetPort: https selector: control-plane: controller-manager status: loadBalancer: {} ``` When this Service is included in several Bundle versions and an upgrade of the operator is performed OLM fails to upgrade the operator due to the Service with the following error in the new InstallPlan ``` msoriano@localhost:~/go/src/github.com/3scale/apicast-operator (checkupgradebundle)$ oc get installplan install-d4skh -o json | jq .status.conditions [ { "lastTransitionTime": "2020-11-13T18:25:14Z", "lastUpdateTime": "2020-11-13T18:25:14Z", "message": "error updating service: apicast-operator-controller-manager-metrics-service: Service \"apicast-operator-controller-manager-metrics-service\" is invalid: [metadata.resourceVersion: Invalid value: \"\": must be specified for an update, spec.clusterIP: Invalid value: \"\": field is immutable]", "reason": "InstallComponentFailed", "status": "False", "type": "Installed" } ] ``` This seems to effectively prevent upgrading any operator between two versions that include a Service object as part of its Bundle. Version-Release number of selected component (if applicable): I've reproduced the issue in OpenShift 4.6.1 operator-sdk v1.2.0 was used to work with the operator How reproducible: 1. Create an operator bundle with CSV version 1.0.0 defined in channel v1. This bundle must include a K8s Service object among its manifests. 2. Create an operator bundle with CSV version 1.0.1 that replaces 1.0.0 in channel v2. This bundle must include the same K8s Service object that was included in step 1 too. 3. Create an index image that includes both bundles 4. Deploy a custom CatalogSource pointing to this image 5. Install the operator on channel v1 and wait until it is installed and running 6. Go to the operator's Subscription and change channel to v2 to trigger an upgrade 7. See how the new InstallPlan fails to upgrade due to the reported issue Actual results: The upgrade fails to upgrade due to an error in the InstallPlan related to the included Service object. Expected results: The upgrade is performed successfully
It appears that the service object being generated by the operator-sdk is failing validation when OLM is attempting to apply this to the cluster. I'm reassigning this bug to the Operator SDK since it seems that the problem is with that specific Service, and not anything OLM is doing to apply services in the general case.
Reassigning this back to OLM. I absolutely misread this, not realizing this is an upgrade from one version of the service manifest to a second. In this case, OLM should definitely be handling an update to the spec of the service, especially if the underlying manifest hasn't actually changed at all.
verify it on 4.7. LGTM -- [root@preserve-olm-env operator-registry]# tree manifests/teiid-1898500 manifests/teiid-1898500 |-- 0.0.1 | |-- teiid.0.0.1.clusterserviceversion.yaml | `-- virtualdatabases.teiid.io.crd.yaml |-- 0.1.0 | |-- teiid.0.1.0.clusterserviceversion.yaml | `-- virtualdatabase.crd.yaml |-- 0.1.1 | |-- teiid.0.1.1.clusterserviceversion.yaml | `-- virtualdatabase.crd.yaml |-- 0.2.0 | |-- teiid.io_virtualdatabases_crd.yaml | `-- teiid.v0.2.0.clusterserviceversion.yaml |-- 0.3.0 | |-- teiid.io_virtualdatabases_crd.yaml | |-- teiid_service.yaml | `-- teiid.v0.3.0.clusterserviceversion.yaml |-- 0.4.0 | |-- teiid.io_virtualdatabases_crd.yaml | |-- teiid_service.yaml | `-- teiid.v0.4.0.clusterserviceversion.yaml |-- ci.yaml `-- teiid.package.yaml 6 directories, 16 files [root@preserve-olm-env operator-registry]# cat manifests/teiid-1898500/0.3.0/teiid_service.yaml apiVersion: v1 kind: Service metadata: creationTimestamp: null labels: app: teiid-operator name: teiid-service spec: ports: - name: https port: 8443 targetPort: https selector: app: teiid-operator status: loadBalancer: {} [root@preserve-olm-env operator-registry]# cat manifests/teiid-1898500/0.4.0/teiid_service.yaml apiVersion: v1 kind: Service metadata: creationTimestamp: null labels: app: teiid-operator name: teiid-service spec: ports: - name: https port: 8443 targetPort: https selector: app: teiid-operator status: loadBalancer: {} [root@preserve-olm-env operator-registry]# [root@preserve-olm-env operator-registry]# ./bin/opm alpha bundle build --directory /root/kuiwang/operator-registry/manifests/teiid-1898500/0.3.0 --tag quay.io/kuiwang/teiid-operator:v1898500-3 -p teiid -c alpha -e alpha INFO[0000] Building annotations.yaml INFO[0000] Writing annotations.yaml in /root/kuiwang/operator-registry/manifests/teiid-1898500/metadata INFO[0000] Building Dockerfile INFO[0000] Writing bundle.Dockerfile in /root/kuiwang/operator-registry INFO[0000] Building bundle image Sending build context to Docker daemon 117.6MB Step 1/9 : FROM scratch ---> Step 2/9 : LABEL operators.operatorframework.io.bundle.mediatype.v1=registry+v1 ---> Using cache ---> 17f4d6cc02f6 Step 3/9 : LABEL operators.operatorframework.io.bundle.manifests.v1=manifests/ ---> Using cache ---> ed5b62e609a0 Step 4/9 : LABEL operators.operatorframework.io.bundle.metadata.v1=metadata/ ---> Using cache ---> 958a7490fbd5 Step 5/9 : LABEL operators.operatorframework.io.bundle.package.v1=teiid ---> Using cache ---> 660a23efdfcf Step 6/9 : LABEL operators.operatorframework.io.bundle.channels.v1=alpha ---> Running in 52324d71259a Removing intermediate container 52324d71259a ---> 89d62d90f67d Step 7/9 : LABEL operators.operatorframework.io.bundle.channel.default.v1=alpha ---> Running in 6eb437988134 Removing intermediate container 6eb437988134 ---> 10ffde972c83 Step 8/9 : COPY manifests/teiid-1898500/0.3.0 /manifests/ ---> 9f0f3474803a Step 9/9 : COPY manifests/teiid-1898500/metadata /metadata/ ---> bf886260d189 Successfully built bf886260d189 Successfully tagged quay.io/kuiwang/teiid-operator:v1898500-3 [root@preserve-olm-env operator-registry]# docker push quay.io/kuiwang/teiid-operator:v1898500-3 The push refers to repository [quay.io/kuiwang/teiid-operator] f733c016c96f: Pushed 90b0a18e463d: Pushed v1898500-3: digest: sha256:43457b119b4332054cdd27746366fe197987cf4d3f54079d4dcc4c41f33aa34e size: 733 [root@preserve-olm-env operator-registry]# rm -fr bundle.Dockerfile manifests/teiid-1898500/metadata/ [root@preserve-olm-env operator-registry]# [root@preserve-olm-env operator-registry]# ./bin/opm alpha bundle build --directory /root/kuiwang/operator-registry/manifests/teiid-1898500/0.4.0 --tag quay.io/kuiwang/teiid-operator:v1898500-4 -p teiid -c beta -e beta INFO[0000] Building annotations.yaml INFO[0000] Writing annotations.yaml in /root/kuiwang/operator-registry/manifests/teiid-1898500/metadata INFO[0000] Building Dockerfile INFO[0000] Writing bundle.Dockerfile in /root/kuiwang/operator-registry INFO[0000] Building bundle image Sending build context to Docker daemon 117.6MB Step 1/9 : FROM scratch ---> Step 2/9 : LABEL operators.operatorframework.io.bundle.mediatype.v1=registry+v1 ---> Using cache ---> 17f4d6cc02f6 Step 3/9 : LABEL operators.operatorframework.io.bundle.manifests.v1=manifests/ ---> Using cache ---> ed5b62e609a0 Step 4/9 : LABEL operators.operatorframework.io.bundle.metadata.v1=metadata/ ---> Using cache ---> 958a7490fbd5 Step 5/9 : LABEL operators.operatorframework.io.bundle.package.v1=teiid ---> Using cache ---> 660a23efdfcf Step 6/9 : LABEL operators.operatorframework.io.bundle.channels.v1=beta ---> Using cache ---> c841860d7e05 Step 7/9 : LABEL operators.operatorframework.io.bundle.channel.default.v1=beta ---> Using cache ---> add28127624d Step 8/9 : COPY manifests/teiid-1898500/0.4.0 /manifests/ ---> c76d8d190c3b Step 9/9 : COPY manifests/teiid-1898500/metadata /metadata/ ---> 73fa5e8e73e7 Successfully built 73fa5e8e73e7 Successfully tagged quay.io/kuiwang/teiid-operator:v1898500-4 [root@preserve-olm-env operator-registry]# docker push quay.io/kuiwang/teiid-operator:v1898500-4 The push refers to repository [quay.io/kuiwang/teiid-operator] bd961919d71f: Pushed 2cb0281357ce: Pushed v1898500-4: digest: sha256:a391c30fbce9e159eea312aa8193936bbbb637e9ae98087246828ee121aaa788 size: 733 [root@preserve-olm-env operator-registry]# rm -fr bundle.Dockerfile manifests/teiid-1898500/metadata/ [root@preserve-olm-env operator-registry]# [root@preserve-olm-env operator-registry]# ./bin/opm index add --bundles quay.io/kuiwang/teiid-operator:v1898500-3 --tag quay.io/kuiwang/teiid-index:1898500 -c docker INFO[0000] building the index bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]" INFO[0000] running /usr/bin/docker pull quay.io/kuiwang/teiid-operator:v1898500-3 bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]" INFO[0000] running docker create bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]" INFO[0000] running docker cp bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]" INFO[0000] running docker rm bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]" INFO[0000] Could not find optional dependencies file dir=bundle_tmp154897146 file=bundle_tmp154897146/metadata load=annotations INFO[0000] found csv, loading bundle dir=bundle_tmp154897146 file=bundle_tmp154897146/manifests load=bundle INFO[0000] loading bundle file dir=bundle_tmp154897146/manifests file=teiid.io_virtualdatabases_crd.yaml load=bundle INFO[0000] loading bundle file dir=bundle_tmp154897146/manifests file=teiid.v0.3.0.clusterserviceversion.yaml load=bundle INFO[0000] loading bundle file dir=bundle_tmp154897146/manifests file=teiid_service.yaml load=bundle INFO[0001] Generating dockerfile bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]" INFO[0001] writing dockerfile: index.Dockerfile121648264 bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]" INFO[0001] running docker build bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]" INFO[0001] [docker build -f index.Dockerfile121648264 -t quay.io/kuiwang/teiid-index:1898500 .] bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]" [root@preserve-olm-env operator-registry]# docker push quay.io/kuiwang/teiid-index:1898500 The push refers to repository [quay.io/kuiwang/teiid-index] b8a0e050ae65: Pushed 53b2ccb878f2: Mounted from operator-framework/upstream-opm-builder 772b412a6a9d: Mounted from operator-framework/upstream-opm-builder 371d4702865c: Layer already exists 4570b0d18853: Layer already exists ace0eda3e3be: Layer already exists 1898500: digest: sha256:c2d7fcab488f615fd356060a804cf2f145249105547a9e0b2813615de7f1620f size: 1578 [root@preserve-olm-env operator-registry]# ./bin/opm index add --bundles quay.io/kuiwang/teiid-operator:v1898500-4 --from-index quay.io/kuiwang/teiid-index:1898500 --tag quay.io/kuiwang/teiid-index:1898500 -c docker INFO[0000] building the index bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0000] Pulling previous image quay.io/kuiwang/teiid-index:1898500 to get metadata bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0000] running /usr/bin/docker pull quay.io/kuiwang/teiid-index:1898500 bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0000] running /usr/bin/docker pull quay.io/kuiwang/teiid-index:1898500 bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0001] Getting label data from previous image bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0001] running docker inspect bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0001] running docker create bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0001] running docker cp bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0001] running docker rm bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0001] running /usr/bin/docker pull quay.io/kuiwang/teiid-operator:v1898500-4 bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0002] running docker create bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0002] running docker cp bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0002] running docker rm bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0002] Could not find optional dependencies file dir=bundle_tmp866806378 file=bundle_tmp866806378/metadata load=annotations INFO[0002] found csv, loading bundle dir=bundle_tmp866806378 file=bundle_tmp866806378/manifests load=bundle INFO[0002] loading bundle file dir=bundle_tmp866806378/manifests file=teiid.io_virtualdatabases_crd.yaml load=bundle INFO[0002] loading bundle file dir=bundle_tmp866806378/manifests file=teiid.v0.4.0.clusterserviceversion.yaml load=bundle INFO[0002] loading bundle file dir=bundle_tmp866806378/manifests file=teiid_service.yaml load=bundle INFO[0002] Generating dockerfile bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0002] writing dockerfile: index.Dockerfile291380344 bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0002] running docker build bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" INFO[0002] [docker build -f index.Dockerfile291380344 -t quay.io/kuiwang/teiid-index:1898500 .] bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]" [root@preserve-olm-env operator-registry]# docker push quay.io/kuiwang/teiid-index:1898500 The push refers to repository [quay.io/kuiwang/teiid-index] e42f38b23694: Pushed 53b2ccb878f2: Layer already exists 772b412a6a9d: Layer already exists 371d4702865c: Layer already exists 4570b0d18853: Layer already exists ace0eda3e3be: Layer already exists 1898500: digest: sha256:0c3d4787022d27d80efaa3f3ab25ef648dcd1501a6bd8e8d6d042a7623f564ce size: 1578 [root@preserve-olm-env operator-registry]# [root@preserve-olm-env operator-registry]# cd /root/kuiwang/cases/1898500 [root@preserve-olm-env 1898500]# [root@preserve-olm-env 1898500]# oc get pod -n openshift-operator-lifecycle-manager NAME READY STATUS RESTARTS AGE catalog-operator-568d4985c4-44sl7 1/1 Running 0 77m olm-operator-5c9bd9984c-swpz7 1/1 Running 0 77m packageserver-55ff9fbf89-grp25 1/1 Running 0 78m packageserver-55ff9fbf89-n6k5w 1/1 Running 0 79m [root@preserve-olm-env 1898500]# oc exec catalog-operator-568d4985c4-44sl7 -n openshift-operator-lifecycle-manager -- olm --version OLM version: 0.17.0 git commit: d701e9a1361f636445d02140c63763248a7b95f9 [root@preserve-olm-env 1898500]# [root@preserve-olm-env 1898500]# cat og-single.yaml kind: OperatorGroup apiVersion: operators.coreos.com/v1 metadata: name: og-single1 namespace: default spec: targetNamespaces: - default [root@preserve-olm-env 1898500]# oc apply -f og-single.yaml operatorgroup.operators.coreos.com/og-single1 created [root@preserve-olm-env 1898500]# cat teiidcatsrc.yaml apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: teiid namespace: default spec: displayName: "teiid Operators" image: quay.io/kuiwang/teiid-index:1898500 publisher: QE sourceType: grpc [root@preserve-olm-env 1898500]# oc apply -f teiidcatsrc.yaml catalogsource.operators.coreos.com/teiid created [root@preserve-olm-env 1898500]# cat teiidsub.yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: teiid namespace: default spec: source: teiid sourceNamespace: default channel: alpha installPlanApproval: Automatic name: teiid [root@preserve-olm-env 1898500]# oc apply -f teiidsub.yaml subscription.operators.coreos.com/teiid created [root@preserve-olm-env 1898500]# [root@preserve-olm-env 1898500]# oc get csv NAME DISPLAY VERSION REPLACES PHASE teiid.v0.3.0 Teiid 0.3.0 Succeeded [root@preserve-olm-env 1898500]# oc get svc teiid-service -o yaml apiVersion: v1 kind: Service metadata: creationTimestamp: "2020-12-04T07:31:01Z" labels: app: teiid-operator operators.coreos.com/teiid.default: "" ... name: teiid-service namespace: default ownerReferences: - apiVersion: operators.coreos.com/v1alpha1 blockOwnerDeletion: false controller: false kind: ClusterServiceVersion name: teiid.v0.3.0 uid: 1ab3e6cb-43c9-498c-9949-7da541f57e2a resourceVersion: "59529" selfLink: /api/v1/namespaces/default/services/teiid-service uid: 29c5cad6-5216-47b2-9d9f-69f1ebe98e3f spec: clusterIP: 172.30.199.96 ports: - name: https port: 8443 protocol: TCP targetPort: https selector: app: teiid-operator sessionAffinity: None type: ClusterIP status: loadBalancer: {} [root@preserve-olm-env 1898500]# oc patch sub teiid -n default --type merge -p '{"spec": {"channel": "beta"}}' subscription.operators.coreos.com/teiid patched [root@preserve-olm-env 1898500]# [root@preserve-olm-env 1898500]# oc get sub NAME PACKAGE SOURCE CHANNEL teiid teiid teiid beta [root@preserve-olm-env 1898500]# oc get ip NAME CSV APPROVAL APPROVED install-2rmh2 teiid.v0.4.0 Automatic true install-k29vx teiid.v0.3.0 Automatic true [root@preserve-olm-env 1898500]# oc get csv NAME DISPLAY VERSION REPLACES PHASE teiid.v0.4.0 Teiid 0.4.0 teiid.v0.3.0 Succeeded [root@preserve-olm-env 1898500]# oc get svc teiid-service -o yaml apiVersion: v1 kind: Service metadata: creationTimestamp: "2020-12-04T07:31:01Z" labels: app: teiid-operator operators.coreos.com/teiid.default: "" ... name: teiid-service namespace: default ownerReferences: - apiVersion: operators.coreos.com/v1alpha1 blockOwnerDeletion: false controller: false kind: ClusterServiceVersion name: teiid.v0.4.0 uid: ea26c02b-092b-4aa4-8a14-f07f04afbfbf resourceVersion: "62156" selfLink: /api/v1/namespaces/default/services/teiid-service uid: 29c5cad6-5216-47b2-9d9f-69f1ebe98e3f spec: clusterIP: 172.30.199.96 ports: - name: https port: 8443 protocol: TCP targetPort: https selector: app: teiid-operator sessionAffinity: None type: ClusterIP status: loadBalancer: {} --
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633