Bug 1898500

Summary: Failure to upgrade operator when a Service is included in a Bundle
Product: OpenShift Container Platform Reporter: Miguel Soriano <msoriano>
Component: OLMAssignee: Ben Luddy <bluddy>
OLM sub component: OLM QA Contact: kuiwang
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: aos-bugs, bluddy, krizza, msoriano, sople, ttaylor
Version: 4.6Keywords: Triaged, UpcomingSprint
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:34:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1920873    

Description Miguel Soriano 2020-11-17 10:50:27 UTC 
Description of problem:

Hi,

When working on an operator I've detected an issue when trying to upgrade it.

The operator has been created with operator-sdk version 1.2.0. In this version, a K8s Service object is automatically included as part of the operator's bundle:

```
msoriano@localhost:~/go/src/github.com/3scale/apicast-operator (master)$ cat bundle/manifests/apicast-operator-controller-manager-metrics-service_v1_service.yaml 
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    control-plane: controller-manager
  name: apicast-operator-controller-manager-metrics-service
spec:
  ports:
  - name: https
    port: 8443
    targetPort: https
  selector:
    control-plane: controller-manager
status:
  loadBalancer: {}
```

When this Service is included in several Bundle versions and an upgrade of the operator is performed OLM fails to upgrade the operator due to the Service with the following error in the new InstallPlan

```
msoriano@localhost:~/go/src/github.com/3scale/apicast-operator (checkupgradebundle)$ oc get installplan install-d4skh -o json | jq .status.conditions
[
  {
    "lastTransitionTime": "2020-11-13T18:25:14Z",
    "lastUpdateTime": "2020-11-13T18:25:14Z",
    "message": "error updating service: apicast-operator-controller-manager-metrics-service: Service \"apicast-operator-controller-manager-metrics-service\" is invalid: [metadata.resourceVersion: Invalid value: \"\": must be specified for an update, spec.clusterIP: Invalid value: \"\": field is immutable]",
    "reason": "InstallComponentFailed",
    "status": "False",
    "type": "Installed"
  }
]
```

This seems to effectively prevent upgrading any operator between two versions that include a Service object as part of its Bundle.

Version-Release number of selected component (if applicable):

I've reproduced the issue in OpenShift 4.6.1
operator-sdk v1.2.0 was used to work with the operator

How reproducible:

1. Create an operator bundle with CSV version 1.0.0 defined in channel v1. This bundle must include a K8s Service object among its manifests.
2. Create an operator bundle with CSV version 1.0.1 that replaces 1.0.0 in channel v2. This bundle must include the same K8s Service object that was included in step 1 too.
3. Create an index image that includes both bundles
4. Deploy a custom CatalogSource pointing to this image
5. Install the operator on channel v1 and wait until it is installed and running
6. Go to the operator's Subscription and change channel to v2 to trigger an upgrade
7. See how the new InstallPlan fails to upgrade due to the reported issue

Actual results:

The upgrade fails to upgrade due to an error in the InstallPlan related to the included Service object.

Expected results:

The upgrade is performed successfully
Comment 1 Kevin Rizza 2020-11-17 16:31:48 UTC 
It appears that the service object being generated by the operator-sdk is failing validation when OLM is attempting to apply this to the cluster. I'm reassigning this bug to the Operator SDK since it seems that the problem is with that specific Service, and not anything OLM is doing to apply services in the general case.
Comment 2 Kevin Rizza 2020-11-17 16:46:02 UTC 
Reassigning this back to OLM. I absolutely misread this, not realizing this is an upgrade from one version of the service manifest to a second. In this case, OLM should definitely be handling an update to the spec of the service, especially if the underlying manifest hasn't actually changed at all.
Comment 14 kuiwang 2020-12-04 07:42:18 UTC 
verify it on 4.7. LGTM
--
[root@preserve-olm-env operator-registry]# tree manifests/teiid-1898500
manifests/teiid-1898500
|-- 0.0.1
|   |-- teiid.0.0.1.clusterserviceversion.yaml
|   `-- virtualdatabases.teiid.io.crd.yaml
|-- 0.1.0
|   |-- teiid.0.1.0.clusterserviceversion.yaml
|   `-- virtualdatabase.crd.yaml
|-- 0.1.1
|   |-- teiid.0.1.1.clusterserviceversion.yaml
|   `-- virtualdatabase.crd.yaml
|-- 0.2.0
|   |-- teiid.io_virtualdatabases_crd.yaml
|   `-- teiid.v0.2.0.clusterserviceversion.yaml
|-- 0.3.0
|   |-- teiid.io_virtualdatabases_crd.yaml
|   |-- teiid_service.yaml
|   `-- teiid.v0.3.0.clusterserviceversion.yaml
|-- 0.4.0
|   |-- teiid.io_virtualdatabases_crd.yaml
|   |-- teiid_service.yaml
|   `-- teiid.v0.4.0.clusterserviceversion.yaml
|-- ci.yaml
`-- teiid.package.yaml

6 directories, 16 files
[root@preserve-olm-env operator-registry]# cat manifests/teiid-1898500/0.3.0/teiid_service.yaml 
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: teiid-operator
  name: teiid-service
spec:
  ports:
  - name: https
    port: 8443
    targetPort: https
  selector:
    app: teiid-operator
status:
  loadBalancer: {}
[root@preserve-olm-env operator-registry]# cat manifests/teiid-1898500/0.4.0/teiid_service.yaml 
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: teiid-operator
  name: teiid-service
spec:
  ports:
  - name: https
    port: 8443
    targetPort: https
  selector:
    app: teiid-operator
status:
  loadBalancer: {}
[root@preserve-olm-env operator-registry]# 


[root@preserve-olm-env operator-registry]# ./bin/opm alpha bundle build --directory /root/kuiwang/operator-registry/manifests/teiid-1898500/0.3.0 --tag quay.io/kuiwang/teiid-operator:v1898500-3 -p teiid -c alpha -e alpha
INFO[0000] Building annotations.yaml                    
INFO[0000] Writing annotations.yaml in /root/kuiwang/operator-registry/manifests/teiid-1898500/metadata 
INFO[0000] Building Dockerfile                          
INFO[0000] Writing bundle.Dockerfile in /root/kuiwang/operator-registry 
INFO[0000] Building bundle image                        
Sending build context to Docker daemon  117.6MB
Step 1/9 : FROM scratch
 ---> 
Step 2/9 : LABEL operators.operatorframework.io.bundle.mediatype.v1=registry+v1
 ---> Using cache
 ---> 17f4d6cc02f6
Step 3/9 : LABEL operators.operatorframework.io.bundle.manifests.v1=manifests/
 ---> Using cache
 ---> ed5b62e609a0
Step 4/9 : LABEL operators.operatorframework.io.bundle.metadata.v1=metadata/
 ---> Using cache
 ---> 958a7490fbd5
Step 5/9 : LABEL operators.operatorframework.io.bundle.package.v1=teiid
 ---> Using cache
 ---> 660a23efdfcf
Step 6/9 : LABEL operators.operatorframework.io.bundle.channels.v1=alpha
 ---> Running in 52324d71259a
Removing intermediate container 52324d71259a
 ---> 89d62d90f67d
Step 7/9 : LABEL operators.operatorframework.io.bundle.channel.default.v1=alpha
 ---> Running in 6eb437988134
Removing intermediate container 6eb437988134
 ---> 10ffde972c83
Step 8/9 : COPY manifests/teiid-1898500/0.3.0 /manifests/
 ---> 9f0f3474803a
Step 9/9 : COPY manifests/teiid-1898500/metadata /metadata/
 ---> bf886260d189
Successfully built bf886260d189
Successfully tagged quay.io/kuiwang/teiid-operator:v1898500-3
[root@preserve-olm-env operator-registry]# docker push quay.io/kuiwang/teiid-operator:v1898500-3
The push refers to repository [quay.io/kuiwang/teiid-operator]
f733c016c96f: Pushed 
90b0a18e463d: Pushed 
v1898500-3: digest: sha256:43457b119b4332054cdd27746366fe197987cf4d3f54079d4dcc4c41f33aa34e size: 733
[root@preserve-olm-env operator-registry]# rm -fr bundle.Dockerfile manifests/teiid-1898500/metadata/
[root@preserve-olm-env operator-registry]# 


[root@preserve-olm-env operator-registry]# ./bin/opm alpha bundle build --directory /root/kuiwang/operator-registry/manifests/teiid-1898500/0.4.0 --tag quay.io/kuiwang/teiid-operator:v1898500-4 -p teiid -c beta -e beta
INFO[0000] Building annotations.yaml                    
INFO[0000] Writing annotations.yaml in /root/kuiwang/operator-registry/manifests/teiid-1898500/metadata 
INFO[0000] Building Dockerfile                          
INFO[0000] Writing bundle.Dockerfile in /root/kuiwang/operator-registry 
INFO[0000] Building bundle image                        
Sending build context to Docker daemon  117.6MB
Step 1/9 : FROM scratch
 ---> 
Step 2/9 : LABEL operators.operatorframework.io.bundle.mediatype.v1=registry+v1
 ---> Using cache
 ---> 17f4d6cc02f6
Step 3/9 : LABEL operators.operatorframework.io.bundle.manifests.v1=manifests/
 ---> Using cache
 ---> ed5b62e609a0
Step 4/9 : LABEL operators.operatorframework.io.bundle.metadata.v1=metadata/
 ---> Using cache
 ---> 958a7490fbd5
Step 5/9 : LABEL operators.operatorframework.io.bundle.package.v1=teiid
 ---> Using cache
 ---> 660a23efdfcf
Step 6/9 : LABEL operators.operatorframework.io.bundle.channels.v1=beta
 ---> Using cache
 ---> c841860d7e05
Step 7/9 : LABEL operators.operatorframework.io.bundle.channel.default.v1=beta
 ---> Using cache
 ---> add28127624d
Step 8/9 : COPY manifests/teiid-1898500/0.4.0 /manifests/
 ---> c76d8d190c3b
Step 9/9 : COPY manifests/teiid-1898500/metadata /metadata/
 ---> 73fa5e8e73e7
Successfully built 73fa5e8e73e7
Successfully tagged quay.io/kuiwang/teiid-operator:v1898500-4
[root@preserve-olm-env operator-registry]# docker push quay.io/kuiwang/teiid-operator:v1898500-4
The push refers to repository [quay.io/kuiwang/teiid-operator]
bd961919d71f: Pushed 
2cb0281357ce: Pushed 
v1898500-4: digest: sha256:a391c30fbce9e159eea312aa8193936bbbb637e9ae98087246828ee121aaa788 size: 733
[root@preserve-olm-env operator-registry]# rm -fr bundle.Dockerfile manifests/teiid-1898500/metadata/
[root@preserve-olm-env operator-registry]# 

[root@preserve-olm-env operator-registry]# ./bin/opm index add --bundles quay.io/kuiwang/teiid-operator:v1898500-3 --tag quay.io/kuiwang/teiid-index:1898500 -c docker
INFO[0000] building the index                            bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]"
INFO[0000] running /usr/bin/docker pull quay.io/kuiwang/teiid-operator:v1898500-3  bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]"
INFO[0000] running docker create                         bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]"
INFO[0000] running docker cp                             bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]"
INFO[0000] running docker rm                             bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]"
INFO[0000] Could not find optional dependencies file     dir=bundle_tmp154897146 file=bundle_tmp154897146/metadata load=annotations
INFO[0000] found csv, loading bundle                     dir=bundle_tmp154897146 file=bundle_tmp154897146/manifests load=bundle
INFO[0000] loading bundle file                           dir=bundle_tmp154897146/manifests file=teiid.io_virtualdatabases_crd.yaml load=bundle
INFO[0000] loading bundle file                           dir=bundle_tmp154897146/manifests file=teiid.v0.3.0.clusterserviceversion.yaml load=bundle
INFO[0000] loading bundle file                           dir=bundle_tmp154897146/manifests file=teiid_service.yaml load=bundle
INFO[0001] Generating dockerfile                         bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]"
INFO[0001] writing dockerfile: index.Dockerfile121648264  bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]"
INFO[0001] running docker build                          bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]"
INFO[0001] [docker build -f index.Dockerfile121648264 -t quay.io/kuiwang/teiid-index:1898500 .]  bundles="[quay.io/kuiwang/teiid-operator:v1898500-3]"
[root@preserve-olm-env operator-registry]# docker push quay.io/kuiwang/teiid-index:1898500
The push refers to repository [quay.io/kuiwang/teiid-index]
b8a0e050ae65: Pushed 
53b2ccb878f2: Mounted from operator-framework/upstream-opm-builder 
772b412a6a9d: Mounted from operator-framework/upstream-opm-builder 
371d4702865c: Layer already exists 
4570b0d18853: Layer already exists 
ace0eda3e3be: Layer already exists 
1898500: digest: sha256:c2d7fcab488f615fd356060a804cf2f145249105547a9e0b2813615de7f1620f size: 1578
[root@preserve-olm-env operator-registry]# ./bin/opm index add --bundles quay.io/kuiwang/teiid-operator:v1898500-4 --from-index quay.io/kuiwang/teiid-index:1898500 --tag quay.io/kuiwang/teiid-index:1898500 -c docker
INFO[0000] building the index                            bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0000] Pulling previous image quay.io/kuiwang/teiid-index:1898500 to get metadata  bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0000] running /usr/bin/docker pull quay.io/kuiwang/teiid-index:1898500  bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0000] running /usr/bin/docker pull quay.io/kuiwang/teiid-index:1898500  bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0001] Getting label data from previous image        bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0001] running docker inspect                        bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0001] running docker create                         bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0001] running docker cp                             bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0001] running docker rm                             bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0001] running /usr/bin/docker pull quay.io/kuiwang/teiid-operator:v1898500-4  bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0002] running docker create                         bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0002] running docker cp                             bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0002] running docker rm                             bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0002] Could not find optional dependencies file     dir=bundle_tmp866806378 file=bundle_tmp866806378/metadata load=annotations
INFO[0002] found csv, loading bundle                     dir=bundle_tmp866806378 file=bundle_tmp866806378/manifests load=bundle
INFO[0002] loading bundle file                           dir=bundle_tmp866806378/manifests file=teiid.io_virtualdatabases_crd.yaml load=bundle
INFO[0002] loading bundle file                           dir=bundle_tmp866806378/manifests file=teiid.v0.4.0.clusterserviceversion.yaml load=bundle
INFO[0002] loading bundle file                           dir=bundle_tmp866806378/manifests file=teiid_service.yaml load=bundle
INFO[0002] Generating dockerfile                         bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0002] writing dockerfile: index.Dockerfile291380344  bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0002] running docker build                          bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
INFO[0002] [docker build -f index.Dockerfile291380344 -t quay.io/kuiwang/teiid-index:1898500 .]  bundles="[quay.io/kuiwang/teiid-operator:v1898500-4]"
[root@preserve-olm-env operator-registry]# docker push quay.io/kuiwang/teiid-index:1898500
The push refers to repository [quay.io/kuiwang/teiid-index]
e42f38b23694: Pushed 
53b2ccb878f2: Layer already exists 
772b412a6a9d: Layer already exists 
371d4702865c: Layer already exists 
4570b0d18853: Layer already exists 
ace0eda3e3be: Layer already exists 
1898500: digest: sha256:0c3d4787022d27d80efaa3f3ab25ef648dcd1501a6bd8e8d6d042a7623f564ce size: 1578
[root@preserve-olm-env operator-registry]# 
[root@preserve-olm-env operator-registry]# cd /root/kuiwang/cases/1898500
[root@preserve-olm-env 1898500]# 
[root@preserve-olm-env 1898500]# oc get pod -n openshift-operator-lifecycle-manager
NAME                                READY   STATUS    RESTARTS   AGE
catalog-operator-568d4985c4-44sl7   1/1     Running   0          77m
olm-operator-5c9bd9984c-swpz7       1/1     Running   0          77m
packageserver-55ff9fbf89-grp25      1/1     Running   0          78m
packageserver-55ff9fbf89-n6k5w      1/1     Running   0          79m
[root@preserve-olm-env 1898500]# oc exec catalog-operator-568d4985c4-44sl7 -n openshift-operator-lifecycle-manager -- olm --version
OLM version: 0.17.0
git commit: d701e9a1361f636445d02140c63763248a7b95f9
[root@preserve-olm-env 1898500]# 
[root@preserve-olm-env 1898500]# cat og-single.yaml 
kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
  name: og-single1
  namespace: default
spec:
  targetNamespaces:
  - default
[root@preserve-olm-env 1898500]# oc apply -f og-single.yaml 
operatorgroup.operators.coreos.com/og-single1 created
[root@preserve-olm-env 1898500]# cat teiidcatsrc.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: teiid
  namespace: default
spec:
  displayName: "teiid Operators"
  image: quay.io/kuiwang/teiid-index:1898500
  publisher: QE
  sourceType: grpc
[root@preserve-olm-env 1898500]# oc apply -f teiidcatsrc.yaml 
catalogsource.operators.coreos.com/teiid created


[root@preserve-olm-env 1898500]# cat teiidsub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: teiid
  namespace: default
spec:
  source: teiid
  sourceNamespace: default

  channel: alpha
  installPlanApproval: Automatic
  name: teiid
[root@preserve-olm-env 1898500]# oc apply -f teiidsub.yaml 
subscription.operators.coreos.com/teiid created
[root@preserve-olm-env 1898500]# 


[root@preserve-olm-env 1898500]# oc get csv
NAME           DISPLAY   VERSION   REPLACES   PHASE
teiid.v0.3.0   Teiid     0.3.0                Succeeded

[root@preserve-olm-env 1898500]# oc get svc teiid-service -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2020-12-04T07:31:01Z"
  labels:
    app: teiid-operator
    operators.coreos.com/teiid.default: ""
...
  name: teiid-service
  namespace: default
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: ClusterServiceVersion
    name: teiid.v0.3.0
    uid: 1ab3e6cb-43c9-498c-9949-7da541f57e2a
  resourceVersion: "59529"
  selfLink: /api/v1/namespaces/default/services/teiid-service
  uid: 29c5cad6-5216-47b2-9d9f-69f1ebe98e3f
spec:
  clusterIP: 172.30.199.96
  ports:
  - name: https
    port: 8443
    protocol: TCP
    targetPort: https
  selector:
    app: teiid-operator
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

[root@preserve-olm-env 1898500]# oc patch sub teiid -n default --type merge -p '{"spec": {"channel": "beta"}}'
subscription.operators.coreos.com/teiid patched
[root@preserve-olm-env 1898500]# 

[root@preserve-olm-env 1898500]# oc get sub
NAME    PACKAGE   SOURCE   CHANNEL
teiid   teiid     teiid    beta
[root@preserve-olm-env 1898500]# oc get ip
NAME            CSV            APPROVAL    APPROVED
install-2rmh2   teiid.v0.4.0   Automatic   true
install-k29vx   teiid.v0.3.0   Automatic   true
[root@preserve-olm-env 1898500]# oc get csv
NAME           DISPLAY   VERSION   REPLACES       PHASE
teiid.v0.4.0   Teiid     0.4.0     teiid.v0.3.0   Succeeded
[root@preserve-olm-env 1898500]# oc get svc teiid-service -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2020-12-04T07:31:01Z"
  labels:
    app: teiid-operator
    operators.coreos.com/teiid.default: ""
...
  name: teiid-service
  namespace: default
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: ClusterServiceVersion
    name: teiid.v0.4.0
    uid: ea26c02b-092b-4aa4-8a14-f07f04afbfbf
  resourceVersion: "62156"
  selfLink: /api/v1/namespaces/default/services/teiid-service
  uid: 29c5cad6-5216-47b2-9d9f-69f1ebe98e3f
spec:
  clusterIP: 172.30.199.96
  ports:
  - name: https
    port: 8443
    protocol: TCP
    targetPort: https
  selector:
    app: teiid-operator
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}


--
Comment 18 errata-xmlrpc 2021-02-24 15:34:16 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633