Bug 1842399 - Operator add multiple ownerRefs to packageserver-service service with the same UUID
Summary: Operator add multiple ownerRefs to packageserver-service service with the sam...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.5
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.6.0
Assignee: Nick Hale
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-01 08:05 UTC by Michal Fojtik
Modified: 2020-10-27 16:03 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: OLM appeneded OwnerReferences to Service resources installed for operators that provide APIServices. Consequence: Whenever an operator of this class was redeployed by OLM -- e.g. during a cert rotation -- a duplicate OwnerReference was appended to the related Service; the number of OwnerReferences grows unbounded. Fix: Use Set semantics when adding OwnerReferences by updating an existing OwnerReference if found. Result: The number of OwnerReferences appended to a Service by OLM is bounded.
Clone Of:
Environment:
Last Closed: 2020-10-27 16:02:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:03:28 UTC

Description Michal Fojtik 2020-06-01 08:05:52 UTC
Description of problem:

While investigating other problems, I came across this line in kube-apiserver log:

E0601 02:15:24.358789       1 structuredmerge.go:103] [SHOULD NOT HAPPEN] failed to create typed new object of type /v1, Kind=Service: errors:
  .metadata.ownerReferences: duplicate entries for key [uid="b0fb0082-8ec4-4206-8cdc-1faad5774dad"]
  .metadata.ownerReferences: duplicate entries for key [uid="b0fb0082-8ec4-4206-8cdc-1faad5774dad"]
  .metadata.ownerReferences: duplicate entries for key [uid="b0fb0082-8ec4-4206-8cdc-1faad5774dad"]

This led me to search for that UUID in services.json, where I found:

{
            "apiVersion": "v1",
            "kind": "Service",
            "metadata": {
                "creationTimestamp": "2020-06-01T02:20:46Z",
                "name": "packageserver-service",
                "namespace": "openshift-operator-lifecycle-manager",
                "ownerReferences": [
                    {
                        "apiVersion": "operators.coreos.com/v1alpha1",
                        "blockOwnerDeletion": false,
                        "controller": false,
                        "kind": "ClusterServiceVersion",
                        "name": "packageserver",
                        "uid": "b0fb0082-8ec4-4206-8cdc-1faad5774dad"
                    },
                    {
                        "apiVersion": "operators.coreos.com/v1alpha1",
                        "blockOwnerDeletion": false,
                        "controller": false,
                        "kind": "ClusterServiceVersion",
                        "name": "packageserver",
                        "uid": "b0fb0082-8ec4-4206-8cdc-1faad5774dad"
                    },
                    {
                        "apiVersion": "operators.coreos.com/v1alpha1",
                        "blockOwnerDeletion": false,
                        "controller": false,
                        "kind": "ClusterServiceVersion",
                        "name": "packageserver",
                        "uid": "b0fb0082-8ec4-4206-8cdc-1faad5774dad"
                    },
                    {
                        "apiVersion": "operators.coreos.com/v1alpha1",
                        "blockOwnerDeletion": false,
                        "controller": false,
                        "kind": "ClusterServiceVersion",
                        "name": "packageserver",
                        "uid": "b0fb0082-8ec4-4206-8cdc-1faad5774dad"
                    },
                    {
                        "apiVersion": "operators.coreos.com/v1alpha1",
                        "blockOwnerDeletion": false,
                        "controller": false,
                        "kind": "ClusterServiceVersion",
                        "name": "packageserver",
                        "uid": "b0fb0082-8ec4-4206-8cdc-1faad5774dad"
                    },
                    {
                        "apiVersion": "operators.coreos.com/v1alpha1",
                        "blockOwnerDeletion": false,
                        "controller": false,
                        "kind": "ClusterServiceVersion",
                        "name": "packageserver",
                        "uid": "b0fb0082-8ec4-4206-8cdc-1faad5774dad"
                    },
                    {
                        "apiVersion": "operators.coreos.com/v1alpha1",
                        "blockOwnerDeletion": false,
                        "controller": false,
                        "kind": "ClusterServiceVersion",
                        "name": "packageserver",
                        "uid": "b0fb0082-8ec4-4206-8cdc-1faad5774dad"
                    },
                    {
                        "apiVersion": "operators.coreos.com/v1alpha1",
                        "blockOwnerDeletion": false,
                        "controller": false,
                        "kind": "ClusterServiceVersion",
                        "name": "packageserver",
                        "uid": "b0fb0082-8ec4-4206-8cdc-1faad5774dad"
                    },
                    {
                        "apiVersion": "operators.coreos.com/v1alpha1",
                        "blockOwnerDeletion": false,
                        "controller": false,
                        "kind": "ClusterServiceVersion",
                        "name": "packageserver",
                        "uid": "b0fb0082-8ec4-4206-8cdc-1faad5774dad"
                    }
                ],


This show a bug in operator that incorrectly update the ownerRefs in the service.
The kube-apiserver should stop this, but unfortunately upstream probably missed this case and changing this now will likely break existing objects.


Version-Release number of selected component (if applicable):

4.x

How reproducible:

Check the services.

Steps to Reproduce:
1.
2.
3.

Actual results:

Multiple owner references found in packageserver-service with single UUID.

Expected results:

Only one ownerRef should exists with one UUID.

Additional info:

Comment 9 Jian Zhang 2020-07-30 06:54:31 UTC
Cluster version is 4.6.0-0.nightly-2020-07-29-210846
OLM version:
mac:~ jianzhang$ oc exec catalog-operator-6cd8478667-qqxzn -- olm --version
OLM version: 0.16.0
git commit: 69478e6938b1737585df8cfbd3b4eaf06401b3d2

1, check the "packageserver-service" service.
mac:~ jianzhang$ oc get svc -n openshift-operator-lifecycle-manager packageserver-service -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2020-07-30T06:02:32Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:ownerReferences:
          .: {}
          k:{"uid":"a7c98b73-9c84-4d4d-89ac-fcec086319fe"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:spec:
        f:ports:
          .: {}
          k:{"port":5443,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {}
          f:app: {}
        f:sessionAffinity: {}
        f:type: {}
    manager: olm
    operation: Update
    time: "2020-07-30T06:02:32Z"
  name: packageserver-service
  namespace: openshift-operator-lifecycle-manager
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: ClusterServiceVersion
    name: packageserver
    uid: a7c98b73-9c84-4d4d-89ac-fcec086319fe
  resourceVersion: "36418"
  selfLink: /api/v1/namespaces/openshift-operator-lifecycle-manager/services/packageserver-service
  uid: 8fb92c63-80ec-4ea1-a5f7-f6d91995cbda
spec:
  clusterIP: 172.30.44.128
  ports:
  - name: "5443"
    port: 5443
    protocol: TCP
    targetPort: 5443
  selector:
    app: packageserver
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

mac:~ jianzhang$ oc get csv -n openshift-operator-lifecycle-manager
NAME            DISPLAY          VERSION   REPLACES   PHASE
packageserver   Package Server   0.15.1               Succeeded

Only one CSV referenced, LGTM, verify it.

Comment 11 errata-xmlrpc 2020-10-27 16:02:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.