Bug 2094303

Summary: If og is configured with sa, operator installation will be failed.
Product: OpenShift Container Platform Reporter: xzha
Component: OLMAssignee: Vu Dinh <vdinh>
OLM sub component: OLM QA Contact: xzha
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: urgent CC: agreene, jiazha, tflannag, tyslaton, vdinh, vsolanki, vsolasa
Version: 4.11Keywords: Triaged
Target Milestone: ---Flags: agreene: needinfo-
agreene: needinfo-
Target Release: 4.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-08-10 11:16:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description xzha 2022-06-07 11:12:45 UTC
Description of problem:
If og is configured with sa, operator installation will be failed.
error message is:
time="2022-06-07T08:41:00Z" level=warning msg="skipping secret scoped-dockercfg-hnxmp - <nil>" mode=scoped namespace=test-41174 sa=scoped
time="2022-06-07T08:41:00Z" level=error msg="failed to get a client for plan execution: the service account does not have any API secret sa=test-41174/scoped"
E0607 08:41:00.302468       1 queueinformer_operator.go:290] sync {"update" "test-41174/install-pqslc"} failed: the service account does not have any API secret sa=test-41174/scoped


Version-Release number of selected component (if applicable):
zhaoxia@xzha-mac operator-framework-olm % oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-06-06-025509   True        False         11h     Cluster version is 4.11.0-0.nightly-2022-06-06-025509
zhaoxia@xzha-mac operator-framework-olm % oc adm release info registry.ci.openshift.org/ocp/release:4.11.0-0.nightly-2022-06-06-025509 --commits|grep operator-lifecycle-manager
  operator-lifecycle-manager                     https://github.com/openshift/operator-framework-olm                         7f8ad598528b2d029fac23dac6d860c433cbf962


How reproducible:
always

Steps to Reproduce:
1.install og with sa
zhaoxia@xzha-mac OCP-41174 % oc new-project test-41174
zhaoxia@xzha-mac OCP-41174 % oc apply ogwithsa.yaml
zhaoxia@xzha-mac OCP-41174 % cat ogwithsa.yaml 
kind: OperatorGroup
apiVersion: operators.coreos.com/v1
metadata:
  name: og
  namespace: test-41174
spec:
  serviceAccountName: "scoped"
  targetNamespaces:
  - test-41174
2.create sa, Role, RoleBinding
zhaoxia@xzha-mac OCP-41174 % oc create sa scoped -n test-41174
zhaoxia@xzha-mac OCP-41174 % oc apply -f role.yaml
zhaoxia@xzha-mac OCP-41174 % cat role.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: scoped
  namespace: test-41174
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"] 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: scoped-bindings
  namespace: test-41174
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: scoped
subjects:
- kind: ServiceAccount
  name: scoped
  namespace: test-41174

3.create sub
zhaoxia@xzha-mac OCP-41174 % oc apply -f sub.yaml 
zhaoxia@xzha-mac OCP-41174 % cat sub.yaml                     
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-operator
  namespace: test-41174
spec:
  channel: singlenamespace-alpha
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace

4. check status

zhaoxia@xzha-mac OCP-41174 % oc get og -o yaml
apiVersion: v1
items:
- apiVersion: operators.coreos.com/v1
  kind: OperatorGroup
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"operators.coreos.com/v1","kind":"OperatorGroup","metadata":{"annotations":{},"name":"og","namespace":"test-41174"},"spec":{"serviceAccountName":"scoped","targetNamespaces":["test-41174"]}}
    creationTimestamp: "2022-06-07T08:32:12Z"
    generation: 1
    name: og
    namespace: test-41174
    resourceVersion: "397517"
    uid: e5b265ff-8057-433e-ad3c-f7b1bcb0f15d
  spec:
    serviceAccountName: scoped
    targetNamespaces:
    - test-41174
    upgradeStrategy: Default
  status:
    lastUpdated: "2022-06-07T08:32:12Z"
    namespaces:
    - test-41174
    serviceAccountRef:
      apiVersion: v1
      kind: ServiceAccount
      name: scoped
      namespace: test-41174
      resourceVersion: "397228"
      uid: ec2d6394-dd34-45d1-8ad5-26f8dacacba1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

zhaoxia@xzha-mac OCP-41174 % oc get sa scoped -o yaml
apiVersion: v1
imagePullSecrets:
- name: scoped-dockercfg-hnxmp
kind: ServiceAccount
metadata:
  creationTimestamp: "2022-06-07T08:32:02Z"
  name: scoped
  namespace: test-41174
  resourceVersion: "397228"
  uid: ec2d6394-dd34-45d1-8ad5-26f8dacacba1
secrets:
- name: scoped-dockercfg-hnxmp

zhaoxia@xzha-mac OCP-41174 % oc get secrets scoped-dockercfg-hnxmp -o yaml
apiVersion: v1
...
metadata:
  annotations:
    kubernetes.io/service-account.name: scoped
    kubernetes.io/service-account.uid: ec2d6394-dd34-45d1-8ad5-26f8dacacba1
    openshift.io/token-secret.name: scoped-token-dl7pq
    ...
  creationTimestamp: "2022-06-07T08:32:02Z"
  name: scoped-dockercfg-hnxmp
  namespace: test-41174
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: false
    controller: true
    kind: Secret
    name: scoped-token-dl7pq
    uid: a35309d3-56c6-4214-98ef-847b7af811f4
  resourceVersion: "397226"
  uid: 9a006918-bb37-4e73-8c6c-46e9883875dc
type: kubernetes.io/dockercfg


zhaoxia@xzha-mac OCP-41174 % oc get secrets
NAME                       TYPE                                  DATA   AGE
builder-dockercfg-klq9f    kubernetes.io/dockercfg               1      167m
builder-token-xfjvf        kubernetes.io/service-account-token   4      167m
default-dockercfg-7kjcj    kubernetes.io/dockercfg               1      167m
default-token-hqxrk        kubernetes.io/service-account-token   4      167m
deployer-dockercfg-j8j8l   kubernetes.io/dockercfg               1      167m
deployer-token-nfwws       kubernetes.io/service-account-token   4      167m
scoped-dockercfg-hnxmp     kubernetes.io/dockercfg               1      167m
scoped-token-dl7pq         kubernetes.io/service-account-token   4      167m

zhaoxia@xzha-mac OCP-41174 % oc get ip -o yaml
conditions:
    - lastTransitionTime: "2022-06-07T08:33:21Z"
      lastUpdateTime: "2022-06-07T11:02:13Z"
      message: the service account does not have any API secret sa=test-41174/scoped
      reason: InstallComponentFailed
      status: "False"
      type: Installed
    message: the service account does not have any API secret sa=test-41174/scoped
    phase: Failed


Actual results:
installplan failed due to "the service account does not have any API secret"

Expected results:
installplan success.

Additional info:

catalog:
time="2022-06-07T08:41:00Z" level=warning msg="skipping secret scoped-dockercfg-hnxmp - <nil>" mode=scoped namespace=test-41174 sa=scoped
time="2022-06-07T08:41:00Z" level=error msg="failed to get a client for plan execution: the service account does not have any API secret sa=test-41174/scoped"
E0607 08:41:00.302468       1 queueinformer_operator.go:290] sync {"update" "test-41174/install-pqslc"} failed: the service account does not have any API secret sa=test-41174/scoped

Comment 1 Jian Zhang 2022-06-07 11:31:52 UTC
If the OperatorGroup doesn't refer to a SA, it works well. Like below:

mac:operator-framework-olm jianzhang$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-06-06-025509   True        False         11h     Cluster version is 4.11.0-0.nightly-2022-06-06-025509

mac:operator-framework-olm jianzhang$ oc get og operator-sdk-og -o yaml
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  annotations:
    olm.providedAPIs: Nginx.v1.helmdemo.example.com
  creationTimestamp: "2022-06-07T09:48:33Z"
  generation: 1
  name: operator-sdk-og
  namespace: default
  resourceVersion: "528053"
  uid: 15f8e283-e084-475d-be62-2ea4f91317db
spec:
  upgradeStrategy: Default
status:
  lastUpdated: "2022-06-07T09:48:33Z"
  namespaces:
  - ""

mac:operator-framework-olm jianzhang$ oc get sub
NAME               PACKAGE   SOURCE          CHANNEL
nginx-v0-0-1-sub   nginx     nginx-catalog   alpha
mac:operator-framework-olm jianzhang$ oc get ip
NAME            CSV            APPROVAL   APPROVED
install-7j8vg   nginx.v0.0.1   Manual     true
mac:operator-framework-olm jianzhang$ oc get csv
NAME           DISPLAY   VERSION   REPLACES   PHASE
nginx.v0.0.1   nginx     0.0.1                Succeeded
mac:operator-framework-olm jianzhang$ oc get sa
NAME                       SECRETS   AGE
builder                    1         12h
default                    1         12h
deployer                   1         12h
garbage                    1         4h57m
nginx-controller-manager   1         100m
nginx-sample               1         70m
mac:operator-framework-olm jianzhang$ oc get sa nginx-controller-manager -o yaml
apiVersion: v1
imagePullSecrets:
- name: nginx-controller-manager-dockercfg-d8rsq
kind: ServiceAccount
metadata:
  creationTimestamp: "2022-06-07T09:49:07Z"
  labels:
    operators.coreos.com/nginx.default: ""
  name: nginx-controller-manager
  namespace: default
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: ClusterServiceVersion
    name: nginx.v0.0.1
    uid: 30124412-b2fe-483e-b526-533f596883a8
  resourceVersion: "528156"
  uid: 77df0b99-bfd9-4208-b422-3c5cf2a8e82b
secrets:
- name: nginx-controller-manager-dockercfg-d8rsq

mac:operator-framework-olm jianzhang$ oc get secret nginx-controller-manager-dockercfg-d8rsq -o=jsonpath={.type}
kubernetes.io/dockercfg

Comment 9 xzha 2022-06-20 08:30:42 UTC
verify:
zhaoxia@xzha-mac ocp-41174 % oc get clusterversion
NAME      VERSION                                                   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.ci.test-2022-06-20-070726-ci-ln-m8gl1b2-latest   True        False         39m     Cluster version is 4.11.0-0.ci.test-2022-06-20-070726-ci-ln-m8gl1b2-latest

zhaoxia@xzha-mac ocp-41174 % oc exec catalog-operator-6c8cc9465-842c6  -- olm --version
OLM version: 0.19.0
git commit: 496e4d7b3baa9e8b1e0201eed0838cd5297bb6e2

1, zhaoxia@xzha-mac ocp-41174 % oc new-project test-41174

2, create og, sa, role, rolebinding
zhaoxia@xzha-mac ocp-41174 % oc apply -f ogwithsa.yaml 
operatorgroup.operators.coreos.com/og created
zhaoxia@xzha-mac ocp-41174 % oc create sa scoped -n test-41174
serviceaccount/scoped created
zhaoxia@xzha-mac ocp-41174 % oc apply -f role.yaml
role.rbac.authorization.k8s.io/scoped created
rolebinding.rbac.authorization.k8s.io/scoped-bindings created

3, create sub 
zhaoxia@xzha-mac ocp-41174 % oc apply -f sub.yaml 
subscription.operators.coreos.com/etcd-operator created 
zhaoxia@xzha-mac ocp-41174 % cat sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-operator
  namespace: test-41174
spec:
  channel: singlenamespace-alpha
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace

4, check csv
zhaoxia@xzha-mac ocp-41174 % oc get csv -n test-41174
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.4   etcd      0.9.4     etcdoperator.v0.9.2   Succeeded

LGTM, verified.

Comment 11 xzha 2022-06-21 07:17:03 UTC
verify:
zhaoxia@xzha-mac ocp-41174 % oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-06-21-040754   True        False         5m38s   Cluster version is 4.11.0-0.nightly-2022-06-21-040754
zhaoxia@xzha-mac ocp-41174 % oc adm release info registry.ci.openshift.org/ocp/release:4.11.0-0.nightly-2022-06-21-040754 --commits|grep operator-lifecycle-manager
  operator-lifecycle-manager                     https://github.com/openshift/operator-framework-olm                         a8dabfadab5ed258254c4f55309dcd9ee39679df

1,  oc new-project test-41174

2, create og, sa, role, rolebinding
zhaoxia@xzha-mac ocp-41174 % oc apply -f ogwithsa.yaml 
operatorgroup.operators.coreos.com/og created
zhaoxia@xzha-mac ocp-41174 % oc create sa scoped -n test-41174
serviceaccount/scoped created
zhaoxia@xzha-mac ocp-41174 % oc apply -f role.yaml
role.rbac.authorization.k8s.io/scoped created
rolebinding.rbac.authorization.k8s.io/scoped-bindings created

3, create sub 
zhaoxia@xzha-mac ocp-41174 % oc apply -f sub.yaml 
subscription.operators.coreos.com/etcd-operator created 
zhaoxia@xzha-mac ocp-41174 % cat sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: etcd-operator
  namespace: test-41174
spec:
  channel: singlenamespace-alpha
  installPlanApproval: Automatic
  name: etcd
  source: community-operators
  sourceNamespace: openshift-marketplace

4, check csv
zhaoxia@xzha-mac ocp-41174 % oc get csv
NAME                  DISPLAY   VERSION   REPLACES              PHASE
etcdoperator.v0.9.4   etcd      0.9.4     etcdoperator.v0.9.2   Succeeded


LGTM, verified.

Comment 13 errata-xmlrpc 2022-08-10 11:16:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069

Comment 31 Red Hat Bugzilla 2023-12-31 04:25:16 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days