Bug 2085237

Summary: Resources referenced by a channel are not annotated with backup label
Product: Red Hat Advanced Cluster Management for Kubernetes Reporter: Thuy Nguyen <thnguyen>
Component: App LifecycleAssignee: Xiangjing Li <xiangli>
Status: CLOSED ERRATA QA Contact:
Severity: high Docs Contact: bswope <bswope>
Priority: unspecified    
Version: rhacm-2.5CC: ming, yuhe
Target Milestone: ---Flags: bot-tracker-sync: rhacm-2.5+
Target Release: rhacm-2.5   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-06-09 02:12:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Thuy Nguyen 2022-05-12 20:48:46 UTC
Description of the problem: Resources referenced by a channel are not annotated with backup label

Release version:

Operator snapshot version:
ACM 2.5.0-DOWNSTREAM-2022-05-10-21-30-52

OCP version:

Browser Info:

Steps to reproduce:
1. Create app from private git repo with custom ca cert


Actual results:
Configmap and secret referenced by the channel are not annotated with backup labels

Expected results:

Additional info:

oc get channel -n gst09-0installdev09red-chesterfieldcom-testadmin-testrepo-ns gst09-0installdev09red-chesterfieldcom-testadmin-testrepo -oyaml
apiVersion: apps.open-cluster-management.io/v1
kind: Channel
metadata:
  annotations:
    apps.open-cluster-management.io/reconcile-rate: medium
    open-cluster-management.io/user-group: c3lzdGVtOmNsdXN0ZXItYWRtaW5zLHN5c3RlbTphdXRoZW50aWNhdGVk
    open-cluster-management.io/user-identity: a3ViZTphZG1pbg==
  creationTimestamp: "2022-05-12T20:11:51Z"
  generation: 1
  name: gst09-0installdev09red-chesterfieldcom-testadmin-testrepo
  namespace: gst09-0installdev09red-chesterfieldcom-testadmin-testrepo-ns
  resourceVersion: "937438"
  uid: 92c097cf-9c31-4881-8797-74611c7c8110
spec:
  configMapRef:
    name: git-ca
  pathname: https://gogs-svc-default.apps.mist09-0.install.dev09.red-chesterfield.com/testadmin/testrepo.git
  secretRef:
    name: gst09-0installdev09red-chesterfieldcom-testadmin-testrepo-auth
  type: Git


oc get cm -n gst09-0installdev09red-chesterfieldcom-testadmin-testrepo-ns git-ca -oyaml
apiVersion: v1
data:
  caCerts: |
    <certificate>
kind: ConfigMap
metadata:
  creationTimestamp: "2022-05-12T20:17:33Z"
  name: git-ca
  namespace: gst09-0installdev09red-chesterfieldcom-testadmin-testrepo-ns
  resourceVersion: "960865"
  uid: 3182ddf0-3f7f-4535-906d-4b46258da4f3


oc get secret -n gst09-0installdev09red-chesterfieldcom-testadmin-testrepo-ns  gst09-0installdev09red-chesterfieldcom-testadmin-testrepo-auth -oyaml
apiVersion: v1
data:
  accessToken: <token>
  user: <user>
kind: Secret
metadata:
  creationTimestamp: "2022-05-12T20:11:51Z"
  name: gst09-0installdev09red-chesterfieldcom-testadmin-testrepo-auth
  namespace: gst09-0installdev09red-chesterfieldcom-testadmin-testrepo-ns
  resourceVersion: "937445"
  uid: 57e579f8-1666-4c44-aea9-713e2d8d169d
type: Opaque

Comment 1 Mike Ng 2022-05-13 14:15:48 UTC
Still investigating but in the log it shows the secret and the configmap was not found.

{"level":"error","ts":"2022-05-12T20:11:51.695Z","logger":"controllers.channel","caller":"channel/channel_controller.go:258","msg":"failed to update referred secret label","error":"failed to get the reference object Secret: secrets \"gst09-0installdev09red-chesterfieldcom-testadmin-testrepo-auth\" not found","errorVerbose":"secrets \"gst09-0installdev09red-chesterfieldcom-testadmin-testrepo-auth\" not found

{"level":"error","ts":"2022-05-12T20:11:51.856Z","logger":"controllers.channel","caller":"channel/channel_controller.go:274","msg":"failed to update referred configMap label","error":"failed to get the reference object ConfigMap: configmaps \"git-ca\" not found","errorVerbose":"configmaps \"git-ca\" not found\nfailed to get the reference object ConfigMap

Comment 2 bot-tracker-sync 2022-05-17 17:56:18 UTC
G2Bsync 1129031708 comment 
 thuyn-581 Tue, 17 May 2022 15:45:51 UTC 
 G2BSync -
Retest failed on 2.5.0-RC3. Referenced configmap still does not have the label added.

```
# oc get channel -n gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo-ns gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo -ojsonpath='{.spec}' | jq .
{
  "configMapRef": {
    "name": "git-ca"
  },
  "pathname": "https://gogs-svc-default.apps.ocp4-az-dr-2.az.dev06.red-chesterfield.com/testadmin/testrepo.git",
  "secretRef": {
    "name": "gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo-auth"
  },
  "type": "Git"
}

# oc get cm -n gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo-ns git-ca -oyaml
apiVersion: v1
data:
  caCerts: |
    ---
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      ---
  creationTimestamp: "2022-05-17T15:33:30Z"
  name: git-ca
  namespace: gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo-ns
  resourceVersion: "93169632"
  uid: 7f17e118-a4d2-4e66-b4c0-a475ae5d974f
```

channel log -
```
{"level":"info","ts":"2022-05-17T15:31:57.129Z","logger":"controllers.channel","caller":"channel/channel_controller.go:312","msg":"Set label serving-channel to object: gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo-ns/gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo-auth","channel-reconcile":"gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo-ns/gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo"}
{"level":"error","ts":"2022-05-17T15:31:57.333Z","logger":"controllers.channel","caller":"channel/channel_controller.go:274","msg":"failed to update referred configMap label","error":"failed to get the reference object ConfigMap: configmaps \"git-ca\" not found","errorVerbose":"configmaps \"git-ca\" not found\nfailed to get the reference object ConfigMap\nopen-cluster-management.io/multicloud-operators-channel/pkg/controller/channel.(*ReconcileChannel).updatedReferencedObjectLabels\n\t/remote-source/app/pkg/controller/channel/channel_controller.go:297\nopen-cluster-management.io/multicloud-operators-channel/pkg/controller/channel.(*ReconcileChannel).handleReferencedObjects\n\t/remote-source/app/pkg/controller/channel/channel_controller.go:273\nopen-cluster-management.io/multicloud-operators-channel/pkg/controller/channel.(*ReconcileChannel).Reconcile\n\t/remote-source/app/pkg/controller/channel/channel_controller.go:236\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1581","stacktrace":"open-cluster-management.io/multicloud-operators-channel/pkg/controller/channel.(*ReconcileChannel).handleReferencedObjects\n\t/remote-source/app/pkg/controller/channel/channel_controller.go:274\nopen-cluster-management.io/multicloud-operators-channel/pkg/controller/channel.(*ReconcileChannel).Reconcile\n\t/remote-source/app/pkg/controller/channel/channel_controller.go:236\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.0/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2022-05-17T15:31:57.392Z","logger":"controllers.channel","caller":"channel/channel_controller.go:527","msg":"The channel gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo-ns/gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo is not in the ACM Namespace ocm, skipping...","channel-reconcile":"gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo-ns/gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo"}
{"level":"info","ts":"2022-05-17T15:31:57.392Z","logger":"controllers.channel","caller":"channel/channel_controller.go:244","msg":"Finish channel reconcile loop for gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo-ns/gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo","channel-reconcile":"gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo-ns/gcp4-az-dr-2azdev06red-chesterfieldcom-testadmin-testrepo"}

```

Comment 3 bot-tracker-sync 2022-05-17 17:56:19 UTC
G2Bsync 1129083812 comment 
 mikeshng Tue, 17 May 2022 16:35:36 UTC 
 G2Bsync
Hi Brian @smithbw88 ,
Could you please check if the fix is included in RC3? 
I checked the code in the release 2.5 branch:
https://github.com/stolostron/multicloud-operators-channel/blob/release-2.5/pkg/controller/channel/channel_controller.go#L274
and it shouldn't be possible to show this error message 
```
... channel_controller.go:274","msg":"failed to update referred ...
```
Thanks!
@thuyn-581 FYI.

Comment 4 bot-tracker-sync 2022-05-18 16:12:06 UTC
G2Bsync 1130202866 comment 
 thuyn-581 Wed, 18 May 2022 15:59:29 UTC 
 G2BSync-
Retest passed in 2.5.0-RC4.

Comment 8 errata-xmlrpc 2022-06-09 02:12:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Advanced Cluster Management 2.5 security updates, images, and bug fixes), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:4956