Bug 2085083

Summary:	Policies fail to copy to cluster namespace after ACM upgrade
Product:	Red Hat Advanced Cluster Management for Kubernetes	Reporter:	Ian Miller <imiller>
Component:	App Lifecycle	Assignee:	Xiangjing Li <xiangli>
Status:	CLOSED ERRATA	QA Contact:	Almen Ng <almng>
Severity:	high	Docs Contact:	bswope <bswope>
Priority:	unspecified
Version:	rhacm-2.4.z	CC:	dho, ecai, fdewaley, jgato, jramsay, xiangli, yuhe
Target Milestone:	---	Flags:	dho: qe_test_coverage+ bot-tracker-sync: rhacm-2.5+
Target Release:	rhacm-2.5
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-06-09 02:12:04 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ian Miller 2022-05-12 18:43:07 UTC

Description of the problem: After an ACM upgrade from 2.4.3 to 2.4.4 policies are no longer copied into the cluster namespace.

The affected policies are created by one of our operators and prior to the ACM upgrade worked correctly. Specifically, when the controller creates the policy, placementrule, and placementbinding the child policy is created in the cluster namespace. After the upgrade the child policy never shows up in the cluster namespace. After doing some digging we found error messages like this in the multicluster-operators-application-7c44cfbd5d-4vpsw multicluster-operators-placementrule container:

E0510 19:55:55.956629 1 placementrule_controller.go:226] Status update -.ztp-install/cnfde21-common-cnfde21-config-policy with err:placementrules.apps.open-cluster-management.io "cnfde21-common-cnfde21-config-policy" is forbidden: User "system:serviceaccount:openshift-cluster-group-upgrades:cluster-group-upgrades-controller-manager" cannot update resource "placementrules/status" in API group "apps.open-cluster-management.io" in the namespace "ztp-install"

The referenced ServiceAccount is bound to a Role which provides access to the necessary resources. Our operator creates this SA and role with (among others):
- apiGroups:
- apps.open-cluster-management.io
resources:
- placementrules
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
When I add in placementrules/status the errors in the multicluster-operators-application are resolved and the child policies are immediately created as expected.

This issue appears to happen only after an upgrade of ACM. We saw it first (from recollection, we did not get to root cause on the first occurrence) on a hub cluster when ACM upgraded from 2.4.2 to 2.4.3, and then this week on a hub which upgraded from 2.4.3 to 2.4.4.

Release version: 2.4.4

Operator snapshot version:

OCP version: 4.9.21

Browser Info:

Steps to reproduce:
1.
2.
3.

Actual results:

Expected results:

Additional info:

Comment 5 errata-xmlrpc 2022-06-09 02:12:04 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Advanced Cluster Management 2.5 security updates, images, and bug fixes), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:4956

Comment 6 Jose Gato 2022-06-09 07:43:44 UTC

Hi,
I see the issue has been fixed on ACM 2.5. Is it planned to be fixed also on 2.4? I am using RHACM2.4.4 and I was affected by that bug (fixed with a workaround). But, when a new RHACM2.4.5 is available, because of the upgrade, would I have the problem again?