Bug 2099808 - adding observability add-on fails with 'size of manifest exceeds the 50k limit'
Summary: adding observability add-on fails with 'size of manifest exceeds the 50k limit'
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Advanced Cluster Management for Kubernetes
Classification: Red Hat
Component: Core Services / Observability
Version: rhacm-2.5
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: ---
Assignee: Subbarao Meduri
QA Contact: Xiang Yin
URL:
Whiteboard:
Depends On:
Blocks: 2091487
TreeView+ depends on / blocked
 
Reported: 2022-06-21 18:14 UTC by dfroehli
Modified: 2022-08-16 14:26 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-08-16 12:47:42 UTC
Target Upstream Version:
Embargoed:
bot-tracker-sync: rhacm-2.5.z+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github stolostron backlog issues 23650 0 None None None 2022-06-21 21:19:57 UTC

Description dfroehli 2022-06-21 18:14:06 UTC
Description of the problem:
Observability Deployed - only the local cluster shows up in grafana.
observability-addon is not visibile / deployed to managed cluster


```
2022-06-21T15:57:36.857Z	ERROR	controller_placementrule	Failed to create manifestwork	{“error”: “admission webhook \“manifestworkvalidators.admission.work.open-cluster-management.io\” denied the request: the size of manifests is 52739 bytes which exceeds the 50k limit”}
```
This affects ALL managed clusters in this env (which uses RHV and bare metal clusters)

Obersvability is thus not working at all, hence sev=high


Release version:
RHACM 2.5.0

Operator snapshot version:
egistry.redhat.io/rhacm2/multicluster-observability-rhel8-operator@sha256:dd888f81c33bd51608b54fef35bd63de48ea0eb60502be8c497e1a63482b9a57

OCP version:
4.9.36

Browser Info:
Chrome

Additional info:
hub and managed cluster are on Red Hat VPN - contact me on slack or gchat to get access and see yourself.


observability operator log snippet.

2022-06-21T17:54:04.705Z	INFO	controller_placementrule	Monitoring operator should be installed in cluster	{"cluster_name": "stormshift-ocp10", "request.name": "stormshift-ocp10", "request.namespace": ""}
2022-06-21T17:54:04.705Z	INFO	controller_placementrule	observabilityaddon already existed/unchanged	{"namespace": "stormshift-ocp10"}
2022-06-21T17:54:04.705Z	INFO	controller_placementrule	clusterrolebinding endpoint-observability-mco-rolebinding already existed/unchanged	{"namespace": "stormshift-ocp10"}
2022-06-21T17:54:04.705Z	INFO	controller_placementrule	rolebinding endpoint-observability-res-rolebinding already existed/unchanged	{"namespace": "stormshift-ocp10"}
2022-06-21T17:54:04.713Z	INFO	controller_placementrule	Creating manifestwork	{"namespace": "stormshift-ocp10", "name": "stormshift-ocp10-observability"}
2022-06-21T17:54:04.745Z	ERROR	controller_placementrule	Failed to create manifestwork	{"namespace": "stormshift-ocp10", "name": "stormshift-ocp10-observability", "error": "admission webhook \"manifestworkvalidators.admission.work.open-cluster-management.io\" denied the request: the size of manifests is 52750 bytes which exceeds the 50k limit"}
github.com/stolostron/multicluster-observability-operator/operators/multiclusterobservability/controllers/placementrule.createManagedClusterRes
	/remote-source/app/operators/multiclusterobservability/controllers/placementrule/placementrule_controller.go:431
github.com/stolostron/multicluster-observability-operator/operators/multiclusterobservability/controllers/placementrule.createAllRelatedRes
	/remote-source/app/operators/multiclusterobservability/controllers/placementrule/placementrule_controller.go:351
github.com/stolostron/multicluster-observability-operator/operators/multiclusterobservability/controllers/placementrule.(*PlacementRuleReconciler).Reconcile
	/remote-source/app/operators/multiclusterobservability/controllers/placementrule/placementrule_controller.go:158
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311

Comment 3 dfroehli 2022-07-15 07:55:52 UTC
Thx for the workaround. Step #1 did it already for me! Regarding Step #2, I did add the annotation to the config map. is that correct?

Comment 6 llan 2022-08-09 16:58:55 UTC
@dfroehli
any update?


Note You need to log in before you can comment on or make changes to this bug.