Bug 2260331

+++ This bug was initially created as a clone of Bug #2258681 +++

Description of problem (please be detailed as possible and provide log
snippests):

Backingstore Reconcilliation happening over 2000 times an hour

Please review the code with me. The pjg(backingstore.go) and the controller(backingstore_controller.go) below that calls it.

// MapSecretToBackingStores returns a list of backingstores that uses the secret in their secretRefernce
// used by backingstore_contorller to watch secrets changes
func MapSecretToBackingStores(secret types.NamespacedName) []reconcile.Request {
	log := util.Logger()
	log.Infof("checking which backingstore to reconcile. mapping secret %v to backingstores", secret)
	bsList := &nbv1.BackingStoreList{
		TypeMeta: metav1.TypeMeta{Kind: "BackingStoreList"},
	}
	if !util.KubeList(bsList, &client.ListOptions{Namespace: secret.Namespace}) {
		log.Infof("Cloud not found backingStores in namespace %q, while trying to find Backingstore that uses %s secrte", secret.Namespace, secret.Name)
		return nil
	}

	reqs := []reconcile.Request{}

	for _, bs := range bsList.Items {
		bsSecret, err := util.GetBackingStoreSecret(&bs)
		if err != nil {
			log.Errorf(err.Error())
		}
		if bsSecret != nil && bsSecret.Name == secret.Name {
			reqs = append(reqs, reconcile.Request{
				NamespacedName: types.NamespacedName{
					Name:      bs.Name,
					Namespace: bs.Namespace,
				},
			})
		}
	}
	log.Infof("will reconcile these backingstores: %v", reqs)

	return reqs
}


and then from its controller thats calling above

	// setting another handler to watch events on secrets that not necessarily owned by the Backingstore.
	// only one OwnerReference can be a controller see:
	// https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/controller/controllerutil/controllerutil.go#L54
	secretsHandler := handler.EnqueueRequestsFromMapFunc(func(ctx context.Context, obj client.Object) []reconcile.Request {
		return backingstore.MapSecretToBackingStores(types.NamespacedName{
			Name:      obj.GetName(),
			Namespace: obj.GetNamespace(),
		})
	})
	err = c.Watch(source.Kind(mgr.GetCache(), &corev1.Secret{}), secretsHandler, logEventsPredicate)
	if err != nil {
		return err
	}


should there be a limit on how many times it checks every hour?


Version of all relevant components (if applicable): ODF 4.13


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)? it does not stop operations, it slows down operations


Is there any workaround available to the best of your knowledge? the code is working, but this is an enquiry whether or not it should be limited


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible? n/a customer environment only


Can this issue reproduce from the UI? n/a customer environment only


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.
2.
3.


Actual results:


Expected results:


Additional info:

--- Additional comment from Alexander on 2024-01-17 02:55:51 UTC ---

after viewing the code could you tell me if there should be a limit on how many times it checks every hour?

--- Additional comment from RHEL Program Management on 2024-01-17 02:56:03 UTC ---

This bug having no release flag set previously, is now set with release flag 'odf‑4.15.0' to '?', and so is being proposed to be fixed at the ODF 4.15.0 release. Note that the 3 Acks (pm_ack, devel_ack, qa_ack), if any previously set while release flag was missing, have now been reset since the Acks are to be set against a release flag.

--- Additional comment from Liran Mauda on 2024-01-17 12:05:33 UTC ---

Hi Alexander,

It works as expected, we can remove the log entry if it overloads.

NooBaa operator is in a constant cycle of reconciling as designed, it is the way all operator works, they watch for changes and act upon them, the fact that they see a message in the logs does not mean NooBaa did something upon a change. (https://bugzilla.redhat.com/show_bug.cgi?id=2255586#c10)

Please take a look at the README in the PR: https://github.com/noobaa/noobaa-operator/pull/1257

Best Regards,
Liran.

--- Additional comment from Alexander on 2024-01-19 01:39:48 UTC ---

Thanks for checking it Liran, Yes i've passed the same message to the cu 
but why is this happening to the bucket in random spurs? affecting users fail to upload files to S3


oobaa-core:

HostedAgents/35 core.rpc.ice:: TLS ERROR: tcp://10.137.18.28:0=>tcp://10.137.18.28:60365 [Error: 0070D7D4CC7F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:ssl/record/ssl3_record.c:354:
 [WebServer/38 core.rpc.ice:: TLS ERROR: tcp://[10.137.18.28]:60365=>tcp://[::ffff:10.137.18.28]:39836 Error: Client network socket disconnected before secure TLS connection was established
[WebServer/38]  core.server.node_services.nodes_monitor:: _test_network_to_server:: node has gateway_errors noobaa-internal-agent-60ddcf9325819600232dc98f [Error: N2N ICE CLOSED]
[WebServer/38]  core.rpc.rpc:: RPC._request: response ERROR srv agent_api.test_network_perf reqid 1@n2n://60ddcf9425819600232dc991(zdai35kaxw.i) connid n2n://60ddcf9425819600232dc991(zdai35kaxw.i) params { source: •[32m'n2n://nodes_monitor'•[39m, target: •[32m'n2n://60ddcf9425819600232dc991'•[39m, response_length: •[33m1•[39m }  [Error: N2N ICE CLOSED]
(node:38) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 13323)
 WebServer/38]  core.server.node_services.node_allocator:: allocate_node: no nodes for allocation in pool set [ •[32m'noobaa-default-backing-store'•[39m ] [] []
 [WebServer/38]   core.server.node_services.nodes_monitor:: noobaa-internal-agent-60ddcf9325819600232dc98f not writable. reasons: in detention (gateway_errors at Thu Jan 04 2024 14:03:16 GMT+0000 (Coordinated Universal Time)), storage is full (Thu Jan 01 1970 00:00:00 GMT+0000 (Coordinated Universal Time)) - •[33m[Duplicated message. Suppressing for 30 seconds]•[


noobaa-operator:

time="2024-01-04T01:14:56Z" level=info msg="SetPhase: Rejected" backingstore=openshift-storage/noobaa-default-backing-store
time="2024-01-04T01:15:15Z" level=info msg="SetPhase: Rejected" bucketclass=openshift-storage/noobaa-default-bucket-class

--- Additional comment from Liran Mauda on 2024-01-21 06:34:31 UTC ---

Hi Alexander,

The reconciliation is not in random spurs, and it does not affecting users fail to upload files to S3
It works as expected.

If the customer is having an issue with uploading to s3, please open a new BZ, and let it be assigned, 
Please attach to the new BZ all the details and the must-gather.

Can we close this BZ?

Best Regards,
Liran.

--- Additional comment from Liran Mauda on 2024-01-21 06:51:33 UTC ---

Hi Alexander,

I have removed the logs from the code.

Can we move this BZ to post (any other issues should be in a different BZ :) ) 

Best Regards,
Liran.

--- Additional comment from Alexander on 2024-01-22 00:56:30 UTC ---

Hi Liran,

Thanks again for your quick action on this. No problem, leave it with me to investigate further. 

Will this update where the logging is removed from the code be patched on a z-stream?

--- Additional comment from Liran Mauda on 2024-01-22 10:01:44 UTC ---

Hi Alexander,

Backporting to a z-stream is decided at the z-stream program,

You can talk to Eran Tamir

Best Regards,
Liran.

--- Additional comment from Eran Tamir on 2024-01-22 13:13:28 UTC ---

We will consider backporting it to 4.13

--- Additional comment from RHEL Program Management on 2024-01-25 12:14:03 UTC ---

This BZ is being approved for ODF 4.15.0 release, upon receipt of the 3 ACKs (PM,Devel,QA) for the release flag 'odf‑4.15.0

--- Additional comment from RHEL Program Management on 2024-01-25 12:14:03 UTC ---

Since this bug has been approved for ODF 4.15.0 release, through release flag 'odf-4.15.0+', the Target Release is being set to 'ODF 4.15.0