Bug 2023614

Summary: ERROR deleter could not get provisioner configmap
Product: OpenShift Container Platform Reporter: Shubham Naik <shunaik>
Component: StorageAssignee: Hemant Kumar <hekumar>
Storage sub component: Local Storage Operator QA Contact: Penghao Wang <pewang>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: unspecified CC: aos-bugs, bzvonar, hekumar, jsafrane, keyoung, rsandu
Version: 4.9   
Target Milestone: ---   
Target Release: 4.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-29 07:16:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2001765    
Bug Blocks:    

Description Shubham Naik 2021-11-16 07:38:41 UTC
Description of problem:
--
If for some reason 2021-11-11T12:29:06.603Z ERROR   deleter could not get provisioner configmap {"Request.Namespace": "", "Request.Name": "local-pv-795631ca", "error": "ConfigMap \"local-provisioner\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/src/github.com/openshift/local-storage-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/src/github.com/openshift/local-storage-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/src/github.com/openshift/local-storage-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214
--

Ericsson just reproduced this again. New logs linked

Comment 2 Jan Safranek 2021-11-16 13:37:01 UTC
I think more complete logs are in comment #18 in https://bugzilla.redhat.com/show_bug.cgi?id=2014083#c18.

To me it looks like that the config map was being deleted and recreated constantly. Sometimes it does not exist (see above), sometimes it does:

2021-11-12T11:08:45.528Z        INFO    deleter Deleting Pvs through sig storage deleter        {"Request.Namespace": "openshift-local-storage", "Request.Name": ""}
I1112 11:08:45.528803 2304574 deleter.go:195] Start cleanup for pv local-pv-f8110370
I1112 11:08:45.528893 2304574 deleter.go:275] Deleting PV block volume "local-pv-f8110370" device hostpath "/mnt/local-storage/general/dm-name-autopart-lv_31", mountpath "/mnt/local-storage/general/dm-name-autopart-lv_31"
2021-11-12T11:08:45.528Z        DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"PersistentVolume","name":"local-pv-f8110370","uid":"6a3e653d-1467-43f2-a600-da14dddf5c25","apiVersion":"v1","resourceVersion":"12309229"}, "reason": "VolumeDelete", "message": "Starting cleanup of Block PV \"local-pv-f8110370\", this may take a while"}

Comment 5 Hemant Kumar 2022-03-04 17:33:17 UTC
I took a look and this is happening because on first run code does not have `ownerNamespace` set properly for PVs. So it is looking for ConfigMap in wrong namespace - like "" vs "openshift-local-storage" namespace. We should fix this, but the error should not affect any end user feature.

Comment 6 Hemant Kumar 2022-03-04 22:28:19 UTC
Aaaand I found this was fixed here - https://github.com/openshift/local-storage-operator/pull/278

Comment 7 Hemant Kumar 2022-03-04 22:31:59 UTC
Opened https://github.com/openshift/local-storage-operator/pull/331 to fix

Comment 13 errata-xmlrpc 2022-03-29 07:16:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.26 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1022