Created attachment 1752804 [details]
Nodes description, oc get cm, oc get secrets and file  02848245.txt w multiple info

Description of problem:

Customer is using Openstack Cinder as storage infrastructure. 
Currently it was verified that Cinder and OpenStack is not communicating properly anymore especially after upgrading from 4.5.2 to 4.5.27, causing to fail the creation of PVC with:

Pending Status
- Waiting to first consumer to be created before binding

The same happens to whenever trying to create a persistent storage even with sample applications.

I found these lines in the kube-controller-manager
It looks that the pv created is: pvc-a4a959c2-79fe-461f-b7a6-745876e65959  -- 

Now, this 'Operation for "delete-pvc-a4a959c2-79fe-461f-b7a6-745876e65959[61bed42a-7ba7-4cf6-8424-5f725fe2b67b]" failed.' is what bugs me -  Looks like it is trying to delete.
Checking the sosreport from the node I didnt find any reference regarding the ES volumes, specially the PV.

2021-01-27T13:06:50.081631814Z I0127 13:06:50.081180       1 pv_cont2021-01-27T13:06:50.081670453Z roller.go:1285] isVolumeReleased[pvc-a4a959c2-79fe-461f-b7a6-745876e65959]: volume is released
    2021-01-27T13:06:50.081670453Z W0127 13:06:50.081189       1 plugins.go:721] WARNING: kubernetes.io/cinder built-in volume provider is now deprecated. The Cinder volume provider is deprecated and will be removed in a future release
    2021-01-27T13:06:50.081670453Z E0127 13:06:50.081307       1 goroutinemap.go:150] Operation for "delete-pvc-a4a959c2-79fe-461f-b7a6-745876e65959[61bed42a-7ba7-4cf6-8424-5f725fe2b67b]" failed. No retries permitted until 2021-01-27 13:06:50.581238771 +0000 UTC m=+16487.315893317 (durationBeforeRetry 500ms). Error: "recovered from panic \"runtime error: invalid memory address or nil pointer dereference\". (err=<nil>) Call stack:\ngoroutine 52734 [running]:\ngithub.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime.RecoverFromPanic(0xc007b0ffd0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:158 +0xb5\npanic(0x3c3cca0, 0x74c02e0)\n\t/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/panic.go:679 +0x1b2\ngithub.com/openshift/origin/vendor/github.com/gophercloud/gophercloud/openstack.initClientOpts(0x0, 0x44b5a3e, 0x8, 0x0, 0x0, 0xc0017cbda0, 0x8, 0x44b188d, 0x6, 0x44b5a3e, ...)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/gophercloud/gophercloud/openstack/client.go:297 +0x96\ngithub.com/openshift/origin/vendor/github.com/gophercloud/gophercloud/openstack.NewBlockStorageV3(...)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/gophercloud/gophercloud/openstack/client.go:353\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).NewBlockStorageV3(0xc001616f00, 0x0, 0x4b860b4, 0xc0000649e0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack_client.go:74 +0xc7\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).volumeService(0xc001616f00, 0x0, 0x0, 0xc0000649e0, 0xc007b0f850, 0x435e91, 0x4662c58)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack.go:913 +0x4e6\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).getVolume(0xc001616f00, 0xc0030af830, 0x24, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack_volumes.go:456 +0x76\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).diskIsUsed(0xc001616f00, 0xc0030af830, 0x24, 0xc007b0fa08, 0x73, 0xc007b0faf0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack_volumes.go:705 +0x5a\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).DeleteVolume(0xc001616f00, 0xc0030af830, 0x24, 0x0, 0x0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack_volumes.go:584 +0x4c\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/volume/cinder.(*DiskUtil).DeleteVolume(0x751c810, 0xc00befe510, 0xbffc773a84d6e5bd, 0x74fadc0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/volume/cinder/cinder_util.go:139 +0x8d\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/volume/cinder.(*cinderVolumeDeleter).Delete(0xc00befe510, 0x14, 0x44c1a64)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/volume/cinder/cinder.go:553 +0x3e\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume.(*PersistentVolumeController).doDeleteVolume(0xc00068e000, 0xc00a447180, 0xc0000cc001, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1344 +0x3c8\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume.(*PersistentVolumeController).deleteVolumeOperation(0xc00068e000, 0xc00a447180, 0x0, 0x2000000000040, 0x200010001, 0x8)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1201 +0x29b\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume.(*PersistentVolumeController).reclaimVolume.func2(0xc000187ed0, 0xc000187e40)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1058 +0x41\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/goroutinemap.(*goRoutineMap).Run.func1(0xc000d17290, 0xc00a0e26c0, 0x55, 0xc00a17ed60, 0x0, 0x0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:115 +0x12b\ncreated by github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/goroutinemap.(*goRoutineMap).Run\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:108 +0x2ea\n"


Version-Release number of selected component (if applicable):

4.5.27

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:

The PV Is not created

Expected results:

PV to be created

Additional info:

File 02848245.txt from compressed file provides:
- describe ES pod
- get pods -o wide -n openshift-logging
- clouds.yaml
- osp credentials
- describe pvc
- oc get pv (no pv)
- describe storage class

Must-gather and sosreport from infra node are provided in supportshell

Let me know if there is something else that we can provide.