Created attachment 1752804[details]
Nodes description, oc get cm, oc get secrets and file 02848245.txt w multiple info
Description of problem:
Customer is using Openstack Cinder as storage infrastructure.
Currently it was verified that Cinder and OpenStack is not communicating properly anymore especially after upgrading from 4.5.2 to 4.5.27, causing to fail the creation of PVC with:
Pending Status
- Waiting to first consumer to be created before binding
The same happens to whenever trying to create a persistent storage even with sample applications.
I found these lines in the kube-controller-manager
It looks that the pv created is: pvc-a4a959c2-79fe-461f-b7a6-745876e65959 --
Now, this 'Operation for "delete-pvc-a4a959c2-79fe-461f-b7a6-745876e65959[61bed42a-7ba7-4cf6-8424-5f725fe2b67b]" failed.' is what bugs me - Looks like it is trying to delete.
Checking the sosreport from the node I didnt find any reference regarding the ES volumes, specially the PV.
2021-01-27T13:06:50.081631814Z I0127 13:06:50.081180 1 pv_cont2021-01-27T13:06:50.081670453Z roller.go:1285] isVolumeReleased[pvc-a4a959c2-79fe-461f-b7a6-745876e65959]: volume is released
2021-01-27T13:06:50.081670453Z W0127 13:06:50.081189 1 plugins.go:721] WARNING: kubernetes.io/cinder built-in volume provider is now deprecated. The Cinder volume provider is deprecated and will be removed in a future release
2021-01-27T13:06:50.081670453Z E0127 13:06:50.081307 1 goroutinemap.go:150] Operation for "delete-pvc-a4a959c2-79fe-461f-b7a6-745876e65959[61bed42a-7ba7-4cf6-8424-5f725fe2b67b]" failed. No retries permitted until 2021-01-27 13:06:50.581238771 +0000 UTC m=+16487.315893317 (durationBeforeRetry 500ms). Error: "recovered from panic \"runtime error: invalid memory address or nil pointer dereference\". (err=<nil>) Call stack:\ngoroutine 52734 [running]:\ngithub.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime.RecoverFromPanic(0xc007b0ffd0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:158 +0xb5\npanic(0x3c3cca0, 0x74c02e0)\n\t/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/panic.go:679 +0x1b2\ngithub.com/openshift/origin/vendor/github.com/gophercloud/gophercloud/openstack.initClientOpts(0x0, 0x44b5a3e, 0x8, 0x0, 0x0, 0xc0017cbda0, 0x8, 0x44b188d, 0x6, 0x44b5a3e, ...)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/gophercloud/gophercloud/openstack/client.go:297 +0x96\ngithub.com/openshift/origin/vendor/github.com/gophercloud/gophercloud/openstack.NewBlockStorageV3(...)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/gophercloud/gophercloud/openstack/client.go:353\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).NewBlockStorageV3(0xc001616f00, 0x0, 0x4b860b4, 0xc0000649e0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack_client.go:74 +0xc7\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).volumeService(0xc001616f00, 0x0, 0x0, 0xc0000649e0, 0xc007b0f850, 0x435e91, 0x4662c58)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack.go:913 +0x4e6\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).getVolume(0xc001616f00, 0xc0030af830, 0x24, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack_volumes.go:456 +0x76\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).diskIsUsed(0xc001616f00, 0xc0030af830, 0x24, 0xc007b0fa08, 0x73, 0xc007b0faf0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack_volumes.go:705 +0x5a\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).DeleteVolume(0xc001616f00, 0xc0030af830, 0x24, 0x0, 0x0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack_volumes.go:584 +0x4c\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/volume/cinder.(*DiskUtil).DeleteVolume(0x751c810, 0xc00befe510, 0xbffc773a84d6e5bd, 0x74fadc0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/volume/cinder/cinder_util.go:139 +0x8d\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/volume/cinder.(*cinderVolumeDeleter).Delete(0xc00befe510, 0x14, 0x44c1a64)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/volume/cinder/cinder.go:553 +0x3e\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume.(*PersistentVolumeController).doDeleteVolume(0xc00068e000, 0xc00a447180, 0xc0000cc001, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1344 +0x3c8\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume.(*PersistentVolumeController).deleteVolumeOperation(0xc00068e000, 0xc00a447180, 0x0, 0x2000000000040, 0x200010001, 0x8)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1201 +0x29b\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume.(*PersistentVolumeController).reclaimVolume.func2(0xc000187ed0, 0xc000187e40)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1058 +0x41\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/goroutinemap.(*goRoutineMap).Run.func1(0xc000d17290, 0xc00a0e26c0, 0x55, 0xc00a17ed60, 0x0, 0x0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:115 +0x12b\ncreated by github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/goroutinemap.(*goRoutineMap).Run\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:108 +0x2ea\n"
Version-Release number of selected component (if applicable):
4.5.27
How reproducible:
Steps to Reproduce:
1.
2.
3.
Actual results:
The PV Is not created
Expected results:
PV to be created
Additional info:
File 02848245.txt from compressed file provides:
- describe ES pod
- get pods -o wide -n openshift-logging
- clouds.yaml
- osp credentials
- describe pvc
- oc get pv (no pv)
- describe storage class
Must-gather and sosreport from infra node are provided in supportshell
Let me know if there is something else that we can provide.
Created attachment 1752804 [details] Nodes description, oc get cm, oc get secrets and file 02848245.txt w multiple info Description of problem: Customer is using Openstack Cinder as storage infrastructure. Currently it was verified that Cinder and OpenStack is not communicating properly anymore especially after upgrading from 4.5.2 to 4.5.27, causing to fail the creation of PVC with: Pending Status - Waiting to first consumer to be created before binding The same happens to whenever trying to create a persistent storage even with sample applications. I found these lines in the kube-controller-manager It looks that the pv created is: pvc-a4a959c2-79fe-461f-b7a6-745876e65959 -- Now, this 'Operation for "delete-pvc-a4a959c2-79fe-461f-b7a6-745876e65959[61bed42a-7ba7-4cf6-8424-5f725fe2b67b]" failed.' is what bugs me - Looks like it is trying to delete. Checking the sosreport from the node I didnt find any reference regarding the ES volumes, specially the PV. 2021-01-27T13:06:50.081631814Z I0127 13:06:50.081180 1 pv_cont2021-01-27T13:06:50.081670453Z roller.go:1285] isVolumeReleased[pvc-a4a959c2-79fe-461f-b7a6-745876e65959]: volume is released 2021-01-27T13:06:50.081670453Z W0127 13:06:50.081189 1 plugins.go:721] WARNING: kubernetes.io/cinder built-in volume provider is now deprecated. The Cinder volume provider is deprecated and will be removed in a future release 2021-01-27T13:06:50.081670453Z E0127 13:06:50.081307 1 goroutinemap.go:150] Operation for "delete-pvc-a4a959c2-79fe-461f-b7a6-745876e65959[61bed42a-7ba7-4cf6-8424-5f725fe2b67b]" failed. No retries permitted until 2021-01-27 13:06:50.581238771 +0000 UTC m=+16487.315893317 (durationBeforeRetry 500ms). Error: "recovered from panic \"runtime error: invalid memory address or nil pointer dereference\". (err=<nil>) Call stack:\ngoroutine 52734 [running]:\ngithub.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime.RecoverFromPanic(0xc007b0ffd0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:158 +0xb5\npanic(0x3c3cca0, 0x74c02e0)\n\t/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/panic.go:679 +0x1b2\ngithub.com/openshift/origin/vendor/github.com/gophercloud/gophercloud/openstack.initClientOpts(0x0, 0x44b5a3e, 0x8, 0x0, 0x0, 0xc0017cbda0, 0x8, 0x44b188d, 0x6, 0x44b5a3e, ...)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/gophercloud/gophercloud/openstack/client.go:297 +0x96\ngithub.com/openshift/origin/vendor/github.com/gophercloud/gophercloud/openstack.NewBlockStorageV3(...)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/gophercloud/gophercloud/openstack/client.go:353\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).NewBlockStorageV3(0xc001616f00, 0x0, 0x4b860b4, 0xc0000649e0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack_client.go:74 +0xc7\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).volumeService(0xc001616f00, 0x0, 0x0, 0xc0000649e0, 0xc007b0f850, 0x435e91, 0x4662c58)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack.go:913 +0x4e6\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).getVolume(0xc001616f00, 0xc0030af830, 0x24, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack_volumes.go:456 +0x76\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).diskIsUsed(0xc001616f00, 0xc0030af830, 0x24, 0xc007b0fa08, 0x73, 0xc007b0faf0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack_volumes.go:705 +0x5a\ngithub.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack.(*OpenStack).DeleteVolume(0xc001616f00, 0xc0030af830, 0x24, 0x0, 0x0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/legacy-cloud-providers/openstack/openstack_volumes.go:584 +0x4c\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/volume/cinder.(*DiskUtil).DeleteVolume(0x751c810, 0xc00befe510, 0xbffc773a84d6e5bd, 0x74fadc0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/volume/cinder/cinder_util.go:139 +0x8d\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/volume/cinder.(*cinderVolumeDeleter).Delete(0xc00befe510, 0x14, 0x44c1a64)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/volume/cinder/cinder.go:553 +0x3e\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume.(*PersistentVolumeController).doDeleteVolume(0xc00068e000, 0xc00a447180, 0xc0000cc001, 0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1344 +0x3c8\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume.(*PersistentVolumeController).deleteVolumeOperation(0xc00068e000, 0xc00a447180, 0x0, 0x2000000000040, 0x200010001, 0x8)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1201 +0x29b\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume.(*PersistentVolumeController).reclaimVolume.func2(0xc000187ed0, 0xc000187e40)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/persistentvolume/pv_controller.go:1058 +0x41\ngithub.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/goroutinemap.(*goRoutineMap).Run.func1(0xc000d17290, 0xc00a0e26c0, 0x55, 0xc00a17ed60, 0x0, 0x0)\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:115 +0x12b\ncreated by github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/goroutinemap.(*goRoutineMap).Run\n\t/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/util/goroutinemap/goroutinemap.go:108 +0x2ea\n" Version-Release number of selected component (if applicable): 4.5.27 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: The PV Is not created Expected results: PV to be created Additional info: File 02848245.txt from compressed file provides: - describe ES pod - get pods -o wide -n openshift-logging - clouds.yaml - osp credentials - describe pvc - oc get pv (no pv) - describe storage class Must-gather and sosreport from infra node are provided in supportshell Let me know if there is something else that we can provide.