Bug 2164633
| Summary: | KubePersistentVolumeFillingUp - False Alert firing for PVCs 0% free inodes | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Vincent S. Cojot <vcojot> |
| Component: | ceph-monitoring | Assignee: | Juan Miguel Olmo <jolmomar> |
| Status: | CLOSED DUPLICATE | QA Contact: | Harish NV Rao <hnallurv> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.11 | CC: | dkamboj, muagarwa, ocs-bugs, odf-bz-bot |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-04-03 09:07:41 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Vincent S. Cojot
2023-01-25 20:59:14 UTC
The PersistentVolume claimed by ocs4-image-registry-storage in Namespace Namespace NS openshift-image-registry only has 0% free inodes. But: ~ oc project openshift-image-registry Already on project "openshift-image-registry" on server "https://api.ocp4d.openshift.lasthome.solace.krynn:6443". ~ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE image-registry-storage Bound nfs-registry-storage 100Gi RWX 167m ocs4-image-registry-storage Bound pvc-94ba42bd-abf7-40f1-94a1-c7b081a857d1 2Ti RWX ocs-storagecluster-cephfs 125m ~ oc describe pvc ocs4-image-registry-storage Name: ocs4-image-registry-storage Namespace: openshift-image-registry StorageClass: ocs-storagecluster-cephfs Status: Bound Volume: pvc-94ba42bd-abf7-40f1-94a1-c7b081a857d1 Labels: <none> Annotations: pv.kubernetes.io/bind-completed: yes pv.kubernetes.io/bound-by-controller: yes volume.beta.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com volume.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com Finalizers: [kubernetes.io/pvc-protection] Capacity: 2Ti Access Modes: RWX VolumeMode: Filesystem Used By: image-registry-66c87f5f7f-n9qkw Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ExternalProvisioning 125m persistentvolume-controller waiting for a volume to be created, either by external provisioner "openshift-storage.cephfs.csi.ceph.com" or manually created by system administrator Normal Provisioning 125m openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-6bf84b7448-lbcd2_044132e5-b735-476c-85ab-4b119b65a1e3 External provisioner is provisioning volume for claim "openshift-image-registry/ocs4-image-registry-storage" Normal ProvisioningSucceeded 125m openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-6bf84b7448-lbcd2_044132e5-b735-476c-85ab-4b119b65a1e3 Successfully provisioned volume pvc-94ba42bd-abf7-40f1-94a1-c7b081a857d1 ~ ~$ oc get pods NAME READY STATUS RESTARTS AGE cluster-image-registry-operator-fb799676d-9xm5n 1/1 Running 2 3h10m image-registry-66c87f5f7f-n9qkw 1/1 Running 0 128m node-ca-5s46q 1/1 Running 1 152m node-ca-7vzlh 1/1 Running 1 152m node-ca-bps8l 1/1 Running 1 152m node-ca-dj99t 1/1 Running 1 178m node-ca-drm9b 1/1 Running 1 178m node-ca-fsdps 1/1 Running 1 152m node-ca-g7f4c 1/1 Running 2 178m node-ca-hfbkk 1/1 Running 1 152m node-ca-jvdfv 1/1 Running 1 178m node-ca-p8n48 1/1 Running 1 152m node-ca-vzwpb 1/1 Running 1 178m node-ca-wwz74 1/1 Running 1 178m ~$ oc rsh image-registry-66c87f5f7f-n9qkw sh-4.4$ df -ik /registry Filesystem Inodes IUsed IFree IUse% Mounted on 172.30.84.102:6789,172.30.55.247:6789,172.30.233.132:6789:/volumes/csi/csi-vol-7d709fad-9ce2-11ed-a74f-0a580ae0100c/9939d8de-ffb2-4216-9518-86ed58593ab0 2789 - - - /registry sh-4.4$ df -k /registry Filesystem 1K-blocks Used Available Use% Mounted on 172.30.84.102:6789,172.30.55.247:6789,172.30.233.132:6789:/volumes/csi/csi-vol-7d709fad-9ce2-11ed-a74f-0a580ae0100c/9939d8de-ffb2-4216-9518-86ed58593ab0 2147483648 0 2147483648 0% /registry the 2Ti volume is 100% empty ~$ oc get csv -n openshift-local-storage NAME DISPLAY VERSION REPLACES PHASE local-storage-operator.v4.12.0-202301042354 Local Storage 4.12.0-202301042354 Succeeded netobserv-operator.v0.2.2 NetObserv Operator 0.2.2 netobserv-operator.v0.2.1 Succeeded ~$ oc get csv -n openshift-storage NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.11.4 NooBaa Operator 4.11.4 mcg-operator.v4.11.3 Succeeded netobserv-operator.v0.2.2 NetObserv Operator 0.2.2 netobserv-operator.v0.2.1 Succeeded ocs-operator.v4.11.4 OpenShift Container Storage 4.11.4 ocs-operator.v4.11.3 Succeeded odf-csi-addons-operator.v4.11.4 CSI Addons 4.11.4 odf-csi-addons-operator.v4.11.3 Succeeded odf-operator.v4.11.4 OpenShift Data Foundation 4.11.4 odf-operator.v4.11.3 Succeeded This was solved in ODF 4.12. (not backported to ODF 4.11) See: https://bugzilla.redhat.com/show_bug.cgi?id=2132270 In the case the fix should be backported to ODF 4.11, PM should reopen the original bug asking for the backport. *** This bug has been marked as a duplicate of bug 2132270 *** |