| Summary: | Volume ID is missing in output message when expanding volume which is not mounted. | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Chao Yang <chaoyang> |
| Component: | Storage | Assignee: | Jan Safranek <jsafrane> |
| Storage sub component: | Storage | QA Contact: | Wei Duan <wduan> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | low | ||
| Priority: | unspecified | CC: | jnowicki, jsafrane, wduan |
| Version: | 4.10 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.11.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-08-10 10:43:03 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Bug Depends On: | 2059593 | ||
| Bug Blocks: | |||
Assigning to IBM. Arashad, feel free to reassign this to someone who's responsible for the CSI driver. And notice "severity: low" - it's a bug and it should be fixed, but it does not block 4.10. Anyway, fixing the error message "Description:The volume ID '%!d(MISSING)'" looks trivial. Offline volume expansion not supported from vpc infrastructure, for volume expand pvc should be in use by POD. We can improve the error msgs but expansion will not be supported in this case PR has been merged https://github.com/kubernetes-sigs/ibm-vpc-block-csi-driver/pull/62 fixed in https://github.com/kubernetes-sigs/ibm-vpc-block-csi-driver/releases/tag/v4.1.2 release, please use this release for this issue We need to update to v4.1.2 before 4.11 feature freeze. Verified following the "Steps to Reproduce" 1.Install OCP on ibmcloud 2.Create pvc/pod 3.Delete pod 4.Expand volume from 11Gi to 12Gi 5.oc describe pvc The offline resize doesn't succeed (Always showing Resizing) but no error message observed in BZ description Type Reason Age From Message ---- ------ ---- ---- ------- Warning VolumeResizeFailed 3h external-resizer vpc.block.csi.ibm.io resize volume "pvc-575e17a0-b799-4487-a0c0-937bc307d306" by resizer "vpc.block.csi.ibm.io" failed: rpc error: code = Unavailable desc = error reading from server: EOF Normal Resizing 3h external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 Normal Resizing 167m external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 Normal Resizing 154m external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 Normal Resizing 142m external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 Normal Resizing 129m external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 Normal Resizing 116m external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 Normal Resizing 103m external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 Normal Resizing 90m external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 Normal Resizing 77m external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 Normal Resizing 64m external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 Normal Resizing 52m external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 Normal Resizing 39m external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 Normal Resizing 26m external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 Normal Resizing 13m external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 After re-create the pod and consume this pvc, resize succeeded finially. Normal Resizing 38m external-resizer vpc.block.csi.ibm.io External resizer is resizing volume pvc-575e17a0-b799-4487-a0c0-937bc307d306 Normal FileSystemResizeRequired 25m external-resizer vpc.block.csi.ibm.io Require file system resize of volume on node Normal FileSystemResizeSuccessful 25m (x2 over 7h54m) kubelet MountVolume.NodeExpandVolume succeeded for volume "pvc-575e17a0-b799-4487-a0c0-937bc307d306" wduan-0615d-i-w2g25-worker-3-4lhb4 Change status to "Verified", the process and logs look good to me. For offline keeping resizing, tracked by another BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2059593 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069 |
Description of problem: Volume ID is missing in output message when expanding volume which is not mounted. Version-Release number of selected component (if applicable): 4.10.0-0.nightly-2022-01-16-191814 How reproducible: Always Steps to Reproduce: 1.Install OCP on ibmcloud 2.Create pvc/pod 3.Delete pod 4.Expand volume from 10Gi to 11Gi 5.oc describe pvc Warning ExternalExpanding 27m volume_expand Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC. Warning VolumeResizeFailed 14m external-resizer vpc.block.csi.ibm.io resize volume "pvc-3d3d522a-3a80-44b9-92ff-7a9db7864814" by resizer "vpc.block.csi.ibm.io" failed: rpc error: code = Internal desc = {RequestID: 5beb070a-c2a9-49e1-b0bc-76653ab52396 , Code: InternalError, Description: Internal error occurred, BackendError: {Code:FailedToExpandVolume, Type:ExpansionFailed, Description:The volume ID '%!d(MISSING)' could not be expanded from your VPC., BackendError:Trace Code:1ada6fb0-b753-49be-9f83-02e77b497eee, Bad Request Please check http://www.bluemix.com/help#volume_not_attached, RC:500}, Action: Please check 'BackendError' tag for more details} 6.oc get pv/pvc-3d3d522a-3a80-44b9-92ff-7a9db7864814 -o json | jq .spec.csi { “driver”: “vpc.block.csi.ibm.io”, “fsType”: “ext4", “volumeAttributes”: { “clusterID”: “”, “failure-domain.beta.kubernetes.io/region”: “eu-gb”, “failure-domain.beta.kubernetes.io/zone”: “eu-gb-2”, “iops”: “3000”, “storage.kubernetes.io/csiProvisionerIdentity”: “1642492112548-8081-vpc.block.csi.ibm.io”, “tags”: “”, “volumeCRN”: “crn:v1:bluemix:public:is:eu-gb-2:a/fdc2e14cf8bc4d53a67f972dc2e2c861::volume:r018-5f5703d1-9297-4aa8-9896-b0d2f800e2d2”, “volumeId”: “r018-5f5703d1-9297-4aa8-9896-b0d2f800e2d2” }, “volumeHandle”: “r018-5f5703d1-9297-4aa8-9896-b0d2f800e2d2” } Actual results: Volume ID is missing in output message when expanding volume which is not mounted. Expected results: Volume ID should not be null. Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info: