Bug 1937196 - [aws ebs csi driver] events for block volume expansion may cause confusion
Summary: [aws ebs csi driver] events for block volume expansion may cause confusion
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.8
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 4.10.0
Assignee: Hemant Kumar
QA Contact: Penghao Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-10 06:12 UTC by Qin Ping
Modified: 2022-03-12 04:35 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-12 04:34:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-12 04:35:19 UTC

Description Qin Ping 2021-03-10 06:12:59 UTC
Description of problem:
events for block volume expansion may cause confusion

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-03-08-184701

How reproducible:
Always

Steps to Reproduce:
1. Created a block PVC with gp2-csi storageclass
2. Create a Pod consuming this PVC
3. Expand the PVC
4. Check PVC status and events

Actual results:
PVC is expanded successfully, from 1Gi to 3Gi. But the events say filesystem resize is required and successful.

$ oc describe pvc
Name:          test-pvc-5
Namespace:     default
StorageClass:  gp2-csi
Status:        Bound
Volume:        pvc-f91cf9a7-5198-47b1-9395-019861669e68
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
               volume.kubernetes.io/selected-node: ip-10-0-196-185.us-east-2.compute.internal
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      3Gi
Access Modes:  RWO
VolumeMode:    Block
Used By:       deployment-5-5d77f6566b-c94dp
Events:
  Type     Reason                      Age                From                                                                 Message
  ----     ------                      ----               ----                                                                 -------
  Normal   WaitForFirstConsumer        11m (x6 over 12m)  persistentvolume-controller                                          waiting for first consumer to be created before binding
  Normal   Provisioning                10m                ebs.csi.aws.com_ip-10-0-171-28_9df8b7d1-34fd-4354-9573-dc0e5a40557e  External provisioner is provisioning volume for claim "default/test-pvc-5"
  Normal   ExternalProvisioning        10m                persistentvolume-controller                                          waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
  Normal   ProvisioningSucceeded       10m                ebs.csi.aws.com_ip-10-0-171-28_9df8b7d1-34fd-4354-9573-dc0e5a40557e  Successfully provisioned volume pvc-f91cf9a7-5198-47b1-9395-019861669e68
  Normal   Resizing                    10m                external-resizer ebs.csi.aws.com                                     External resizer is resizing volume pvc-f91cf9a7-5198-47b1-9395-019861669e68
  Warning  ExternalExpanding           10m                volume_expand                                                        Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
  Normal   FileSystemResizeRequired    10m                external-resizer ebs.csi.aws.com                                     Require file system resize of volume on node
  Normal   FileSystemResizeSuccessful  9m27s              kubelet                                                              MountVolume.NodeExpandVolume succeeded for volume "pvc-f91cf9a7-5198-47b1-9395-019861669e68"



Expected results:
Filesystem resize is not required.

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 3 Hemant Kumar 2021-06-07 15:19:14 UTC
The BZ for aws driver should be fixed via https://github.com/openshift/aws-ebs-csi-driver/pull/187

Comment 5 Penghao Wang 2022-01-13 08:49:02 UTC
1. Created a block PVC with gp2-csi storageclass and create a Pod consuming this PVC wait for the pod running
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: block-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: gp2-csi
  volumeMode: Block
---
# Pod with block PVC
apiVersion: v1
kind: Pod
metadata:
  name: podwithblock
spec:
  containers:
    - name: httpd
      image: quay.io/pewang/storagetest:base
      ports:
        - containerPort: 8080
      command: ["/bin/sh", "-c"]
      args: [ "tail -f /dev/null" ]
      volumeDevices:
        - name: data
          devicePath: /dev/block
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: block

2. Expand the PVC and check PVC status and events by `oc describe pvc/block-pvc`
None "Filesystem resize" event.

oc describe pvc/block-pvc
Name:          block-pvc
Namespace:     default
StorageClass:  gp2-csi
Status:        Bound
Volume:        pvc-83a77320-c6f0-4e50-b9bd-2e0a61260b02
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
               volume.kubernetes.io/selected-node: ip-10-0-205-14.us-east-2.compute.internal
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      2Gi
Access Modes:  RWO
VolumeMode:    Block
Used By:       podwithblock
Events:
  Type     Reason                  Age                    From                                                                  Message
  ----     ------                  ----                   ----                                                                  -------
  Normal   WaitForFirstConsumer    3m (x3 over 3m26s)     persistentvolume-controller                                           waiting for first consumer to be created before binding
  Normal   Provisioning            2m48s                  ebs.csi.aws.com_ip-10-0-187-212_23d89bf3-c8c5-4b2e-8756-2bc46b9d2a21  External provisioner is provisioning volume for claim "default/block-pvc"
  Normal   ExternalProvisioning    2m45s (x2 over 2m48s)  persistentvolume-controller                                           waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator
  Normal   ProvisioningSucceeded   2m45s                  ebs.csi.aws.com_ip-10-0-187-212_23d89bf3-c8c5-4b2e-8756-2bc46b9d2a21  Successfully provisioned volume pvc-83a77320-c6f0-4e50-b9bd-2e0a61260b02
  Normal   Resizing                39s                    external-resizer ebs.csi.aws.com                                      External resizer is resizing volume pvc-83a77320-c6f0-4e50-b9bd-2e0a61260b02
  Warning  ExternalExpanding       39s                    volume_expand                                                         Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
  Normal   VolumeResizeSuccessful  32s                    external-resizer ebs.csi.aws.com                                      Resize volume succeeded


Verified passed on 4.10.0-0.nightly-2022-01-11-065245.

Comment 8 errata-xmlrpc 2022-03-12 04:34:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.