Bug 2040504 - Change AWS EBS GP3 IOPS in MachineSet doesn't take affect in OpenShift 4
Summary: Change AWS EBS GP3 IOPS in MachineSet doesn't take affect in OpenShift 4
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Compute
Version: 4.8
Hardware: x86_64
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.10.0
Assignee: Joel Speed
QA Contact: Huali Liu
URL:
Whiteboard:
Depends On:
Blocks: 2065483
TreeView+ depends on / blocked
 
Reported: 2022-01-13 21:38 UTC by JS
Modified: 2022-10-26 07:32 UTC (History)
0 users

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The AWS Machine controllers did not set the IOPS value for block devices other than IO1 type Consequence: IOPS fields was ignored for GP3 block devices Fix: Set IOPS on all supported block device types Result: Users can now set IOPS for the block device attached to the Machine
Clone Of:
Environment:
Last Closed: 2022-03-10 16:39:13 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-api-provider-aws pull 18 0 None open Bug 2040504: Ensure IOPS setting is honoured on all supported block device types 2022-01-14 10:44:00 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:39:33 UTC

Description JS 2022-01-13 21:38:02 UTC
Description of problem:
After changing the IOPS size, for gp3 AWS EBS type defined in the MachineSet the new value it’s not configured properly within AWS EBS.

Version-Release number of selected component (if applicable):
Tested with OpenShift Container Platform 4.8.26 installed on top of AWS

How reproducible:
Always

Steps to Reproduce:
===> 1) Get MachineSet

List `MachineSet` and use a suitable one

~~~
$ oc get machineset -n openshift-machine-api
NAME                                            DESIRED   CURRENT   READY   AVAILABLE   AGE
aws-ocp-worker-us-east-1a   1         1         1       1           3h30m
aws-ocp-worker-us-east-1b   1         1         1       1           3h30m
aws-ocp-worker-us-east-1c   1         1         1       1           3h30m
aws-ocp-worker-us-east-1d   0         0                             3h30m
aws-ocp-worker-us-east-1e   0         0                             3h30m
aws-ocp-worker-us-east-1f   0         0                             3h30m
~~~

===> 2) Scale down 
Deleting the node to scale up after with new configurations

~~~
$ oc scale --replicas=0 machineset aws-ocp-worker-us-east-1a -n openshift-machine-api
machineset.machine.openshift.io/aws-ocp-worker-us-east-1a scaled
~~~

===> 3) Change AWS EBS configs
The current config might have `gp2` settings adapt to `gp3`

~~~
$ oc edit machineset aws-ocp-worker-us-east-1a -n openshift-machine-api
~~~

##### Old config
~~~
...
    spec:
      metadata: {}
      providerSpec:
        value:
          ami:
            id: ami-06e6531aef9359b1e
          apiVersion: awsproviderconfig.openshift.io/v1beta1
          blockDevices:
          - ebs:
              encrypted: true
              iops: 0
              kmsKey:
                arn: ""
              volumeSize: 120
              volumeType: gp2
...
~~~

##### New config
~~~
...
    spec:
      metadata: {}
      providerSpec:
        value:
          ami:
            id: ami-06e6531aef9359b1e
          apiVersion: awsproviderconfig.openshift.io/v1beta1
          blockDevices:
          - ebs:
              encrypted: true
              iops: 5000
              kmsKey:
                arn: ""
              volumeSize: 120
              volumeType: gp3
...
~~~

Save and continue

===> 4) Scale up
To create a new node at AWS and apply new configurations

~~~
$ oc scale --replicas=1 machineset aws-ocp-worker-us-east-1a -n openshift-machine-api
machineset.machine.openshift.io/aws-ocp-worker-us-east-1a scaled
~~~

It's expected that the current output changes from:
~~~
$ oc get machineset -n openshift-machine-api                                                          
NAME                                            DESIRED   CURRENT   READY   AVAILABLE   AGE
aws-ocp-worker-us-east-1a   0         0                             3h32m
aws-ocp-worker-us-east-1b   1         1         1       1           3h32m
aws-ocp-worker-us-east-1c   1         1         1       1           3h32m
aws-ocp-worker-us-east-1d   0         0                             3h32m
aws-ocp-worker-us-east-1e   0         0                             3h32m
aws-ocp-worker-us-east-1f   0         0                             3h32m
~~~

To:
~~~
$ oc get machineset -n openshift-machine-api
NAME                                            DESIRED   CURRENT   READY   AVAILABLE   AGE
aws-ocp-worker-us-east-1a   1         1         1       1           3h35m
aws-ocp-worker-us-east-1b   1         1         1       1           3h35m
aws-ocp-worker-us-east-1c   1         1         1       1           3h35m
aws-ocp-worker-us-east-1d   0         0                             3h35m
aws-ocp-worker-us-east-1e   0         0                             3h35m
aws-ocp-worker-us-east-1f   0         0                             3h35m
~~~

===> 5) Confirm via AWS console
It's possible to confirm that configured value, `5000`, was not applied.


Actual results:
New node it's created with an EBS, `gp3`, configured with `3000` IOPS.

Expected results:
New node created with an EBS, `gp3`, configured with `5000` IOPS.

Comment 1 Joel Speed 2022-01-14 10:36:46 UTC
Currently we only set the IOPS for `io1` block devices (https://github.com/openshift/machine-api-provider-aws/blob/main/pkg/actuators/machine/instances.go#L255). Based on the documentation (https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/create-volume.html) it seems we should also be including `io2` and `gp3` disks in this check.

Should be a relatively straight forward fix.

Comment 6 Huali Liu 2022-01-19 05:31:20 UTC
Verified on 4.10.0-0.nightly-2022-01-17-223655
Steps:

1.Get MachineSet
liuhuali@Lius-MacBook-Pro huali-test % oc get machineset
NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
huliu-035-xlb2l-worker-us-east-2a   1         1         1       1           3h7m
huliu-035-xlb2l-worker-us-east-2b   1         1         1       1           3h7m
huliu-035-xlb2l-worker-us-east-2c   1         1         1       1           3h7m

2.Scale down
liuhuali@Lius-MacBook-Pro huali-test % oc scale --replicas=0 machineset huliu-035-xlb2l-worker-us-east-2a
machineset.machine.openshift.io/huliu-035-xlb2l-worker-us-east-2a scaled

3.Change AWS EBS configs
liuhuali@Lius-MacBook-Pro huali-test % oc edit machineset huliu-035-xlb2l-worker-us-east-2a
machineset.machine.openshift.io/huliu-035-xlb2l-worker-us-east-2a edited
##### Old config
~~~
...
    spec:
      metadata: {}
      providerSpec:
        value:
          ami:
            id: ami-0c5b210f3fd086930
          apiVersion: awsproviderconfig.openshift.io/v1beta1
          blockDevices:
          - ebs:
              encrypted: true
              iops: 0
              kmsKey:
                arn: ""
              volumeSize: 120
              volumeType: gp3
...
~~~

##### New config
~~~
...
    spec:
      metadata: {}
      providerSpec:
        value:
          ami:
            id: ami-0c5b210f3fd086930
          apiVersion: awsproviderconfig.openshift.io/v1beta1
          blockDevices:
          - ebs:
              encrypted: true
              iops: 5000
              kmsKey:
                arn: ""
              volumeSize: 120
              volumeType: gp3
...
~~~

4.Scale up
liuhuali@Lius-MacBook-Pro huali-test % oc scale --replicas=1 machineset huliu-035-xlb2l-worker-us-east-2a
machineset.machine.openshift.io/huliu-035-xlb2l-worker-us-east-2a scaled
liuhuali@Lius-MacBook-Pro huali-test % oc get machineset
NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
huliu-035-xlb2l-worker-us-east-2a   1         1         1       1           3h25m
huliu-035-xlb2l-worker-us-east-2b   1         1         1       1           3h25m
huliu-035-xlb2l-worker-us-east-2c   1         1         1       1           3h25m
liuhuali@Lius-MacBook-Pro huali-test % 

5.Confirm via AWS console
New node created with an EBS, `gp3`, configured with `5000` IOPS.

Comment 10 errata-xmlrpc 2022-03-10 16:39:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.