Bug 2040504
| Summary: | Change AWS EBS GP3 IOPS in MachineSet doesn't take affect in OpenShift 4 | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | JS <josantos> |
| Component: | Cloud Compute | Assignee: | Joel Speed <jspeed> |
| Cloud Compute sub component: | Other Providers | QA Contact: | Huali Liu <huliu> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | Keywords: | NeedsTestCase |
| Version: | 4.8 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.10.0 | ||
| Hardware: | x86_64 | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: The AWS Machine controllers did not set the IOPS value for block devices other than IO1 type
Consequence: IOPS fields was ignored for GP3 block devices
Fix: Set IOPS on all supported block device types
Result: Users can now set IOPS for the block device attached to the Machine
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-03-10 16:39:13 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2065483 | ||
Currently we only set the IOPS for `io1` block devices (https://github.com/openshift/machine-api-provider-aws/blob/main/pkg/actuators/machine/instances.go#L255). Based on the documentation (https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ec2/create-volume.html) it seems we should also be including `io2` and `gp3` disks in this check. Should be a relatively straight forward fix. Verified on 4.10.0-0.nightly-2022-01-17-223655
Steps:
1.Get MachineSet
liuhuali@Lius-MacBook-Pro huali-test % oc get machineset
NAME DESIRED CURRENT READY AVAILABLE AGE
huliu-035-xlb2l-worker-us-east-2a 1 1 1 1 3h7m
huliu-035-xlb2l-worker-us-east-2b 1 1 1 1 3h7m
huliu-035-xlb2l-worker-us-east-2c 1 1 1 1 3h7m
2.Scale down
liuhuali@Lius-MacBook-Pro huali-test % oc scale --replicas=0 machineset huliu-035-xlb2l-worker-us-east-2a
machineset.machine.openshift.io/huliu-035-xlb2l-worker-us-east-2a scaled
3.Change AWS EBS configs
liuhuali@Lius-MacBook-Pro huali-test % oc edit machineset huliu-035-xlb2l-worker-us-east-2a
machineset.machine.openshift.io/huliu-035-xlb2l-worker-us-east-2a edited
##### Old config
~~~
...
spec:
metadata: {}
providerSpec:
value:
ami:
id: ami-0c5b210f3fd086930
apiVersion: awsproviderconfig.openshift.io/v1beta1
blockDevices:
- ebs:
encrypted: true
iops: 0
kmsKey:
arn: ""
volumeSize: 120
volumeType: gp3
...
~~~
##### New config
~~~
...
spec:
metadata: {}
providerSpec:
value:
ami:
id: ami-0c5b210f3fd086930
apiVersion: awsproviderconfig.openshift.io/v1beta1
blockDevices:
- ebs:
encrypted: true
iops: 5000
kmsKey:
arn: ""
volumeSize: 120
volumeType: gp3
...
~~~
4.Scale up
liuhuali@Lius-MacBook-Pro huali-test % oc scale --replicas=1 machineset huliu-035-xlb2l-worker-us-east-2a
machineset.machine.openshift.io/huliu-035-xlb2l-worker-us-east-2a scaled
liuhuali@Lius-MacBook-Pro huali-test % oc get machineset
NAME DESIRED CURRENT READY AVAILABLE AGE
huliu-035-xlb2l-worker-us-east-2a 1 1 1 1 3h25m
huliu-035-xlb2l-worker-us-east-2b 1 1 1 1 3h25m
huliu-035-xlb2l-worker-us-east-2c 1 1 1 1 3h25m
liuhuali@Lius-MacBook-Pro huali-test %
5.Confirm via AWS console
New node created with an EBS, `gp3`, configured with `5000` IOPS.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056 |
Description of problem: After changing the IOPS size, for gp3 AWS EBS type defined in the MachineSet the new value it’s not configured properly within AWS EBS. Version-Release number of selected component (if applicable): Tested with OpenShift Container Platform 4.8.26 installed on top of AWS How reproducible: Always Steps to Reproduce: ===> 1) Get MachineSet List `MachineSet` and use a suitable one ~~~ $ oc get machineset -n openshift-machine-api NAME DESIRED CURRENT READY AVAILABLE AGE aws-ocp-worker-us-east-1a 1 1 1 1 3h30m aws-ocp-worker-us-east-1b 1 1 1 1 3h30m aws-ocp-worker-us-east-1c 1 1 1 1 3h30m aws-ocp-worker-us-east-1d 0 0 3h30m aws-ocp-worker-us-east-1e 0 0 3h30m aws-ocp-worker-us-east-1f 0 0 3h30m ~~~ ===> 2) Scale down Deleting the node to scale up after with new configurations ~~~ $ oc scale --replicas=0 machineset aws-ocp-worker-us-east-1a -n openshift-machine-api machineset.machine.openshift.io/aws-ocp-worker-us-east-1a scaled ~~~ ===> 3) Change AWS EBS configs The current config might have `gp2` settings adapt to `gp3` ~~~ $ oc edit machineset aws-ocp-worker-us-east-1a -n openshift-machine-api ~~~ ##### Old config ~~~ ... spec: metadata: {} providerSpec: value: ami: id: ami-06e6531aef9359b1e apiVersion: awsproviderconfig.openshift.io/v1beta1 blockDevices: - ebs: encrypted: true iops: 0 kmsKey: arn: "" volumeSize: 120 volumeType: gp2 ... ~~~ ##### New config ~~~ ... spec: metadata: {} providerSpec: value: ami: id: ami-06e6531aef9359b1e apiVersion: awsproviderconfig.openshift.io/v1beta1 blockDevices: - ebs: encrypted: true iops: 5000 kmsKey: arn: "" volumeSize: 120 volumeType: gp3 ... ~~~ Save and continue ===> 4) Scale up To create a new node at AWS and apply new configurations ~~~ $ oc scale --replicas=1 machineset aws-ocp-worker-us-east-1a -n openshift-machine-api machineset.machine.openshift.io/aws-ocp-worker-us-east-1a scaled ~~~ It's expected that the current output changes from: ~~~ $ oc get machineset -n openshift-machine-api NAME DESIRED CURRENT READY AVAILABLE AGE aws-ocp-worker-us-east-1a 0 0 3h32m aws-ocp-worker-us-east-1b 1 1 1 1 3h32m aws-ocp-worker-us-east-1c 1 1 1 1 3h32m aws-ocp-worker-us-east-1d 0 0 3h32m aws-ocp-worker-us-east-1e 0 0 3h32m aws-ocp-worker-us-east-1f 0 0 3h32m ~~~ To: ~~~ $ oc get machineset -n openshift-machine-api NAME DESIRED CURRENT READY AVAILABLE AGE aws-ocp-worker-us-east-1a 1 1 1 1 3h35m aws-ocp-worker-us-east-1b 1 1 1 1 3h35m aws-ocp-worker-us-east-1c 1 1 1 1 3h35m aws-ocp-worker-us-east-1d 0 0 3h35m aws-ocp-worker-us-east-1e 0 0 3h35m aws-ocp-worker-us-east-1f 0 0 3h35m ~~~ ===> 5) Confirm via AWS console It's possible to confirm that configured value, `5000`, was not applied. Actual results: New node it's created with an EBS, `gp3`, configured with `3000` IOPS. Expected results: New node created with an EBS, `gp3`, configured with `5000` IOPS.