Bug 1958154 - Custom AWS user tags limit not supported (openshift/api says max=25), install fails when >=10
Summary: Custom AWS user tags limit not supported (openshift/api says max=25), install...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.8
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.9.0
Assignee: Aditya Narayanaswamy
QA Contact: Yunfei Jiang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-07 10:37 UTC by Andrew McDermott
Modified: 2021-10-18 17:31 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
AWS S3 bucket allows only a maximum of 10 custom tags and hence we should restrict the number of custom user tags to 8 since we use 2 of our own to tag the resources and to figure out what resources to delete during cluster destruction.
Clone Of:
Environment:
Last Closed: 2021-10-18 17:31:03 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift installer pull 5154 0 None None None 2021-08-17 15:54:39 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:31:28 UTC

Description Andrew McDermott 2021-05-07 10:37:25 UTC
I was doing a demo of the experimental AWS user tags feature and had
the following custom tags defined.

  platform:
    aws:
      experimentalPropagateUserTags: true
      userTags:
        managedBy: redhat.com
        administrator: root
        customTag-1: customValue-2021-05-07-100959
        customTag-2: customValue-2021-05-07-101004
        customTag-3: customValue-2021-05-07-101009
        customTag-4: customValue-2021-05-07-101014
        customTag-5: customValue-2021-05-07-101019
        customTag-6: customValue-2021-05-07-101024
        customTag-7: customValue-2021-05-07-101029
        customTag-8: customValue-2021-05-07-101034

Creating the cluster I get the following error:

$ ./openshift-install create cluster --dir=/Users/aim/r/clusters/4.8.0-0.ci-2021-05-03-215023/aws-amcdermo-2021-05-07-1035
INFO Credentials loaded from the "default" profile in file "/Users/aim/.aws/credentials"
INFO Consuming Install Config from target directory
INFO Creating infrastructure resources...
ERROR
ERROR Error: Error putting object in S3 bucket (amcdermo-2021-05-07-1-lmbxp-bootstrap): BadRequest: Object tags cannot be greater than 10
ERROR 	status code: 400, request id: 4NC1X50HD9A552TR, host id: qWGyIRj9oW5n337vZKf1l6NKzKxj6O5XcmtFRRbeJVTl1+2DdCJtS+10Av4BNREd/hEgUznEC94=
ERROR
ERROR   on ../../../../../private/var/folders/f7/r2n_jh4s3c39sk4m1c9r0z380000z8/T/openshift-install-869013030/bootstrap/main.tf line 26, in resource "aws_s3_bucket_object" "ignition":
ERROR   26: resource "aws_s3_bucket_object" "ignition" {
ERROR
ERROR
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

My install version was: 4.8.0-0.ci-2021-05-03-215023

We say that we can support 25 additional tags:

https://github.com/openshift/api/blob/master/config/v1/0000_10_config-operator_01_infrastructure.crd.yaml#L252

  resourceTags:
    description: resourceTags is a list of additional tags to
      apply to AWS resources created for the cluster. See https://docs.aws.amazon.com/general/latest/gr/aws_tagging.html
      for information on tagging AWS resources. AWS supports a
      maximum of 50 tags per resource. OpenShift reserves 25 tags
      for its use, leaving 25 tags available for the user.
    type: array
    maxItems: 25

This AWS doc says a maximum of 10:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/tagging-managing.html

  "You can associate up to 10 tags with an object. Tags associated
   with an object must have unique tag keys."

And this AWS doc says 50:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions

  "Maximum number of tags per resource – 50"

I tried another install with:

  platform:
    aws:
      region: ${AWS_REGION}
      experimentalPropagateUserTags: true
      userTags:
        managedBy: redhat.com
        administrator: root
        customTag-1: customValue-2021-05-07-100959
        customTag-2: customValue-2021-05-07-101004
        customTag-3: customValue-2021-05-07-101009
        customTag-4: customValue-2021-05-07-101014
        customTag-5: customValue-2021-05-07-101019
        customTag-6: customValue-2021-05-07-101024
        customTag-7: customValue-2021-05-07-101029

And got:

$ ./openshift-install create cluster --dir=/Users/aim/r/clusters/4.8.0-0.ci-2021-05-03-215023/aws-amcdermo-2021-05-07-1044
INFO Credentials loaded from the "default" profile in file "/Users/aim/.aws/credentials"
INFO Consuming Install Config from target directory
INFO Creating infrastructure resources...
ERROR
ERROR Error: Error putting object in S3 bucket (amcdermo-2021-05-07-1-kwckf-bootstrap): BadRequest: Object tags cannot be greater than 10
ERROR 	status code: 400, request id: WBARWX5WYZBDV5S2, host id: S4Ljlxgc39zxEL9EjdQzGPoAdpkSh1aCmO5pv+6tNDh0gpMcYzGVLx//pasSijRWX4ST9N2iEp4=
ERROR
ERROR   on ../../../../../private/var/folders/f7/r2n_jh4s3c39sk4m1c9r0z380000z8/T/openshift-install-774161306/bootstrap/main.tf line 26, in resource "aws_s3_bucket_object" "ignition":
ERROR   26: resource "aws_s3_bucket_object" "ignition" {
ERROR
ERROR
FATAL failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed to apply Terraform: failed to complete the change

^^ so that one is weird given that there are not 10 tags listed. But
as these are "additional" tags perhaps the installer adds other tags
that I'm not aware of which would make this >= 10.

I tried again with:

  platform:
    aws:
      region: ${AWS_REGION}
      experimentalPropagateUserTags: true
      userTags:
        managedBy: redhat.com
        administrator: root
        customTag-1: customValue-2021-05-07-100959
        customTag-2: customValue-2021-05-07-101004

which was successful.

$ ./openshift-install create cluster --dir=/Users/aim/r/clusters/4.8.0-0.ci-2021-05-03-215023/aws-amcdermo-2021-05-07-1121
INFO Credentials loaded from the "default" profile in file "/Users/aim/.aws/credentials"
INFO Consuming Install Config from target directory
INFO Creating infrastructure resources...
INFO Waiting up to 20m0s for the Kubernetes API at https://api.amcdermo-2021-05-07-1121.devcluster.openshift.com:6443...

Comment 1 Matthew Staebler 2021-05-07 15:59:34 UTC
That's a good find. Since s3 only supports 10 tags, we need to drastically lower the number of user tags that OpenShift claims to support.

Comment 2 Russell Teague 2021-08-02 17:38:52 UTC
Need to address, not prioritized.

Comment 4 Yunfei Jiang 2021-08-26 04:17:14 UTC
verified. PASS.
OCP version: 4.9.0-0.nightly-2021-08-25-185404

install-config.yaml:
platform:
  aws:
    region: us-east-2
    userTags:
        customTag-1: value-1
        customTag-2: value-2
        customTag-3: value-3
        customTag-4: value-4
        customTag-5: value-5
        customTag-6: value-6
        customTag-7: value-7
        customTag-8: value-8
        customTag-9: value-9

./openshift-install create ignition-configs --dir cluster1
FATAL failed to fetch Kubeconfig Admin Client: failed to load asset "Install Config": invalid "install-config.yaml" file: platform.aws.userTags: Invalid value: 9: number of user tags cannot be more than 8

Comment 7 errata-xmlrpc 2021-10-18 17:31:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.